论文标题
任务指南针：用任务前缀缩放多任务预训练
Task Compass: Scaling Multi-task Pre-training with Task Prefix
论文作者
论文摘要
将任务意识的注释数据作为有监督的信号，以协助对大规模无标记数据进行自我监督的学习已成为培训前语言模型的新趋势。现有研究表明，具有大规模监督任务的多任务学习会遭受跨任务的负面影响。为了应对挑战，我们提出了一个任务前缀指导的多任务预训练框架，以探索任务之间的关系。我们在40个数据集上进行了广泛的实验，这表明我们的模型不仅可以作为各种任务的强大基础骨干，而且还可以作为分析任务关系的探测工具。任务关系由前缀在任务之间对齐转移学习绩效。他们还建议通过互补任务提出数据增强的指示，这有助于我们的模型在常识性推理排行榜上实现人类准则结果。代码可从https://github.com/cooelf/compassmtl获得
Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models. Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. To tackle the challenge, we propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. We conduct extensive experiments on 40 datasets, which show that our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships. The task relationships reflected by the prefixes align transfer learning performance between tasks. They also suggest directions for data augmentation with complementary tasks, which help our model achieve human-parity results on commonsense reasoning leaderboards. Code is available at https://github.com/cooelf/CompassMTL
