论文标题
英语中级任务培训也改善了零射的跨语性转移
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
论文作者
论文摘要
中级任务培训---在目标任务再次进行微调之前对中间任务进行预审计的模型 - 通常会在单语英语设置中的语言理解任务上大大改善模型性能。我们研究英语中级培训是否仍然对非英语目标任务有所帮助。使用九个中间语言理解任务，我们在Xtreme基准上的零击跨语言设置中评估了中间任务传输。我们看到了从BUCC和TATOEBA句子检索任务中的中级培训以及对提问目标任务的适度改进。 MNLI，小队和Hellaswag作为中间任务取得了最佳的总体结果，而多任务中级则提供了少量的其他改进。使用我们针对每个目标任务的最佳中级任务模型，我们在Xtreme基准上获得了5.4点的改进，截至2020年6月，我们还设置了最新技术。我们还研究了在中间任务训练中继续进行多种语言MLM，并使用机器上的中间任务数据进行了多种语言MLM，但既不一致地表现出英语互动式培训。
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tasks and moderate improvements on question-answering target tasks. MNLI, SQuAD and HellaSwag achieve the best overall results as intermediate tasks, while multi-task intermediate offers small additional improvements. Using our best intermediate-task models for each target task, we obtain a 5.4 point improvement over XLM-R Large on the XTREME benchmark, setting the state of the art as of June 2020. We also investigate continuing multilingual MLM during intermediate-task training and using machine-translated intermediate-task data, but neither consistently outperforms simply performing English intermediate-task training.
