论文标题
从任务说明中学习
Learning from Task Descriptions
论文作者
论文摘要
通常，机器学习系统通过培训数千个示例来解决新任务。相比之下，人类可以通过阅读一些说明来解决新任务，也许有一两个示例。为了迈出缩小这一差距的一步，我们引入了一个框架，用于开发NLP系统，该系统在阅读了其描述后解决了新任务，从而综合了该领域的先前工作。我们使用新的英语数据集实例化此框架，该框架是针对未见任务的以任务为导向的评估的。将任务描述制定为问题，我们确保每个任务都足够通用，以应用于许多可能的输入，从而全面评估模型解决每个任务的能力。此外，数据集的结构测试了特定类型的系统概括。我们发现，最先进的T5模型在热皮上取得了12％的分数，对NLP研究人员留下了重大挑战。
Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.
