论文标题
语言模型是方向推断的差学习者
Language Models Are Poor Learners of Directional Inference
论文作者
论文摘要
我们通过提示进行微调来检查LMS方向性谓词的能力。我们的分析表明，与他们在标准NLI上的明显成功相反，LMS显示出有限的学习这种定向推断的能力。此外，现有数据集无法测试方向性，并且/或被可以将其作为代理的人工制品所侵害，从而产生了过度优势的结果。作为回应，我们提出了BOOQA（布尔Open QA），这是一个强大的多语性评估基准，用于定向谓词占用，对现有训练集外的外在。在BOOQA上，我们建立了基线并显示了现有的LM启动模型是无能的指导学习者的证据，而不是需要稀疏性的限制。
We examine LMs' competence of directional predicate entailments by supervised fine-tuning with prompts. Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets. On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity.
