论文标题
神经模型是否学习自然语言的单调性推断的系统性？
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
论文作者
论文摘要
尽管语言模型使用神经网络取得了成功，但尚不清楚神经模型在多大程度上具有执行推论的概括能力。在本文中，我们介绍了一种评估神经模型是否可以学习自然语言的单调性推断的系统性，即，对构图的概括进行任意推论的规律性。我们考虑单调性推断的四个方面，并测试模型是否可以系统地解释不同训练/测试分裂的词汇和逻辑现象。一系列实验表明，当句子的句法结构与训练集和测试集之间相似时，三个神经模型会系统地借鉴词汇和逻辑现象的看不见组合的推断。但是，当测试集中的结构略有变化时，模型的性能会显着降低，同时保留了训练集中已经出现的所有词汇和成分。这表明神经模型的概括能力仅限于句法结构几乎与训练集中相同的情况。
Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language, namely, the regularity for performing arbitrary inferences with generalization on composition. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on unseen combinations of lexical and logical phenomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the structures are slightly changed in the test set while retaining all vocabularies and constituents already appearing in the training set. This indicates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set.
