论文标题
预先训练的语言模型作为重新通信者
Pre-trained Language Models as Re-Annotators
论文作者
论文摘要
注释噪声在数据集中很普遍，但是手动修改有缺陷的语料库是耗时且容易出错的。因此，鉴于预先训练的语言模型的先验知识以及所有注释中的预期均匀性，我们试图通过两项任务自动减少语料库中的注释噪声：（1）注释不一致的检测指示注释可信度，（2）注释误差纠正校正，以使异常的异常注释纠正。 我们研究了如何从预训练的语言模型中获取语义敏感的注释表示，即使没有微调即使没有进行微调，也希望将示例与相互邻接位置相同的示例嵌入。我们提出了一个新颖的信誉得分，以揭示基于邻近一致性的注释不一致的可能性。然后，我们通过交叉验证进行注释校正，微调基于训练的语言模型的分类器。用两种方法进一步阐述了注释校正器：（1）内核密度估计和（2）新型远处对比损失的软标记。 我们研究了关系提取中的重新注释，并创建了一个新的手动修订数据集，重新转移，以评估文档级别的重新注册。拟议的信誉得分显示了与人类修订的有前途的协议，在分别检测出关于诱人和docred的不一致方面的二进制F1和72.5的二进制F1。此外，基于遥远的对比度学习和不确定标签的邻居感知分类器可在校正塔克雷和docred的注释时达到高达66.2和57.8的宏F1。这些改进不仅是理论上的：而自动降级的训练集证明了最先进的关系提取模型的性能提高了3.6％。
Annotation noise is widespread in datasets, but manually revising a flawed corpus is time-consuming and error-prone. Hence, given the prior knowledge in Pre-trained Language Models and the expected uniformity across all annotations, we attempt to reduce annotation noise in the corpus through two tasks automatically: (1) Annotation Inconsistency Detection that indicates the credibility of annotations, and (2) Annotation Error Correction that rectifies the abnormal annotations. We investigate how to acquire semantic sensitive annotation representations from Pre-trained Language Models, expecting to embed the examples with identical annotations to the mutually adjacent positions even without fine-tuning. We proposed a novel credibility score to reveal the likelihood of annotation inconsistencies based on the neighbouring consistency. Then, we fine-tune the Pre-trained Language Models based classifier with cross-validation for annotation correction. The annotation corrector is further elaborated with two approaches: (1) soft labelling by Kernel Density Estimation and (2) a novel distant-peer contrastive loss. We study the re-annotation in relation extraction and create a new manually revised dataset, Re-DocRED, for evaluating document-level re-annotation. The proposed credibility scores show promising agreement with human revisions, achieving a Binary F1 of 93.4 and 72.5 in detecting inconsistencies on TACRED and DocRED respectively. Moreover, the neighbour-aware classifiers based on distant-peer contrastive learning and uncertain labels achieve Macro F1 up to 66.2 and 57.8 in correcting annotations on TACRED and DocRED respectively. These improvements are not merely theoretical: Rather, automatically denoised training sets demonstrate up to 3.6% performance improvement for state-of-the-art relation extraction models.
