论文标题
一个简单的对比学习目标，用于减轻神经文本退化
A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration
论文作者
论文摘要
事实证明，跨透明镜目标是自回归语言模型（LMS）的通用培训目标。但是，在不考虑有问题的令牌的惩罚的情况下，使用跨透镜训练的LMS表现出文本变性。为了解决这个问题，已经提出了不可能的培训，以减少LMS预测的不可能的代币的可能性。但是，不可能的不可能考虑标签令牌与不太可能的候选人之间的关系，从而显示出堕落的边缘改善。我们提出了一个新的对比令牌学习目标，该目标继承了跨内向和不可能训练的优势，并避免了它们的局限性。关键思想是教LM，以产生高概率，以实现负面候选人的标签令牌和低概率。关于语言建模和开放域对话生成任务的综合实验表明，提出的对比令牌目标的重复性文本要低得多，而产生质量高于基线方法，从而实现了文本退化的新最先进的表现。
The cross-entropy objective has proved to be an all-purpose training objective for autoregressive language models (LMs). However, without considering the penalization of problematic tokens, LMs trained using cross-entropy exhibit text degeneration. To address this, unlikelihood training has been proposed to reduce the probability of unlikely tokens predicted by LMs. But unlikelihood does not consider the relationship between the label tokens and unlikely token candidates, thus showing marginal improvements in degeneration. We propose a new contrastive token learning objective that inherits the advantages of cross-entropy and unlikelihood training and avoids their limitations. The key idea is to teach a LM to generate high probabilities for label tokens and low probabilities of negative candidates. Comprehensive experiments on language modeling and open-domain dialogue generation tasks show that the proposed contrastive token objective yields much less repetitive texts, with a higher generation quality than baseline approaches, achieving the new state-of-the-art performance on text degeneration.
