论文标题
通过对比度学习无监督的无参考摘要质量评估
Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning
论文作者
论文摘要
对文档摘要系统的评估一直是影响摘要任务成功的关键因素。先前的方法，例如胭脂，主要考虑评估摘要的信息，并需要每个测试摘要的人类引用。在这项工作中，我们建议评估摘要质量，而无需通过无监督的对比学习来评估摘要。具体来说，我们设计了一个新的指标，该指标涵盖了基于Bert的语言品质和语义信息。为了学习指标，对于每个摘要，我们就摘要质量的不同方面构建了不同类型的负样本，并以排名损失来训练我们的模型。新闻编辑室和CNN/每日邮件的实验表明，即使没有参考摘要，我们的新评估方法即使没有其他指标。此外，我们表明我们的方法是一般且可在数据集中传输的。
Evaluation of a document summarization system has been a critical factor to impact the success of the summarization task. Previous approaches, such as ROUGE, mainly consider the informativeness of the assessed summary and require human-generated references for each test summary. In this work, we propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. To learn the metric, for each summary, we construct different types of negative samples with respect to different aspects of the summary qualities, and train our model with a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries. Furthermore, we show that our method is general and transferable across datasets.
