论文标题
句子代表学习具有生成目标而不是对比目标
Sentence Representation Learning with Generative Objective rather than Contrastive Objective
论文作者
论文摘要
尽管提供了令人惊叹的上下文化令牌级表示，但当前的预训练的语言模型会更少注意在他们自我监督的预训练期间准确获取句子级表示。但是，主导当前句子表示学习的对比目标几乎没有语言解释性，并且在下游语义任务上没有表现保证。相反，我们提出了一个基于短语重建的新颖生成的自我监督学习目标。为了克服以前生成方法的缺点，我们通过将一个句子分解为重要短语来仔细地对句子内结构进行建模。实证研究表明，我们的生成学习实现了足够强大的性能提高，不仅在STS基准上，而且在下游的语义检索和重新骑行任务上都超过了当前的最新对比方法。我们的代码可在https://github.com/chengzhipanpan/paser上找到。
Though offering amazing contextualized token-level representations, current pre-trained language models take less attention on accurately acquiring sentence-level representation during their self-supervised pre-training. However, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream semantic tasks. We instead propose a novel generative self-supervised learning objective based on phrase reconstruction. To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases. Empirical studies show that our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks. Our code is available at https://github.com/chengzhipanpan/PaSeR.
