论文标题
分布一致的微调，以进行有效的神经检索
Distribution-Aligned Fine-Tuning for Efficient Neural Retrieval
论文作者
论文摘要
基于双重编码器的神经检索模型由于其语义匹配功能，因此可以很好地取得可观的性能，并可以很好地补充传统的词汇检索器，这使它们成为混合IR系统的共同选择。但是，这些模型在在线查询编码步骤中表现出性能瓶颈，因为相应的查询编码器通常是大型且复杂的变压器模型。 在本文中，我们研究了异质的双编码模型，其中两个编码器是不共享参数或初始化的单独模型。我们从经验上表明，异质的双重编码器易于崩溃，导致它们在使用标准对比度不匹配引起的标准对比度损失进行微调时，输出恒定的微不足道表示。我们提出了DAFT，这是一种简单的两阶段微调方法，它可以对齐两个编码器，以防止它们崩溃。我们进一步证明了如何使用轻量级查询编码器使用DAFT来训练有效的异质双编码模型。
Dual-encoder-based neural retrieval models achieve appreciable performance and complement traditional lexical retrievers well due to their semantic matching capabilities, which makes them a common choice for hybrid IR systems. However, these models exhibit a performance bottleneck in the online query encoding step, as the corresponding query encoders are usually large and complex Transformer models. In this paper we investigate heterogeneous dual-encoder models, where the two encoders are separate models that do not share parameters or initializations. We empirically show that heterogeneous dual-encoders are susceptible to collapsing representations, causing them to output constant trivial representations when they are fine-tuned using a standard contrastive loss due to a distribution mismatch. We propose DAFT, a simple two-stage fine-tuning approach that aligns the two encoders in order to prevent them from collapsing. We further demonstrate how DAFT can be used to train efficient heterogeneous dual-encoder models using lightweight query encoders.
