论文标题
从披肩到理解：改造预先训练的蒙版语言模型再到预训练的机器读取器
From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader
论文作者
论文摘要
我们提出了预训练的机器读取器（PMR），这是一种新的方法，用于改造预先训练的蒙版语言模型（MLMS），以预先训练的机器阅读理解理解（MRC）模型，而无需获取标记的数据。 PMR可以解决现有MLM的模型预训练和下游微调之间的差异。为了构建所提出的PMR，我们通过使用Wikipedia超链接，构建了大量的通用和高质量的MRC风格训练数据，并设计了Wiki锚固提取任务来指导MRC风格的预训练。除了简单性外，PMR还有效地解决了提取任务，例如提取问题答案和命名实体识别。 PMR对现有方法显示出巨大的改进，尤其是在低资源场景中。当将MRC公式中的序列分类任务应用于序列分类任务时，PMR可以提取高质量的理由来解释分类过程，从而提供更大的预测性解释性。 PMR还具有作为解决MRC公式中各种提取和分类任务的统一模型的潜力。
We present Pre-trained Machine Reader (PMR), a novel method for retrofitting pre-trained masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data. PMR can resolve the discrepancy between model pre-training and downstream fine-tuning of existing MLMs. To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data by using Wikipedia hyperlinks and designed a Wiki Anchor Extraction task to guide the MRC-style pre-training. Apart from its simplicity, PMR effectively solves extraction tasks, such as Extractive Question Answering and Named Entity Recognition. PMR shows tremendous improvements over existing approaches, especially in low-resource scenarios. When applied to the sequence classification task in the MRC formulation, PMR enables the extraction of high-quality rationales to explain the classification process, thereby providing greater prediction explainability. PMR also has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation.
