论文标题
BERT无监督域适应的知识蒸馏
Knowledge Distillation for BERT Unsupervised Domain Adaptation
论文作者
论文摘要
预先训练的语言模型伯特（Bert）在一系列自然语言处理任务中带来了重大的性能改进。由于该模型经过大量各种主题的培训，因此它显示出在培训中的数据分布（源数据）和测试（目标数据）在共享相似性的同时不同的域名转移问题的强劲性能。尽管与以前的模型相比，它的改进很大，但由于域的变化，它仍然受到性能降低的损害。为了减轻此类问题，我们提出了一种简单但有效的无监督域适应方法，对抗性适应与蒸馏（AAD），该方法结合了对抗性歧视域适应（ADDA）框架与知识蒸馏。我们在30个域对上的跨域情绪分类任务中评估了我们的方法，从而在文本情感分类中推进了无监督域适应的最先进性能。
A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. Since the model is trained on a large corpus of diverse topics, it shows robust performance for domain shift problems in which data distributions at training (source data) and testing (target data) differ while sharing similarities. Despite its great improvements compared to previous models, it still suffers from performance degradation due to domain shifts. To mitigate such problems, we propose a simple but effective unsupervised domain adaptation method, adversarial adaptation with distillation (AAD), which combines the adversarial discriminative domain adaptation (ADDA) framework with knowledge distillation. We evaluate our approach in the task of cross-domain sentiment classification on 30 domain pairs, advancing the state-of-the-art performance for unsupervised domain adaptation in text sentiment classification.
