论文标题
评估爱沙尼亚的多语言伯特
Evaluating Multilingual BERT for Estonian
论文作者
论文摘要
最近，诸如BERT之类的大型预训练的语言模型已经在许多自然语言处理任务中达到了最先进的表现，但是对于包括爱沙尼亚语的许多语言，尚无伯特，伯特模型。但是，存在几种多语言BERT模型，可以同时处理多种语言，并且也接受了爱沙尼亚数据的培训。在本文中，我们评估了多种多语言模型 - 多语言BERT，多语言蒸馏Bert，XLM和XLM-Roberta-在包括POS和形态标记，NER和文本分类在内的多个NLP任务上。我们的目的是在这些任务的这些多语言BERT模型与现有的基线神经模型之间进行比较。我们的结果表明，多语言BERT模型可以在不同的爱沙尼亚NLP任务上很好地概括为POS和形态标记和文本分类的所有基本模型，并且与NER的最佳基线达到了可比的水平，而XLM-Roberta与其他多语种模型相比，XLM-Roberta可获得最高的结果。
Recently, large pre-trained language models, such as BERT, have reached state-of-the-art performance in many natural language processing tasks, but for many languages, including Estonian, BERT models are not yet available. However, there exist several multilingual BERT models that can handle multiple languages simultaneously and that have been trained also on Estonian data. In this paper, we evaluate four multilingual models -- multilingual BERT, multilingual distilled BERT, XLM and XLM-RoBERTa -- on several NLP tasks including POS and morphological tagging, NER and text classification. Our aim is to establish a comparison between these multilingual BERT models and the existing baseline neural models for these tasks. Our results show that multilingual BERT models can generalise well on different Estonian NLP tasks outperforming all baselines models for POS and morphological tagging and text classification, and reaching the comparable level with the best baseline for NER, with XLM-RoBERTa achieving the highest results compared with other multilingual models.
