论文标题
语料库的相似性措施在各种语言中保持强大
Corpus Similarity Measures Remain Robust Across Diverse Languages
论文作者
论文摘要
本文使用寄存器预测任务进行了39种语言的基于频率语料库相似性的实验。目的是量化（i）不同语料库与同一语言和（ii）单个语音的同质性之间的距离。这两个目标对于衡量基于语料库的语言分析如何从一个数据集概括到另一个数据集都至关重要。问题在于，以前的工作集中在印欧语上，提出了一个问题，即这些措施是否能够在各种语言上提供强大的概括。本文使用寄存器预测任务来评估跨39种语言的竞争措施：他们能够区分代表不同生产环境的语料库？每个实验都将单个语言的三个语料库与所有语言共享的三个数字寄存器进行比较：社交媒体，网页和Wikipedia。结果表明，语料库相似性的衡量标准在不同语言家族，写作系统和形态类型中保留了其有效性。此外，当对室外语料库，应用于低资源语言以及应用于不同的寄存器集时，这些措施仍然很健壮。鉴于我们需要在可用于分析的迅速增加的语料库中进行概括，因此这些发现很重要。
This paper experiments with frequency-based corpus similarity measures across 39 languages using a register prediction task. The goal is to quantify (i) the distance between different corpora from the same language and (ii) the homogeneity of individual corpora. Both of these goals are essential for measuring how well corpus-based linguistic analysis generalizes from one dataset to another. The problem is that previous work has focused on Indo-European languages, raising the question of whether these measures are able to provide robust generalizations across diverse languages. This paper uses a register prediction task to evaluate competing measures across 39 languages: how well are they able to distinguish between corpora representing different contexts of production? Each experiment compares three corpora from a single language, with the same three digital registers shared across all languages: social media, web pages, and Wikipedia. Results show that measures of corpus similarity retain their validity across different language families, writing systems, and types of morphology. Further, the measures remain robust when evaluated on out-of-domain corpora, when applied to low-resource languages, and when applied to different sets of registers. These findings are significant given our need to make generalizations across the rapidly increasing number of corpora available for analysis.
