论文标题
检测文本的语言
Detect Language of Transliterated Texts
论文作者
论文摘要
从其他语言到英语的非正式音译在社交媒体线程，即时消息传递和讨论论坛中很普遍。如果不识别这种音译文本的语言，那些不会说语言的用户无法使用翻译工具理解其内容。我们提出了一种语言识别（LID）系统，并采用特征提取方法，即使培训数据和计算资源有限，它也可以很好地检测到音译文本的语言。我们将这些单词示为语音音节，并使用简单的长期短期内存（LSTM）网络体系结构来检测音译文本的语言。通过密集的实验，我们表明，被音译单词作为语音音节的令牌化有效地代表了它们的因果声模式。因此，语音音节标记化使更简单的模型体系结构更容易学习识别任何语言的特征模式。
Informal transliteration from other languages to English is prevalent in social media threads, instant messaging, and discussion forums. Without identifying the language of such transliterated text, users who do not speak that language cannot understand its content using translation tools. We propose a Language Identification (LID) system, with an approach for feature extraction, which can detect the language of transliterated texts reasonably well even with limited training data and computational resources. We tokenize the words into phonetic syllables and use a simple Long Short-term Memory (LSTM) network architecture to detect the language of transliterated texts. With intensive experiments, we show that the tokenization of transliterated words as phonetic syllables effectively represents their causal sound patterns. Phonetic syllable tokenization, therefore, makes it easier for even simpler model architectures to learn the characteristic patterns to identify any language.
