论文标题
快速准确的深层双向语言表示无监督学习
Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning
论文作者
论文摘要
即使Bert在各种监督的学习任务中取得了成功的绩效改进，但将BERT应用于无监督任务仍然存在一个限制，即需要重复推断计算上下文语言表示。为了解决限制，我们提出了一种新型的深层双向语言模型，称为基于变形金刚的文本自动编码器（T-TA）。 T-TA无需重复计算上下文语言表示，并具有像Bert这样的深层双向体系结构的好处。在CPU环境上的运行时间实验中，所提出的T-TA在重新疗程任务中的基于BERT模型的速度快六倍，而在语义相似性任务中，基于BERT的模型的执行速度快十二倍。此外，与上述任务相比，T-TA显示出比BERT的竞争性甚至更好的精度。
Even though BERT achieves successful performance improvements in various supervised learning tasks, applying BERT for unsupervised tasks still holds a limitation that it requires repetitive inference for computing contextual language representations. To resolve the limitation, we propose a novel deep bidirectional language model called Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and has benefits of the deep bidirectional architecture like BERT. In run-time experiments on CPU environments, the proposed T-TA performs over six times faster than the BERT-based model in the reranking task and twelve times faster in the semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks.
