论文标题
被盗概率：神经语言模型的结构弱点
Stolen Probability: A Structural Weakness of Neural Language Models
论文作者
论文摘要
神经网络语言模型（NNLMS）通过将软性函数应用于通过在高维嵌入空间中使用所有单词向量的点乘积来形成的距离度量，从而生成概率分布。点产品距离度量构成了NNLM的电感偏置的一部分。尽管NNLMS通过这种电感偏差很好地优化了，但我们表明，这导致了嵌入空间的亚最佳排序，在分配概率时，结构上使某些单词牺牲了一些单词，从而使嵌入式空间的次优顺序。我们介绍了数值，理论和经验分析，表明嵌入空间中凸面内部的单词具有其概率，其概率受船体上单词的概率的限制。
Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses showing that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull.
