论文标题
通过logit归一化来缓解神经网络过度自信
Mitigating Neural Network Overconfidence with Logit Normalization
论文作者
论文摘要
检测到分布输入对于在现实世界中安全部署机器学习模型至关重要。但是，众所周知，神经网络会遭受过度自信的问题，在该问题中，它们对分布内和分布的输入产生异常高的信心。在这项工作中，我们表明，可以通过对训练中的logits执行恒定的向量规范来表明，可以通过logit归一化（LogitNorm）来缓解此问题。我们的方法是通过分析的激励，即logit的规范在训练过程中不断增加，从而导致过度自信的产出。因此，LogitNorm背后的主要思想是将网络优化期间输出规范的影响解散。通过LogitNorm培训的神经网络在分布数据之间和分发数据之间产生高度可区分的置信度得分。广泛的实验证明了LogitNorm的优势，在常见基准上，平均FPR95最高可达42.30％。
Detecting out-of-distribution inputs is critical for safe deployment of machine learning models in the real world. However, neural networks are known to suffer from the overconfidence issue, where they produce abnormally high confidence for both in- and out-of-distribution inputs. In this work, we show that this issue can be mitigated through Logit Normalization (LogitNorm) -- a simple fix to the cross-entropy loss -- by enforcing a constant vector norm on the logits in training. Our method is motivated by the analysis that the norm of the logit keeps increasing during training, leading to overconfident output. Our key idea behind LogitNorm is thus to decouple the influence of output's norm during network optimization. Trained with LogitNorm, neural networks produce highly distinguishable confidence scores between in- and out-of-distribution data. Extensive experiments demonstrate the superiority of LogitNorm, reducing the average FPR95 by up to 42.30% on common benchmarks.
