论文标题
使用Hessian Trace的随机估计器正规化深层神经网络
Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace
论文作者
论文摘要
在本文中，我们通过惩罚Hessian的痕迹来开发一种新型的正则化方法，以用于深度神经网络。该正常化程序的动机是由概括误差的最新保证限制。我们解释了它在找到平坦的最小值并避免在动态系统中避免Lyapunov稳定性方面的好处。我们采用Hutchinson方法作为矩阵轨迹的经典无偏估计量，并使用辍学方案进一步加速其计算。实验表明，我们的方法的表现优于现有的正规化器和数据增强方法，例如Jacobian，置信惩罚，标签平滑，切口和混合。
In this paper, we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. We explain its benefits in finding flat minima and avoiding Lyapunov stability in dynamical systems. We adopt the Hutchinson method as a classical unbiased estimator for the trace of a matrix and further accelerate its calculation using a dropout scheme. Experiments demonstrate that our method outperforms existing regularizers and data augmentation methods, such as Jacobian, Confidence Penalty, Label Smoothing, Cutout, and Mixup.
