论文标题
在线协方差矩阵估计随机梯度下降
Online Covariance Matrix Estimation in Stochastic Gradient Descent
论文作者
论文摘要
随机梯度下降（SGD）算法广泛用于参数估计，尤其是用于大型数据集和在线学习。尽管这种递归算法在计算和记忆效率方面很受欢迎，但很少研究量化解决方案的变异性和随机性。本文旨在在线环境中对基于SGD的估计进行统计推断。特别是，我们仅使用SGD的迭代元素为平均SGD迭代的协方差矩阵（ASGD）提出了一个完全在线估计器。我们正式建立了我们的在线估计器的一致性，并表明收敛速率与离线同行相当。基于ASGD的经典渐近正态性结果，我们为模型参数构建了渐近有效的置信区间。收到新的观察结果后，我们可以快速更新协方差矩阵估计和置信区间。这种方法适合在线环境，并充分利用SGD：计算和内存的效率。
The stochastic gradient descent (SGD) algorithm is widely used for parameter estimation, especially for huge data sets and online learning. While this recursive algorithm is popular for computation and memory efficiency, quantifying variability and randomness of the solutions has been rarely studied. This paper aims at conducting statistical inference of SGD-based estimates in an online setting. In particular, we propose a fully online estimator for the covariance matrix of averaged SGD iterates (ASGD) only using the iterates from SGD. We formally establish our online estimator's consistency and show that the convergence rate is comparable to offline counterparts. Based on the classic asymptotic normality results of ASGD, we construct asymptotically valid confidence intervals for model parameters. Upon receiving new observations, we can quickly update the covariance matrix estimate and the confidence intervals. This approach fits in an online setting and takes full advantage of SGD: efficiency in computation and memory.
