论文标题
关于网络上的分布式随机二元优化算法的收敛性
On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network
论文作者
论文摘要
二重性优化已应用于多种机器学习模型，并且近年来已经开发了许多随机的二元优化算法。但是，大多数现有算法都将关注点限制在单机设置上，因此它们无法处理分布式数据。为了解决这个问题，在所有参与者组成网络并在该网络中执行点对点通信的设置，我们基于梯度跟踪通信机制和两个不同的梯度估计器开发了两个新颖的分散的随机二元优化算法。此外，我们通过新的理论分析策略确定了他们针对非convex-Strongle-Convex问题的收敛速率。据我们所知，这是实现这些理论结果的第一项工作。最后，我们将算法应用于实用的机器学习模型，实验结果证实了我们的算法的功效。
Bilevel optimization has been applied to a wide variety of machine learning models, and numerous stochastic bilevel optimization algorithms have been developed in recent years. However, most existing algorithms restrict their focus on the single-machine setting so that they are incapable of handling the distributed data. To address this issue, under the setting where all participants compose a network and perform peer-to-peer communication in this network, we developed two novel decentralized stochastic bilevel optimization algorithms based on the gradient tracking communication mechanism and two different gradient estimators. Additionally, we established their convergence rates for nonconvex-strongly-convex problems with novel theoretical analysis strategies. To our knowledge, this is the first work achieving these theoretical results. Finally, we applied our algorithms to practical machine learning models, and the experimental results confirmed the efficacy of our algorithms.
