论文标题
随机零阶功能约束优化：Oracle复杂性和应用
Stochastic Zeroth-order Functional Constrained Optimization: Oracle Complexity and Applications
论文作者
论文摘要
在机器学习应用程序中经常出现，在功能约束的随机优化问题上，目的函数和约束功能都无法分析。在这项工作中，假设我们只能访问目标和约束函数的嘈杂评估，我们建议和分析随机零阶算法，以解决上述随机优化问题。当功能的域为$ \ mathbb {r}^n $，假设存在$ m $约束函数时，我们建立了$ \ mathcal {o}（（（m+1）n/ε^2）$和$ \ \ \ \ \ \ m narcal {o}（（m+1）N/+1）N/+$^3）$的oracle复杂性，并且在$ shigncal $ a $ a $ a $ nonn cons $ nonn convex cons。适当定义的指标所需的解决方案的准确性。据我们所知，已建立的甲骨文复杂性是文献中第一个这样的结果，即功能受限的随机零阶优化问题。我们通过说明其在抽样算法和神经网络训练的高参数调整问题上的出色性能来证明我们的算法的适用性。
Functionally constrained stochastic optimization problems, where neither the objective function nor the constraint functions are analytically available, arise frequently in machine learning applications. In this work, assuming we only have access to the noisy evaluations of the objective and constraint functions, we propose and analyze stochastic zeroth-order algorithms for solving the above class of stochastic optimization problem. When the domain of the functions is $\mathbb{R}^n$, assuming there are $m$ constraint functions, we establish oracle complexities of order $\mathcal{O}((m+1)n/ε^2)$ and $\mathcal{O}((m+1)n/ε^3)$ respectively in the convex and nonconvex setting, where $ε$ represents the accuracy of the solutions required in appropriately defined metrics. The established oracle complexities are, to our knowledge, the first such results in the literature for functionally constrained stochastic zeroth-order optimization problems. We demonstrate the applicability of our algorithms by illustrating its superior performance on the problem of hyperparameter tuning for sampling algorithms and neural network training.
