论文标题
深度神经网络与一般激活功能的收敛和合并
Convergence of Deep Neural Networks with General Activation Functions and Pooling
论文作者
论文摘要
深度神经网络是代表高维复杂功能的强大系统，在深度学习中起着关键作用。深度神经网络的融合是建立深度学习数学基础的基本问题。在两项最近的研究（ARXIV：2107.12530，2109.13542）中，我们研究了深度relu网络和深卷卷神经网络的融合。仅研究了校正的线性单元（RELU）激活，并且不考虑重要的合并策略。在目前的工作中，我们研究了深神经网络的收敛性，因为深度倾向于无穷大，但对于另外两个重要的激活函数：泄漏的relu和sigmoid函数。合并也将研究。结果，我们证明了Arxiv中建立的足够条件：2107.12530，2109.13542仍然足以满足泄漏的Relu网络。对于诸如Sigmoid函数之类的收缩激活功能，我们建立了较弱的足够条件，可以使深神经网络的均匀收敛。
Deep neural networks, as a powerful system to represent high dimensional complex functions, play a key role in deep learning. Convergence of deep neural networks is a fundamental issue in building the mathematical foundation for deep learning. We investigated the convergence of deep ReLU networks and deep convolutional neural networks in two recent researches (arXiv:2107.12530, 2109.13542). Only the Rectified Linear Unit (ReLU) activation was studied therein, and the important pooling strategy was not considered. In this current work, we study the convergence of deep neural networks as the depth tends to infinity for two other important activation functions: the leaky ReLU and the sigmoid function. Pooling will also be studied. As a result, we prove that the sufficient condition established in arXiv:2107.12530, 2109.13542 is still sufficient for the leaky ReLU networks. For contractive activation functions such as the sigmoid function, we establish a weaker sufficient condition for uniform convergence of deep neural networks.
