论文标题
关于分段线性神经网络区域的数量
On the Number of Regions of Piecewise Linear Neural Networks
论文作者
论文摘要
许多前馈神经网络（NNS）产生连续和分段线性（CPWL）映射。具体而言，它们将输入域分配给映射为仿射的区域。这些所谓的线性区域的数量提供了一种自然度量，以表征CPWL NNS的表现力。在实践中，对该数量的确切确定通常是遥不可及的，并且针对包括Relu和Maxout NN在内的特定体系结构提出了界限。在这项工作中，我们将这些界限推广到具有任意和可能的多元CPWL激活函数的NN。我们首先在CPWL NN的线性区域的最大数量上提供上限和下限，鉴于其激活函数的深度，宽度和线性区域的数量。我们的结果依赖于凸形分区的组合结构，并确认深度的独特作用，该深度本身能够指数增加区域数量。然后，我们引入了一个互补的随机框架，以估计CPWL NN产生的线性区域的平均数量。在合理的假设下，沿任何一维路径的线性区域的预期密度都受深度，宽度和激活复杂度度量（最高缩放系数）的量的限制。这与三种表达能力来源产生了相同的作用：不再观察到深度的指数增长。
Many feedforward neural networks (NNs) generate continuous and piecewise-linear (CPWL) mappings. Specifically, they partition the input domain into regions on which the mapping is affine. The number of these so-called linear regions offers a natural metric to characterize the expressiveness of CPWL NNs. The precise determination of this quantity is often out of reach in practice, and bounds have been proposed for specific architectures, including for ReLU and Maxout NNs. In this work, we generalize these bounds to NNs with arbitrary and possibly multivariate CPWL activation functions. We first provide upper and lower bounds on the maximal number of linear regions of a CPWL NN given its depth, width, and the number of linear regions of its activation functions. Our results rely on the combinatorial structure of convex partitions and confirm the distinctive role of depth which, on its own, is able to exponentially increase the number of regions. We then introduce a complementary stochastic framework to estimate the average number of linear regions produced by a CPWL NN. Under reasonable assumptions, the expected density of linear regions along any 1D path is bounded by the product of depth, width, and a measure of activation complexity (up to a scaling factor). This yields an identical role to the three sources of expressiveness: no exponential growth with depth is observed anymore.
