论文标题
正弦网络的简单初始化和参数化通过其内核带宽
Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth
论文作者
论文摘要
已经提出了具有正弦激活的神经网络，以替代传统激活功能的网络。尽管他们的希望，尤其是对于学习隐式模型，但他们的训练行为尚未完全理解，导致许多经验设计选择没有很好的合理性。在这项工作中，我们首先提出了这种正弦神经网络的简化版本，该版本允许更轻松地实现和简单的理论分析。然后，我们从神经切线内核的角度分析了这些网络的行为，并证明其内核近似具有可调带宽的低通滤波器。最后，我们利用这些见解来告知正弦网络初始化，优化了它们针对每个任务中的每个任务的性能，包括学习隐式模型和求解微分方程。
Neural networks with sinusoidal activations have been proposed as an alternative to networks with traditional activation functions. Despite their promise, particularly for learning implicit models, their training behavior is not yet fully understood, leading to a number of empirical design choices that are not well justified. In this work, we first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth. Finally, we utilize these insights to inform the sinusoidal network initialization, optimizing their performance for each of a series of tasks, including learning implicit models and solving differential equations.
