论文标题
快速可区分的剪辑意识归一化和重新缩放
Fast Differentiable Clipping-Aware Normalization and Rescaling
论文作者
论文摘要
在\ Mathbb {r}^n $中将矢量$ \vecΔ\重新缩放为所需的长度是许多领域的常见操作，例如数据科学和机器学习。当添加$η\vecΔ$被添加到d $中的起点$ \ vec {x} \（其中$ d $是数据域，例如$ d = [0，1]^n $），所得的vector $ \ \ \ \ \ \ \ \ \ vec {v} = \ vec {vec {vec {x}} + n $ n p in $ n n in n in n in in in in in in in in in n in in in n n in n in n in n in n in in n in innun pred。为了强制执行扰动的向量$ v $在$ d $中，$ \ vec {v} $的值可以将其剪切到$ d $。然而，随后的元素剪辑到数据域确实会降低有效的扰动大小，从而干扰$ \vecδ$的重新缩放。最佳重新缩放$η$可以使用二进制搜索迭代估算剪辑后获得所需标准的扰动。但是，这种迭代方法是缓慢且不差异的。在这里，我们表明可以使用快速和可区分的算法在分析上发现最佳重新缩放。我们的算法适用于任何P-Norm，可用于在具有归一化扰动的输入上训练神经网络。我们为基于热切的Pytorch，Tensorflow，Jax和Numpy提供本机实现。
Rescaling a vector $\vecδ \in \mathbb{R}^n$ to a desired length is a common operation in many areas such as data science and machine learning. When the rescaled perturbation $η\vecδ$ is added to a starting point $\vec{x} \in D$ (where $D$ is the data domain, e.g. $D = [0, 1]^n$), the resulting vector $\vec{v} = \vec{x} + η\vecδ$ will in general not be in $D$. To enforce that the perturbed vector $v$ is in $D$, the values of $\vec{v}$ can be clipped to $D$. This subsequent element-wise clipping to the data domain does however reduce the effective perturbation size and thus interferes with the rescaling of $\vecδ$. The optimal rescaling $η$ to obtain a perturbation with the desired norm after the clipping can be iteratively approximated using a binary search. However, such an iterative approach is slow and non-differentiable. Here we show that the optimal rescaling can be found analytically using a fast and differentiable algorithm. Our algorithm works for any p-norm and can be used to train neural networks on inputs with normalized perturbations. We provide native implementations for PyTorch, TensorFlow, JAX, and NumPy based on EagerPy.
