论文标题
CNN比变形金刚更强大吗？
Can CNNs Be More Robust Than Transformers?
论文作者
论文摘要
视觉变形金刚最近的成功是在图像识别中挥舞着卷积神经网络（CNN）的长期优势。具体而言，就对分发样本的鲁棒性而言，最近的研究发现，无论训练设置如何，变压器本质上比CNN更强大。此外，人们认为，变形金刚的这种优越性应该在很大程度上被认为是他们的自我注意力型建筑本身。在本文中，我们通过密切研究变压器的设计来质疑这种信念。我们的发现导致了三种高效的体系结构设计，以提高鲁棒性，但很简单，可以在几行代码中实现，即a）修补输入图像，b）扩大内核大小，c）降低激活层和归一化层。将这些组件聚集在一起，我们能够建立纯CNN体系结构，而没有任何类似于注意力的操作，这些操作比变压器更强大，甚至更强大。我们希望这项工作可以帮助社区更好地了解强大的神经体系结构的设计。该代码可在https://github.com/ucsc-vlaa/robustcnn上公开获取。
The recent success of Vision Transformers is shaking the long dominance of Convolutional Neural Networks (CNNs) in image recognition for a decade. Specifically, in terms of robustness on out-of-distribution samples, recent research finds that Transformers are inherently more robust than CNNs, regardless of different training setups. Moreover, it is believed that such superiority of Transformers should largely be credited to their self-attention-like architectures per se. In this paper, we question that belief by closely examining the design of Transformers. Our findings lead to three highly effective architecture designs for boosting robustness, yet simple enough to be implemented in several lines of code, namely a) patchifying input images, b) enlarging kernel size, and c) reducing activation layers and normalization layers. Bringing these components together, we are able to build pure CNN architectures without any attention-like operations that are as robust as, or even more robust than, Transformers. We hope this work can help the community better understand the design of robust neural architectures. The code is publicly available at https://github.com/UCSC-VLAA/RobustCNN.
