论文标题
持续归一化：重新思考在线持续学习的批处理
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning
论文作者
论文摘要
现有的持续学习方法使用批处理（BN）来促进培训并改善跨任务的概括。但是，持续学习数据的非i.i.i.d和非平稳性质，尤其是在在线环境中，扩大了BN的培训和测试之间的差异，并阻碍了较旧任务的执行。在这项工作中，我们研究了BN在在线持续学习中的交叉任务归一化效应，BN使用偏向当前任务的时刻将测试数据归一化，从而导致更高的灾难性遗忘。这种限制促使我们提出一种简单而有效的方法，我们称之为连续归一化（CN），以促进类似于BN的训练，同时减轻其负面影响。关于不同持续学习算法和在线方案的广泛实验表明，CN是BN的直接替代者，可以提供大量的性能改进。我们的实现可在\ url {https://github.com/phquang/continual-normalization}中获得。
Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation is available at \url{https://github.com/phquang/Continual-Normalization}.
