论文标题
基于自动编码器的初始化，在复发性神经网络中优化短期内存优化
Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization
论文作者
论文摘要
由于梯度消失，培训RNN以学习长期依赖性很难学习。我们使用线性自动编码器进行序列探索基于明确记忆的替代解决方案，该解决方案允许最大化短期内存，并且可以通过封闭形式的解决方案解决而无需反向传播。我们介绍了一个初始化架构，该模式可以预见复发性神经网络的权重，以近似输入序列的线性自动编码器，并显示这种预处理如何更好地支持用长序列解决硬分类任务。我们在顺序和排列的MNIST上测试我们的方法。我们表明，所提出的方法在长序列中实现了更低的重建误差，并且在固定阶段获得了更好的梯度传播。
Training RNNs to learn long-term dependencies is difficult due to vanishing gradients. We explore an alternative solution based on explicit memorization using linear autoencoders for sequences, which allows to maximize the short-term memory and that can be solved with a closed-form solution without backpropagation. We introduce an initialization schema that pretrains the weights of a recurrent neural network to approximate the linear autoencoder of the input sequences and we show how such pretraining can better support solving hard classification tasks with long sequences. We test our approach on sequential and permuted MNIST. We show that the proposed approach achieves a much lower reconstruction error for long sequences and a better gradient propagation during the finetuning phase.
