How can long short-term memory (LSTM) networks overcome limitations in processing continuous input streams? This paper addresses the challenge of LSTM networks, which, without explicit resets, can experience state growth that causes the network to falter. The solution is a novel, adaptive 'forget gate' that allows LSTM cells to learn when to reset, freeing up internal resources. The research revisits benchmark problems where standard LSTM outperforms other recurrent neural network algorithms. However, these algorithms, including LSTM, struggle with continual versions of these problems. By implementing forget gates, LSTM can effectively solve these challenges in an elegant manner. This innovation enhances the capacity of LSTM networks to manage long, uninterrupted sequences, improving their applicability in real-world scenarios requiring continual learning. The concept provides a path forward for continual learning in neural networks.
Published in Neural Computation, this research fits within the journal's focus on neural networks and learning algorithms. The development of a novel forget gate for LSTM networks contributes to the ongoing advancements in recurrent neural networks, aligning with the journal's scope.