Abstract

Extending neural networks to handle sequential inputs of arbitrary lengths (such as time-series, statements in natural language, or mathematical expressions to evaluate) impedes the application of standard fully-connected or convolutional neural networks, which will typically require the input to be fixed in size. To alleviate the issue, recurrent neural networks introduce a learnable unit of computation, capable of processing inputs step-by-step in a length-independent manner. In fact, if feedforward neural networks were concerned with learning desirable functions that consume the input, recurrent neural networks may be seen as learning desirable programs that process the input.

In this lecture, I will introduce recurrent neural networks from first principles, and illustrate the many issues that arise with applying them naïvely. As a popular solution to those issues, the long short-term memory (LSTM) cell will be introduced, with a detailed intuitive and theoretical description of its mode of operation. This will be followed-up with strategies for tackling several kinds of sequential problems using LST Ms, and a survey of real-world applications of this model. Finally, a recently proposed non-recurrent model that shows promising results on sequential tasks will be outlined, to provide perspective for potential future trends in the field. No prior knowledge of neural networks or machine learning is needed, but an entry-level knowledge of supervised learning principles will be beneficial. Code examples that demonstrate how recurrent models can be constructed in only a few lines of Python will also be provided.