Recurrent neural network
From Wikipedia, the free encyclopedia
 |
This article or section is in need of attention from an expert on the subject. WikiProject Computer science or the Computer science Portal may be able to help recruit one. |
A recurrent neural network (RNN) is a class of neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior.
Recurrent neural networks must be approached differently from feedforward neural networks, both when analyzing their behavior and training them. Recurrent neural networks can also behave chaotically. Usually, dynamical systems theory is used to model and analyze them. While a feedforward network propagates data linearly from input to output, recurrent networks (RN) also propagate data from later processing stages to earlier stages.
Contents |
[edit] Architectures
Some of the most common recurrent neural network architectures are described here. The Elman and Jordan networks are also known as "simple recurrent networks" (SRN).
[edit] Elman network
This variation on the multilayer perceptron was invented by Jeff Elman. A three-layer network is used, with the addition of a set of "context units" in the input layer. There are connections from the middle (hidden) layer to these context units fixed with a weight of one[1]. At each time step, the input is propagated in a standard feed-forward fashion, and then a learning rule is applied. The fixed back connections result in the context units always maintaining a copy of the previous values of the hidden units (since they propagate over the connections before the learning rule is applied). Thus the network can maintain a sort of state, allowing it to perform such tasks as sequence-prediction that are beyond the power of a standard multilayer perceptron.
[edit] Jordan network
This network architecture is similar to the Elman network. The context units are however fed from the output layer instead of the hidden layer.
[edit] MMC network
See [2]
[edit] Hopfield network
The Hopfield network is a recurrent neural network in which all connections are symmetric. Invented by John Hopfield in 1982, this network guarantees that its dynamics will converge. If the connections are trained using Hebbian learning then the Hopfield network can perform as robust content-addressable memory, resistant to connection alteration.
[edit] Echo state network
The echo state network (ESN) is a recurrent neural network with a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change and be trained. ESN are good to (re)produce temporal patterns.
[edit] Long short term memory network
The Long short term memory (LSTM) is an artificial neural net structure that unlike traditional RNNs doesn't have the problem of vanishing gradients. It can therefore use long delays and can handle signals that have a mix of low and high frequency components.
[edit] RNN with parametric bias
In this setup the recurrent neural network with parametric bias (RNNPB) is trained to reproduce a sequence with a constant bias input. The network is capable of learning different sequences with different parametric biases. With a trained network is also possible to find the associated parameter for an observed sequence. The sequence is backpropagated through the network to recover the bias which would produce the given sequence.
[edit] Continuous-time RNN
(CTRNN)
[edit] Hierarchical RNN
Here RNN are sparsely connected together through bottlenecks with the idea to isolate different hierarchical functions to different parts of the composite network. [3] [4] [5]
[edit] Recurrent Multilayer Perceptron
(RMLP)
[edit] Pollack’s Sequential Cascaded Networks
[edit] Training
Training in recurrent neural networks is generally very slow.
[edit] Backpropagation through time (BPTT)
In this approach the simple recurrent network is unfolded in time for some iterations and then trained through backpropagation as the feed forward network.
[edit] Real-time recurrent learning (RTRL)
Unlike BPTT this algotirhm is local in time but not local in space [6]
[edit] Genetic algorithms
Since RNN learning is very slow, genetic algorithms is a feasible alternative for weight optimization, especially in unstructured networks.
[edit] References
- ^ Neural Networks as Cybernetic Systems 2nd and revised edition, Holk Cruse [1]
- ^ http://www.tech.plym.ac.uk/socce/ncpw9/Kuehn.pdf
- ^ Dynamic Representation of Movement Primitives in an Evolved Recurrent Neural Network
- ^ doi:10.1016/j.neunet.2004.08.005
- ^ http://adb.sagepub.com/cgi/reprint/13/3/211.pdf
- ^ Neural and Adaptive Systems: Fundamentals through Simulation. J.C. Principe, N.R. Euliano, W.C. Lefebvre
- Mandic, D. & Chambers, J. (2001). Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. Wiley.
- Elman, J.L. (1990). "Finding Structure in Time". Cognitive Science 14: 179-211. doi:.
