1969, as the founders of artificial neural networks, Marin M. Insky and Pappert jointly published the book Perceptron, which proved that the simple linear perceptron has limited functions and cannot solve basic problems such as XOR, and they are also pessimistic about multi-layer networks. These debates have dealt a heavy blow to the research of neural networks, and many scientists have left this field one after another, and the research of neural networks has gone to the low tide of 10 years.
Therefore, the historical significance of BP algorithm: it clearly denies the wrong views of Minsky and others, and is of decisive significance to the second climax of neural network.
This means the position and significance of BP algorithm in the field of neural network.
BP algorithm is by far the most successful neural network learning algorithm. When neural networks are used in practical tasks, most of them are trained by BP algorithm [2], including convolutional neural networks (CNN) under the popular concept of recent deep learning.
BP neural network is such a neural network model, which consists of an input layer, an output layer and one or more hidden layers. Its activation function adopts sigmoid function, and BP algorithm is used to train multilayer feedforward neural network.
The full name of BP algorithm is called error back propagation. The basic idea of the algorithm is that in the feedforward network described by 2. 1, the input signal is input through the input layer, and the output is calculated through the hidden layer, and the output value is compared with the marked value. If there are any errors, the errors will propagate from the output layer to the input layer in the opposite direction. In this process, the gradient descent algorithm is used to adjust the weight of neurons.
The core mathematical tool in BP algorithm is the chain derivative rule of calculus.
The first disadvantage of BP algorithm is the local minimum problem.
BP algorithm is essentially gradient descent, and its objective function to be optimized is very complex, which leads to the inefficiency of BP algorithm.
[1], Philosophical Thinking of BP Algorithm, Cheng Sumei, Hao Zhonghua.
[2], "Machine Learning", by Zhou Zhihua.
[3], Notes on Deep Learning Papers (Derivation and Implementation of CNN Convolutional Neural Network
First release
20 16-06-04 was greatly revised, the derivation process was improved, and the title of the article was revised.
20 16-07-23 modified a formula derivation error and an expression error.