Draft:Layer normalization
Submission declined on 21 September 2023 by HitroMilanese (talk). This submission is not adequately supported by reliable sources. Reliable sources are required so that information can be verified. If you need help with referencing, please see Referencing for beginners and Citing sources.
Where to get help
How to improve a draft
You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Editor resources
|
The page Layer normalization in the mainspace is currently a redirect to Normalization (machine learning). This is a draft article. It is a work in progress open to editing by anyone. Please ensure core content policies are met before publishing it as a live Wikipedia article. Find sources: Google (books · news · scholar · free images · WP refs) · FENS · JSTOR · TWL Last edited by Jay (talk | contribs) 3 months ago. (Update)
Finished drafting? or |
Part of a series on |
Machine learning and data mining |
---|
Layer normalization (also known as layer norm) helps make training of artificial neural networks faster and more stable. It is presented as an alternative to batch normalization. In layer norm, neural networks compute a mean and variance using all the summed inputs to the neuron in a single training step. This information is then used to normalize the summed input to that neuron for each and every training step. Layer normalization is commonly found in recurrent neural networks and in the Transformer (machine learning model).[1][2]
Introduction and Motivation
[edit]Complex artificial neural networks trained with stochastic gradient descent often require multiple days of training and large amounts of computing power[citation needed]. One approach to reducing training time is to standardize the neuron inputs performed during feedforward to make the learning easier. Batch normalization achieves this by standardizing each neuron's input using the data's mean and standard deviation. However, batch normalization requires storing the mean and standard deviation for each layer. This is not possible with an RNN because RNNs typically vary in sequence length. Layer normalization mitigates this by directly producing normalization statistics from the layer inputs so that the statistics do not need to be stored between training cases.
Procedure
[edit]The outputs of hidden layers in a deep neural network tend to be highly correlated with one another. This means the mean and variance of the summed layer inputs can be fixed as constants. The computation for layer norm statistics for all neurons in the same hidden layer is:[2][3]
Assume that is the number of neurons in the hidden layer.
The mean is:
The variance is:
References
[edit]- ^ "Papers with Code - Layer Normalization Explained". paperswithcode.com. Retrieved 2023-08-29.
- ^ a b Lei Ba, Jimmy (21 Jul 2016). "Layer Normalization". arXiv:1607.06450 [stat.ML].
- ^ Xu, Jingjing (16 Nov 2019). "Understanding and Improving Layer Normalization". Advances in Neural Information Processing Systems. arXiv:1911.07013.