Weight initialization

In machine learning and deep learning, weight initialization describes the initial step in creating a neural network. A neural network contains trainable parameters that are modified during training. Before training, these need to be assigned initial values. This assignment step is weight initialization.

The choice of weight initialization method affects the speed of convergence, the scale of neural activation within the network, the scale of gradient signals during backpropagation, and the quality of the final model. Proper initialization is necessary for avoiding issues such as vanishing and exploding gradients, saturation of activation function, etc.

Constant initialization

We discuss the main methods of initialization in the context of a multilayer perceptron (MLP). Specific strategies for initializing other network architectures are discussed in later sections.

For an MLP, there are only two kinds of trainable parameters, called weights and biases. Each layer $l$ contains a weight matrix $W^{(l)}\in \mathbb {R} ^{n_{l-1}\times n_{l}}$and a bias vector $b^{(l)}\in \mathbb {R} ^{n_{l}}$, where $n_{l}$ is the number of neurons in that layer. A weight initialization method is an algorithm for setting the initial values for $W^{(l)},b^{(l)}$ for each layer $l$.

The simplest form is zero initialization:$$W^{(l)}=0,b^{(l)}=0$$Zero initialization is usually used for initializing biases, but it is not used for initializing weights, as it leads to symmetry in the network, causing all neurons to learn the same features.

Recurrent neural networks typically use activation functions with bounded range, such as sigmoid and tanh, since unbounded activation may cause exploding values. (Le, Jaitly, Hinton, 2015) suggested initializing weights in the recurrent parts of the network to identity and zero bias.

In most cases, the biases are initialized to zero, though some situations can use a nonzero initialization. For example, in multiplicative units, such as the forget gate of LSTM, the bias can be initialized to 1 to allow good gradient signal through the gate. For neurons with ReLU activation, one can initialize the bias to a small positive value like 0.1, so that the gradient is likely nonzero at initialization, avoiding the dying ReLU problem.

Random initialization

Random initialization means sampling the weights from a normal distribution or a uniform distribution, usually independently.

LeCun initialization

The uniform random initialization is typically by sampling each entry in $W^{(l)}$ from the uniform distribution ${\mathcal {U}}(\pm 1/{\sqrt {n_{l-1}}})$. It is designed to preserve the variance of neural activations during the forward pass.

This is sometimes called LeCun initialization, as it was popularized in (LeCun et al, 1998).

Glorot initialization

Glorot initialization (or Xavier initialization) was proposed by Xavier Glorot and Yoshua Bengio. It was designed as a compromise between two goals: to preserve activation variance during the forward pass and to preserve gradient variance during the backward pass.

keep the scale of gradients roughly the same in all layers.

For uniform initialization, it samples each entry in $W^{(l)}$ independently and identically from ${\mathcal {U}}(\pm {\sqrt {6/(n_{l+1}+n_{l-1})}})$. In the context, $n_{l-1}$ is also called the "fan-in", and $n_{l+1}$ the "fan-out".

He initialization

As Glorot initialization performs poorly for ReLU activation, He initialization (or Kaiming initialization) was proposed by Kaiming He et al. for networks with ReLU activation. It samples each entry in $W^{(l)}$ from ${\mathcal {N}}(0,{\sqrt {2/n_{l-1}}})$.

Orthogonal initialization

(Saxe et al 2013) proposed orthogonal initialization: initializing weight matrices as random semi-orthogonal matrices, multiplied by a factor that depends on the activation function of the layer. It was designed so that if one initializes a deep linear network this way, then its training time until convergence is independent of depth.

Related to this approach, unitary initialization proposes to parameterize the weight matrices to be unitary matrices, with the result that at initialization they are random unitary matrices (and throughout training, they remain unitary). This is found to improve long-sequence modelling in LSTM.

Orthogonal initialization has been generalized to layer-sequential unit-variance (LSUV) initialization. It is a data-dependent initialization method, and can be used in convolutional neural networks. It first initializes weights of each convolution or fully connected layer with orthonormal matrices. Then, proceeding from the first to the final layer, it runs a forward pass on a random minibatch, and divides the layer's weights by the standard deviation of its output, so that its output has variance approximately 1.

Other random initializations

Fixup initialization is designed specifically for networks with residual connections and without batch normalization, as follows:

1. Initialize the classification layer and the last layer of each residual branch to 0.
2. Initialize every other layer using a standard method (e.g., He et al. (2015)), and scale only the weight layers inside residual branches by $L^{-{\frac {1}{2m-2}}}$.
3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer.

Instead of initializing all weights with random values on the order of $O(1/{\sqrt {n}})$, sparse initialization initialized only a small subset of the weights with larger random values, and the other weights zero, so that the total variance is still on the order of $O(1)$.

Random walk initialization was designed for MLP so that during backpropagation, the L2 norm of gradient at each layer performs an unbiased random walk as one moves from the last layer to the first. The algorithm.

Miscellaneous

For hyperbolic tangent activation function, a particular scaling is sometimes used: $1.7159\tanh(2x/3)$. This was sometimes called "LeCun's tanh". It was designed so that it maps the interval $[-1,+1]$ to itself, thus ensuring that the overall gain is around 1 in "normal operating conditions", and that $|f''(x)|$ is at maximum when $x=-1,+1$, which improves convergence at the end of training.

In self-normalizing neural networks, the SELU activation function $\mathrm {SELU} (x)=\lambda {\begin{cases}x&{\text{if }}x>0\\\alpha e^{x}-\alpha &{\text{if }}x\leq 0\end{cases}}$ with parameters $\lambda \approx 1.0507,\alpha \approx 1.6733$ makes it such that the mean and variance of the output of each layer has $(0,1)$ as an attracting fixed-point. This makes initialization less important, though they recommend initializing weights randomly with variance $1/n_{l-1}$.

History

Random weight initialization was used since Frank Rosenblatt's perceptrons. An early work that described weight initialization specifically was (LeCun et al, 1998).

Before the 2010s era of deep learning, it was common to initialize models by "pre-training" using an unsupervised learning algorithm that is not backpropagation, as it was difficult to directly train deep neural networks by backpropagation. For example, the deep belief network was trained by contrastive divergence is applied to each network in turn, starting from the "lowest" pair of layers.

(Martens, 2010) proposed a quasi-Newton method to directly train deep networks. The work generated considerable excitement that initializing networks without pre-training phase was possible. However, a 2013 paper demonstrated that with well-chosen hyperparameters, momentum gradient descent with weight initialization was sufficient for training neural networks, a combination that is still in use as of 2024.

Since then, the impact of initialization on tuning the variance has become less important, with methods developed to automatically tune variance, like batch normalization tuning the variance of the forward pass, and momentum-based optimizers tuning the variance of the backward pass.

See also

• Backpropagation
• Gradient descent
• Vanishing gradient problem

Further reading

• Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). "8.4 Parameter Initialization Strategies". Deep learning. Adaptive computation and machine learning. Cambridge, Mass: The MIT press. ISBN 978-0-262-03561-3.

• Narkhede, Meenal V.; Bartakke, Prashant P.; Sutaone, Mukul S. (June 28, 2021). "A review on weight initialization strategies for neural networks". Artificial Intelligence Review. 55 (1). Springer Science and Business Media LLC: 291–322. doi:10.1007/s10462-021-10033-z. ISSN 0269-2821.