# Weight Initialization Weight initialization sets initial values for neural network parameters before training, influencing convergence speed and gradient flow. Methods like Xavier and He initialization prevent vanishing or exploding gradients. Weight initialization, also called parameter initialization, is the pre-training step in deep learning that assigns initial values to a neural network's trainable parameters, such as weights and biases. These parameters are modified during training, and the choice of initialization method affects convergence speed, the scale of activations, the scale of gradient signals during backpropagation, and the final model quality. Proper initialization is necessary to avoid issues like vanishing or exploding gradients and activation function saturation. In a multilayer perceptron (MLP), each layer contains a weight matrix and a bias vector. The initialization method determines the starting values for these parameters. While the article focuses on weights, biases are also trainable and are typically initialized separately. For convolutional neural networks (CNNs), the same principles apply to kernels and biases. ## Constant Initialization The simplest form is zero initialization, where all weights and biases are set to zero. This is rarely used for weights because it leads to symmetry: all neurons in a layer compute identical outputs and gradients, so they learn the same features. Zero initialization is commonly used for biases, but weights require random values to break symmetry. Biases are usually initialized to zero, but there are exceptions. In multiplicative units like the forget gate of an LSTM, setting the bias to 1 can improve gradient flow through the gate. For ReLU neurons, a small positive bias (e.g., 0.1) can prevent the dying ReLU problem by ensuring nonzero gradients at initialization. Recurrent neural networks (RNNs) often use bounded activation functions like sigmoid or tanh to avoid exploding values. Some research, such as that by Le, Jaitly, and Hinton in 2015, suggested initializing recurrent weights to the identity matrix and biases to zero, which resembles residual connections and LSTMs without a forget gate. ## Random Initialization Random initialization samples weights from a normal or uniform distribution, typically independently. The choice of distribution and its variance is critical for maintaining signal propagation through the network. ### LeCun Initialization LeCun initialization, popularized by Yann LeCun and colleagues in 1998, aims to preserve activation variance during the forward pass. It samples each weight from a distribution with mean zero and variance 1/n_in, where n_in is the number of inputs to the layer (fan-in). For a uniform distribution, this corresponds to U(±√(3/n_in)). This method works well with activation functions like tanh but can be suboptimal for ReLU. ### Glorot Initialization Glorot initialization, also known as Xavier initialization, was proposed by Xavier Glorot and Yoshua Bengio. It balances two goals: preserving activation variance in the forward pass and preserving gradient variance in the backward pass. For uniform initialization, weights are sampled from U(±√(6/(n_in + n_out))), where n_out is the number of outputs (fan-out). This method is effective for sigmoid or tanh activations but may cause issues with ReLU due to its asymmetric nature. ### He Initialization He initialization, introduced by Kaiming He and colleagues in 2015, is designed for ReLU activations. It sets the variance to 2/n_in, accounting for the fact that ReLU zeroes out half of the activations. For uniform distribution, weights are sampled from U(±√(6/n_in)). This method helps prevent vanishing gradients in deep networks with ReLU. ## Variance Preservation and Gradient Flow The primary goal of initialization is to keep the variance of activations and gradients stable across layers. If weights are too small, activations shrink, leading to vanishing gradients. If too large, activations grow, causing exploding gradients. Glorot and He initialization provide a principled way to set variance based on fan-in and fan-out. For deep networks, the product of weight matrices across layers can cause exponential growth or decay. Proper initialization ensures that the initial signal propagates without extreme scaling, allowing effective training from the start. ## Impact on Training Dynamics Initialization affects not only convergence speed but also the quality of the final model. Poor initialization can lead to slow training, local minima, or failure to converge. For example, zero initialization causes symmetry, while overly large weights can saturate activation functions, leading to small gradients. In practice, initialization is often combined with other techniques like batch normalization, which can reduce sensitivity to initialization. However, good initialization remains important, especially for very deep networks or when normalization is not used. ## Specialized Architectures Different architectures require tailored initialization strategies. For CNNs, kernels are initialized similarly to MLP weights, but the fan-in and fan-out are computed based on the kernel size and number of channels. For transformers, which use [layer normalization](https://www.wikiprompt.org/wiki/layer-normalization) and residual connections, initialization often uses smaller standard deviations, such as 0.02, as seen in models like [BERT](https://www.wikiprompt.org/wiki/bert). For [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s, initialization is a critical factor in training stability. Many modern models use variations of He or Glorot initialization, sometimes with additional scaling based on model depth. ## Practical Considerations In most deep learning frameworks, default initialization methods are provided, but practitioners can customize them. For example, [PyTorch](https://www.wikiprompt.org/wiki/pytorch) and [TensorFlow](https://www.wikiprompt.org/wiki/tensorflow) offer functions for uniform, normal, and orthogonal initialization. Orthogonal initialization, which sets weight matrices to be orthogonal, is sometimes used for RNNs to preserve gradient norms. When training very deep networks, it is common to use residual connections and careful initialization to avoid degradation. The choice of initialization should be aligned with the activation function and the network architecture. ## Historical Context The study of weight initialization dates back to early work on neural networks. LeCun's 1998 paper on gradient-based learning applied to document recognition introduced variance-preserving initialization. Glorot and Bengio's 2010 paper "Understanding the difficulty of training deep feedforward neural networks" highlighted the importance of initialization for deep networks. He et al.'s 2015 paper on deep residual learning further refined methods for ReLU networks. These foundational works have shaped modern practice, and initialization remains an active area of research, especially for extremely deep or large-scale models. ## Conclusion Weight initialization is a fundamental step in training neural networks. Methods like zero, random, LeCun, Glorot, and He initialization each have specific use cases and trade-offs. The choice of initialization can significantly impact training dynamics, convergence, and final performance. Understanding these methods is essential for practitioners working with [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) models, from simple MLPs to complex [transformer](https://www.wikiprompt.org/wiki/transformer) architectures. --- Source: https://www.wikiprompt.org/wiki/weight-initialization License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-07T02:32:29.037541+00:00