Layer normalization (LayerNorm) and batch normalization (BatchNorm) are both normalization techniques that stabilize deep neural network training by keeping activation distributions consistent across layers. The key difference lies in whether the mean and variance are computed across multiple samples in a mini-batch or across all features within a single sample—a distinction that directly explains why transformers rely on LayerNorm.

Summary

Both batch normalization and layer normalization stabilize training by adjusting values toward a mean of 0 and a variance of 1, but they aggregate along different axes. Batch normalization groups the same feature across multiple samples in a mini-batch, whereas layer normalization groups all features within a single sample. Because layer normalization does not depend on mini-batch size or sequence length and performs the exact same computation during training and inference, it is the standard choice for transformers.

Why do neural networks use layer normalization and batch normalization?

One-line answer: To prevent training instability caused by shifting value distributions from layer to layer, normalization rescales values toward mean 0 and variance 1 and then applies learnable scale (γ) and shift (β) parameters.

As inputs pass through multiple layers in a deep network, the distribution of values entering each layer shifts whenever the preceding weights are updated. When values grow or shrink uncontrollably across layers, training becomes unstable.

Normalization computes the mean and variance of values at a layer, adjusts them toward a mean of 0 and a variance of 1, and then applies learnable scale (γ) and shift (β) parameters so the network retains its expressive power. Depending on which axis is grouped to compute the mean and variance, the approach is divided into batch normalization (BatchNorm) and layer normalization (LayerNorm).

What does batch normalization group together?

One-line answer: In a grid where rows are samples and columns are features, batch normalization (BatchNorm) takes a single column and computes the mean and variance of that feature across the mini-batch.

In the Batch Normalization paper by Ioffe and Szegedy, each feature is normalized using the mean and variance computed over the mini-batch, followed by a learnable scale (γ) and shift (β). Visualizing a grid where each row is a sample and each column is a feature, BatchNorm groups a vertical column—the values of one feature across all samples in the mini-batch.

Because BatchNorm relies on mini-batch statistics, two practical properties follow:

  • Moving averages at inference: At inference time, instead of computing statistics from the current batch, it uses population statistics (moving averages collected during training).
  • Sensitivity to batch size: Because statistics are computed across the mini-batch, small batches make the batch statistics noisy.

What does layer normalization group together?

One-line answer: Layer normalization (LayerNorm) takes a single row and computes the mean and variance over all features of a single sample, so it does not depend on the mini-batch size.

In the Layer Normalization paper by Ba, Kiros, and Hinton, the mean and variance are computed over all the features (summed inputs) of a single training case, followed by a learnable scale (γ) and shift (β). On the same sample-by-feature grid, LayerNorm groups a single horizontal row.

Because the statistics come entirely from a single training case, LayerNorm does not depend on the mini-batch size and performs the exact same computation at training and inference time. This makes it straightforward to apply to recurrent and sequence models.

AspectBatch Normalization (BatchNorm)Layer Normalization (LayerNorm)
Grouping axis (rows = samples, columns = features)One column (a single feature across the mini-batch)One row (all features within a single sample)
Mini-batch size dependenceDepends on batch size (small batches make statistics noisy)Does not depend on batch size
Training vs. inference computationUses moving averages collected during training at inferencePerforms the exact same computation at training and inference time

Diagram comparing which axes batch normalization (column) and layer normalization (row) aggregate on a sample-by-feature grid

Why do transformers use LayerNorm?

One-line answer: Because transformers process sequences of different lengths and generate tokens one at a time at inference, LayerNorm normalizes each token’s feature vector by itself so the same calculation applies regardless of batch size or sequence length.

In sequence modeling, input sentences vary in length, and autoregressive inference generates one token at a time. Because LayerNorm normalizes each token’s feature vector independently, the exact same calculation applies whether the model is processing a batch of variable-length sequences during training or generating a single token at inference.

For how residual connections and LayerNorm fit into the overall transformer block alongside attention and feed-forward layers, see Transformer architecture explained simply: Self-Attention, FFN, and positional information (Korean).

Two common placement and simplification variants related to LayerNorm are:

  • Pre-LN vs. Post-LN: Refers to whether LayerNorm is placed before or after the sublayer inside the residual block.
  • RMSNorm: A simplification that normalizes by the root mean square without subtracting the mean.

What are the frequently asked questions?

One-line answer: Here are concise answers on whether CNNs still use BatchNorm, how normalization relates to overfitting, and how it is ordered with activation functions.

Do CNNs still use BatchNorm?

Yes. BatchNorm is still commonly used in many convolutional networks.

Does normalization prevent overfitting?

Normalization’s direct goal is training stability rather than preventing overfitting; for causes and remedies of overfitting, see What is overfitting? How to fix it when training loss falls but validation loss rises.

In what order is normalization placed with activation functions?

Whether normalization is placed before or after the activation function depends on the network architecture; for an overview of activation functions, see What is an activation function? ReLU vs GELU vs sigmoid, and how to choose.

One-line answer: These related posts cover transformer block architecture, overfitting, backpropagation, activation functions, and optimizers.

What are the sources?

One-line answer: This post references the original papers that introduced Batch Normalization and Layer Normalization.