Normalization Layers

Normalization Layers#

Normalization techniques for stabilizing training and improving convergence. Includes batch normalization variants (0D-3D), layer normalization, RMS normalization, group normalization, and weight standardization. Each normalization strategy addresses different aspects of internal covariate shift and gradient flow.

BatchNorm0d

0-D batch normalization.

BatchNorm1d

1-D batch normalization.

BatchNorm2d

2-D batch normalization.

BatchNorm3d

3-D batch normalization.

LayerNorm

Layer normalization layer [1]_.

RMSNorm

Root Mean Square Layer Normalization [1]_.

GroupNorm

Group Normalization layer [1]_.

weight_standardization

Scaled Weight Standardization.