Variance Scaling and Orthogonal Initialization#
Deep and recurrent networks need weights whose scale matches the layer size: too small and signals shrink layer by layer, too large and they blow up. This tutorial covers the initializers that set that scale automatically.
Initializer |
Variance of each weight |
Default distribution |
|---|---|---|
|
|
|
|
|
normal / uniform |
|
|
normal / uniform |
|
|
normal / uniform |
|
rows or columns orthonormal, times |
|
|
orthogonal channel matrix at the kernel centre |
|
|
identity matrix times |
fan is fan_in, fan_out, or their average, chosen by mode. The uniform variants sample from \([-\sqrt{3\,\mathrm{var}}, \sqrt{3\,\mathrm{var}}]\), which has the same variance as the normal ones; truncated_normal samples a normal cut at two standard deviations and rescales it to the same variance.
1. Setup#
import brainstate
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np
from braintools import init
2. Which Axis Is Fan-In?#
braintools.init follows the same weight layout as brainstate layers (and JAX/Flax):
a dense weight has shape
(in, out), so the layer computesy = x @ W;a convolution kernel has shape
(*spatial, in, out), e.g.(3, 3, 16, 32)for a 3x3 kernel from 16 to 32 channels.
The fans are fan_in = in * receptive_field and fan_out = out * receptive_field, where receptive_field is the product of the spatial sizes. For a 1-D shape (n,) both fans are n. If your weights use another layout, for example PyTorch’s channels-first (out, in, kh, kw), select the axes with in_axis and out_axis.
rng = np.random.default_rng(0)
cases = [
("dense (in=400, out=100)", (400, 100), {}),
("dense (in=100, out=400)", (100, 400), {}),
("conv 3x3, 16 -> 64", (3, 3, 16, 64), {}),
("channels-first (64, 16, 3, 3)", (64, 16, 3, 3), dict(in_axis=1, out_axis=0)),
]
print(f"{'shape':32s} {'fan_in':>6s} {'Kaiming std':>11s} {'expected':>8s} {'Xavier std':>10s} {'expected':>8s}")
for name, shape, axes in cases:
kaiming = init.KaimingNormal(**axes)
xavier = init.XavierNormal(**axes)
fan_in, fan_out = kaiming._compute_fans(shape)
k_std = float(np.std(kaiming(shape, rng=rng)))
x_std = float(np.std(xavier(shape, rng=rng)))
print(f"{name:32s} {fan_in:6d} {k_std:11.4f} {np.sqrt(2 / fan_in):8.4f} "
f"{x_std:10.4f} {np.sqrt(2 / (fan_in + fan_out)):8.4f}")
shape fan_in Kaiming std expected Xavier std expected
dense (in=400, out=100) 400 0.0708 0.0707 0.0631 0.0632
dense (in=100, out=400) 100 0.1420 0.1414 0.0635 0.0632
conv 3x3, 16 -> 64 144 0.1189 0.1179 0.0525 0.0527
channels-first (64, 16, 3, 3) 144 0.1182 0.1179 0.0523 0.0527
These initializers plug directly into brainstate layers through w_init; the layer asks for exactly the shape described above.
brainstate.random.seed(0)
dense = brainstate.nn.Linear(400, 100, w_init=init.KaimingNormal())
conv = brainstate.nn.Conv2d((32, 32, 3), 64, kernel_size=3, w_init=init.KaimingNormal())
w_dense = dense.weight.value["weight"]
w_conv = conv.weight.value["weight"]
print(f"Linear kernel {w_dense.shape}: std {float(w_dense.std()):.4f}, expected sqrt(2/400) = {np.sqrt(2 / 400):.4f}")
print(f"Conv2d kernel {w_conv.shape}: std {float(w_conv.std()):.4f}, expected sqrt(2/(3*3*3)) = {np.sqrt(2 / 27):.4f}")
Linear kernel (400, 100): std 0.0708, expected sqrt(2/400) = 0.0707
Conv2d kernel (3, 3, 3, 64): std 0.2683, expected sqrt(2/(3*3*3)) = 0.2722
3. Why the Scale Matters: Signal Propagation#
For a layer \(y = x W\) with independent zero-mean weights of variance \(v\), each output has \(\mathrm{Var}(y) = \mathrm{fan\_in}\cdot v\cdot \mathbb{E}[x^2]\). A ReLU keeps half of a symmetric input, so \(\mathbb{E}[\mathrm{relu}(y)^2] = \mathrm{Var}(y)/2\). Keeping the second moment constant through a ReLU layer therefore needs \(v = 2/\mathrm{fan\_in}\), the Kaiming choice. LeCun’s \(1/\mathrm{fan\_in}\) halves the signal power in every ReLU layer.
The experiment below pushes a batch through 30 ReLU layers of width 256 and records the root-mean-square activation.
def rms_through_relu_net(initializer, depth=30, width=256, seed=0):
rng = np.random.default_rng(seed)
x = rng.standard_normal((512, width))
rms = [np.sqrt(np.mean(x ** 2))]
for _ in range(depth):
W = np.asarray(initializer((width, width), rng=rng))
x = np.maximum(x @ W, 0.0)
rms.append(np.sqrt(np.mean(x ** 2)))
return np.array(rms)
runs = {
"KaimingNormal (2/fan_in)": init.KaimingNormal(),
"LecunNormal (1/fan_in)": init.LecunNormal(),
"Normal(0, 0.01)": init.Normal(0.0, 0.01),
}
results = {name: rms_through_relu_net(i) for name, i in runs.items()}
fig, ax = plt.subplots(figsize=(7, 3.5))
for name, rms in results.items():
ax.semilogy(rms, label=name)
ax.set_xlabel("layer")
ax.set_ylabel("RMS activation")
ax.legend()
plt.show()
for name, rms in results.items():
print(f"{name:26s} RMS after 30 layers: {rms[-1]:.3g}")
print(f"LeCun's expected shrink factor (1/2)^(30/2) = {0.5 ** 15:.2g} (finite width adds noise around it)")
KaimingNormal (2/fan_in) RMS after 30 layers: 1.46
LecunNormal (1/fan_in) RMS after 30 layers: 4.46e-05
Normal(0, 0.01) RMS after 30 layers: 5.93e-29
LeCun's expected shrink factor (1/2)^(30/2) = 3.1e-05 (finite width adds noise around it)
For leaky ReLU with negative slope \(a\), the kept power is \((1 + a^2)/2\) instead of \(1/2\), so the Kaiming variance becomes \(2 / ((1 + a^2)\,\mathrm{fan\_in})\); set nonlinearity='leaky_relu', negative_slope=a. A larger slope needs a slightly smaller variance.
shape = (500, 500)
for slope in [0.0, 0.01, 0.2, 1.0]:
k = init.KaimingNormal(nonlinearity="leaky_relu", negative_slope=slope)
std = float(np.std(k(shape, rng=np.random.default_rng(1))))
print(f"negative_slope={slope:<5}: std {std:.4f}, expected {np.sqrt(2 / ((1 + slope ** 2) * 500)):.4f}")
negative_slope=0.0 : std 0.0632, expected 0.0632
negative_slope=0.01 : std 0.0632, expected 0.0632
negative_slope=0.2 : std 0.0619, expected 0.0620
negative_slope=1.0 : std 0.0447, expected 0.0447
Xavier uses the average of the two fans so that, for linear or tanh-like units near zero, both the forward signal and the backward gradient keep their scale when fan_in != fan_out. For a square layer it equals LeCun. LeCun is the variance used with SELU activations in self-normalizing networks.
The scale argument multiplies the variance, so any gain \(g\) can be expressed as VarianceScaling(scale=g**2, mode=..., distribution=...).
4. Orthogonal Initialization#
Orthogonal(scale) returns a matrix with orthonormal rows or columns, multiplied by scale. For shape (n_in, n_out):
if
n_in <= n_out, the rows are orthonormal (\(W W^\top = I\)), sox @ Wpreserves the norm of every inputx;if
n_in > n_out, the columns are orthonormal (\(W^\top W = I\)).
Either way all singular values equal scale. For shapes with more than two axes, all leading axes are flattened into rows.
rng = np.random.default_rng(0)
for shape in [(64, 64), (32, 128), (128, 32)]:
W = np.asarray(init.Orthogonal()(shape, rng=rng), dtype=np.float64)
sv = np.linalg.svd(W, compute_uv=False)
gram = W @ W.T if shape[0] <= shape[1] else W.T @ W
print(f"{str(shape):11s} singular values in [{sv.min():.6f}, {sv.max():.6f}], "
f"max |Gram - I| = {np.abs(gram - np.eye(len(gram))).max():.1e}")
(64, 64) singular values in [1.000000, 1.000000], max |Gram - I| = 2.9e-07
(32, 128) singular values in [1.000000, 1.000000], max |Gram - I| = 2.7e-07
(128, 32) singular values in [1.000000, 1.000000], max |Gram - I| = 2.8e-07
In a recurrent network the hidden state is multiplied by the same matrix at every step, so the eigenvalues and singular values of \(W\) decide whether signals survive long sequences. A square orthogonal matrix is an exact isometry. A Gaussian matrix with the same per-entry variance \(1/n\) has spectral radius close to 1 but singular values spread from almost 0 up to about 2, so repeated multiplication amplifies the largest directions. Below is a linear recurrence \(h_{t+1} = h_t W\) started from a unit vector.
n, steps = 100, 200
mats = {
"Orthogonal": init.Orthogonal()((n, n), rng=np.random.default_rng(1)),
"Normal(0, 1/sqrt(n))": init.Normal(0.0, 1.0 / np.sqrt(n))((n, n), rng=np.random.default_rng(1)),
}
fig, axes = plt.subplots(1, 2, figsize=(11, 3.8))
for name, W in mats.items():
W = np.asarray(W, dtype=np.float64)
h = np.random.default_rng(2).standard_normal(n)
h /= np.linalg.norm(h)
norms = []
for _ in range(steps):
h = h @ W
norms.append(np.linalg.norm(h))
axes[0].semilogy(norms, label=name)
eig = np.linalg.eigvals(W)
axes[1].scatter(eig.real, eig.imag, s=8, label=name)
sv = np.linalg.svd(W, compute_uv=False)
print(f"{name:21s} spectral radius {np.abs(eig).max():.3f}, singular values [{sv.min():.3f}, {sv.max():.3f}], "
f"|h| after {steps} steps {norms[-1]:.3g}")
theta = np.linspace(0, 2 * np.pi, 200)
axes[0].set_xlabel("step")
axes[0].set_ylabel("|h|")
axes[0].legend()
axes[1].plot(np.cos(theta), np.sin(theta), "k--", lw=0.8)
axes[1].set_aspect("equal")
axes[1].set_title("eigenvalues")
axes[1].legend(loc="lower right", fontsize=8)
fig.tight_layout()
plt.show()
Orthogonal spectral radius 1.000, singular values [1.000, 1.000], |h| after 200 steps 1
Normal(0, 1/sqrt(n)) spectral radius 1.101, singular values [0.017, 1.952], |h| after 200 steps 5.52e+07
With a nonlinearity and inputs the picture is less clean, but starting from an isometry is why Orthogonal is the usual choice for recurrent weight matrices. Use scale slightly below 1 if you want a contracting recurrence.
5. DeltaOrthogonal for Convolutions#
DeltaOrthogonal builds a convolution kernel of shape (*spatial, in, out) that is zero everywhere except at the spatial centre, where it holds an orthogonal (in, out) matrix. With SAME padding the convolution then acts like a norm-preserving 1x1 convolution at initialization (when in <= out), which is what makes very deep plain CNNs trainable (Xiao et al., 2018).
brainstate.random.seed(0)
conv = brainstate.nn.Conv2d((16, 16, 8), 32, kernel_size=3, padding="SAME",
w_init=init.DeltaOrthogonal(), b_init=None)
kernel = np.asarray(conv.weight.value["weight"])
print(f"kernel shape {kernel.shape}")
print(f"non-zero spatial positions: {[(int(i), int(j)) for i, j in zip(*np.nonzero(np.abs(kernel).sum(axis=(2, 3))))]}")
centre = kernel[1, 1].astype(np.float64)
print(f"centre @ centre.T == I: {np.allclose(centre @ centre.T, np.eye(8), atol=1e-6)}")
x = brainstate.random.normal(size=(4, 16, 16, 8))
y = conv(x)
print(f"|x| = {float(jnp.linalg.norm(x)):.4f}, |conv(x)| = {float(jnp.linalg.norm(y)):.4f}")
kernel shape (3, 3, 8, 32)
non-zero spatial positions: [(1, 1)]
centre @ centre.T == I: True
|x| = 90.1426, |conv(x)| = 90.1426
6. Identity#
Identity(scale) returns scale * eye(rows, cols) for 2-D shapes (a rectangular identity when the shape is not square) and puts the identity on the last two axes for higher-dimensional shapes. Combined with a small random perturbation it gives a recurrent matrix that initially copies its state, as in identity-initialized ReLU RNNs (Le et al., 2015). Initializers can be added, so no manual array arithmetic is needed:
print(np.asarray(init.Identity()((3, 5))))
near_identity = init.Identity() + init.Normal(0.0, 0.01)
W = np.asarray(near_identity((100, 100), rng=np.random.default_rng(0)))
print(f"diagonal mean {np.diag(W).mean():.4f}, off-diagonal std {W[~np.eye(100, dtype=bool)].std():.4f}")
[[1. 0. 0. 0. 0.]
[0. 1. 0. 0. 0.]
[0. 0. 1. 0. 0.]]
diagonal mean 0.9993, off-diagonal std 0.0100
7. Choosing an Initializer#
Layer |
Common choice |
|---|---|
Dense or conv layer followed by ReLU |
|
… followed by leaky ReLU |
|
… followed by tanh, sigmoid, or nothing |
|
SELU (self-normalizing) networks |
|
Recurrent weight matrix |
|
Very deep plain CNN |
|
Summary#
The variance-scaling family sets
variance = scale / fan; Kaiming, Xavier, and LeCun are presets ofscaleandmode.Fans follow the
brainstatelayout:(in, out)for dense weights,(*spatial, in, out)for conv kernels; usein_axis/out_axisfor other layouts.Kaiming keeps ReLU activations at a constant scale; LeCun halves the signal power per ReLU layer.
Orthogonalgives unit singular values, so repeated multiplication preserves norms;DeltaOrthogonalextends this to conv kernels.