Variance Scaling and Orthogonal Initialization#

Deep and recurrent networks need weights whose scale matches the layer size: too small and signals shrink layer by layer, too large and they blow up. This tutorial covers the initializers that set that scale automatically.

Initializer

Variance of each weight

Default distribution

VarianceScaling(scale, mode, distribution)

scale / fan

normal (or uniform, truncated_normal)

KaimingNormal / KaimingUniform

2 / fan_in for ReLU, 2 / ((1 + a^2) fan_in) for leaky ReLU

normal / uniform

XavierNormal / XavierUniform

2 / (fan_in + fan_out)

normal / uniform

LecunNormal / LecunUniform

1 / fan_in

normal / uniform

Orthogonal(scale)

rows or columns orthonormal, times scale

DeltaOrthogonal(scale)

orthogonal channel matrix at the kernel centre

Identity(scale)

identity matrix times scale

fan is fan_in, fan_out, or their average, chosen by mode. The uniform variants sample from \([-\sqrt{3\,\mathrm{var}}, \sqrt{3\,\mathrm{var}}]\), which has the same variance as the normal ones; truncated_normal samples a normal cut at two standard deviations and rescales it to the same variance.

1. Setup#

import brainstate
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np

from braintools import init

2. Which Axis Is Fan-In?#

braintools.init follows the same weight layout as brainstate layers (and JAX/Flax):

  • a dense weight has shape (in, out), so the layer computes y = x @ W;

  • a convolution kernel has shape (*spatial, in, out), e.g. (3, 3, 16, 32) for a 3x3 kernel from 16 to 32 channels.

The fans are fan_in = in * receptive_field and fan_out = out * receptive_field, where receptive_field is the product of the spatial sizes. For a 1-D shape (n,) both fans are n. If your weights use another layout, for example PyTorch’s channels-first (out, in, kh, kw), select the axes with in_axis and out_axis.

rng = np.random.default_rng(0)
cases = [
    ("dense (in=400, out=100)", (400, 100), {}),
    ("dense (in=100, out=400)", (100, 400), {}),
    ("conv 3x3, 16 -> 64", (3, 3, 16, 64), {}),
    ("channels-first (64, 16, 3, 3)", (64, 16, 3, 3), dict(in_axis=1, out_axis=0)),
]

print(f"{'shape':32s} {'fan_in':>6s} {'Kaiming std':>11s} {'expected':>8s} {'Xavier std':>10s} {'expected':>8s}")
for name, shape, axes in cases:
    kaiming = init.KaimingNormal(**axes)
    xavier = init.XavierNormal(**axes)
    fan_in, fan_out = kaiming._compute_fans(shape)
    k_std = float(np.std(kaiming(shape, rng=rng)))
    x_std = float(np.std(xavier(shape, rng=rng)))
    print(f"{name:32s} {fan_in:6d} {k_std:11.4f} {np.sqrt(2 / fan_in):8.4f} "
          f"{x_std:10.4f} {np.sqrt(2 / (fan_in + fan_out)):8.4f}")
shape                            fan_in Kaiming std expected Xavier std expected
dense (in=400, out=100)             400      0.0708   0.0707     0.0631   0.0632
dense (in=100, out=400)             100      0.1420   0.1414     0.0635   0.0632
conv 3x3, 16 -> 64                  144      0.1189   0.1179     0.0525   0.0527
channels-first (64, 16, 3, 3)       144      0.1182   0.1179     0.0523   0.0527

These initializers plug directly into brainstate layers through w_init; the layer asks for exactly the shape described above.

brainstate.random.seed(0)
dense = brainstate.nn.Linear(400, 100, w_init=init.KaimingNormal())
conv = brainstate.nn.Conv2d((32, 32, 3), 64, kernel_size=3, w_init=init.KaimingNormal())

w_dense = dense.weight.value["weight"]
w_conv = conv.weight.value["weight"]
print(f"Linear kernel {w_dense.shape}: std {float(w_dense.std()):.4f}, expected sqrt(2/400)   = {np.sqrt(2 / 400):.4f}")
print(f"Conv2d kernel {w_conv.shape}: std {float(w_conv.std()):.4f}, expected sqrt(2/(3*3*3)) = {np.sqrt(2 / 27):.4f}")
Linear kernel (400, 100): std 0.0708, expected sqrt(2/400)   = 0.0707
Conv2d kernel (3, 3, 3, 64): std 0.2683, expected sqrt(2/(3*3*3)) = 0.2722

3. Why the Scale Matters: Signal Propagation#

For a layer \(y = x W\) with independent zero-mean weights of variance \(v\), each output has \(\mathrm{Var}(y) = \mathrm{fan\_in}\cdot v\cdot \mathbb{E}[x^2]\). A ReLU keeps half of a symmetric input, so \(\mathbb{E}[\mathrm{relu}(y)^2] = \mathrm{Var}(y)/2\). Keeping the second moment constant through a ReLU layer therefore needs \(v = 2/\mathrm{fan\_in}\), the Kaiming choice. LeCun’s \(1/\mathrm{fan\_in}\) halves the signal power in every ReLU layer.

The experiment below pushes a batch through 30 ReLU layers of width 256 and records the root-mean-square activation.

def rms_through_relu_net(initializer, depth=30, width=256, seed=0):
    rng = np.random.default_rng(seed)
    x = rng.standard_normal((512, width))
    rms = [np.sqrt(np.mean(x ** 2))]
    for _ in range(depth):
        W = np.asarray(initializer((width, width), rng=rng))
        x = np.maximum(x @ W, 0.0)
        rms.append(np.sqrt(np.mean(x ** 2)))
    return np.array(rms)


runs = {
    "KaimingNormal (2/fan_in)": init.KaimingNormal(),
    "LecunNormal (1/fan_in)": init.LecunNormal(),
    "Normal(0, 0.01)": init.Normal(0.0, 0.01),
}
results = {name: rms_through_relu_net(i) for name, i in runs.items()}

fig, ax = plt.subplots(figsize=(7, 3.5))
for name, rms in results.items():
    ax.semilogy(rms, label=name)
ax.set_xlabel("layer")
ax.set_ylabel("RMS activation")
ax.legend()
plt.show()

for name, rms in results.items():
    print(f"{name:26s} RMS after 30 layers: {rms[-1]:.3g}")
print(f"LeCun's expected shrink factor (1/2)^(30/2) = {0.5 ** 15:.2g} (finite width adds noise around it)")
../_images/4072be04adf04107174b8bee2b18c6655b90e78ee9a9a9ccd8318e575fc6b440.png
KaimingNormal (2/fan_in)   RMS after 30 layers: 1.46
LecunNormal (1/fan_in)     RMS after 30 layers: 4.46e-05
Normal(0, 0.01)            RMS after 30 layers: 5.93e-29
LeCun's expected shrink factor (1/2)^(30/2) = 3.1e-05 (finite width adds noise around it)

For leaky ReLU with negative slope \(a\), the kept power is \((1 + a^2)/2\) instead of \(1/2\), so the Kaiming variance becomes \(2 / ((1 + a^2)\,\mathrm{fan\_in})\); set nonlinearity='leaky_relu', negative_slope=a. A larger slope needs a slightly smaller variance.

shape = (500, 500)
for slope in [0.0, 0.01, 0.2, 1.0]:
    k = init.KaimingNormal(nonlinearity="leaky_relu", negative_slope=slope)
    std = float(np.std(k(shape, rng=np.random.default_rng(1))))
    print(f"negative_slope={slope:<5}: std {std:.4f}, expected {np.sqrt(2 / ((1 + slope ** 2) * 500)):.4f}")
negative_slope=0.0  : std 0.0632, expected 0.0632
negative_slope=0.01 : std 0.0632, expected 0.0632
negative_slope=0.2  : std 0.0619, expected 0.0620
negative_slope=1.0  : std 0.0447, expected 0.0447

Xavier uses the average of the two fans so that, for linear or tanh-like units near zero, both the forward signal and the backward gradient keep their scale when fan_in != fan_out. For a square layer it equals LeCun. LeCun is the variance used with SELU activations in self-normalizing networks.

The scale argument multiplies the variance, so any gain \(g\) can be expressed as VarianceScaling(scale=g**2, mode=..., distribution=...).

4. Orthogonal Initialization#

Orthogonal(scale) returns a matrix with orthonormal rows or columns, multiplied by scale. For shape (n_in, n_out):

  • if n_in <= n_out, the rows are orthonormal (\(W W^\top = I\)), so x @ W preserves the norm of every input x;

  • if n_in > n_out, the columns are orthonormal (\(W^\top W = I\)).

Either way all singular values equal scale. For shapes with more than two axes, all leading axes are flattened into rows.

rng = np.random.default_rng(0)
for shape in [(64, 64), (32, 128), (128, 32)]:
    W = np.asarray(init.Orthogonal()(shape, rng=rng), dtype=np.float64)
    sv = np.linalg.svd(W, compute_uv=False)
    gram = W @ W.T if shape[0] <= shape[1] else W.T @ W
    print(f"{str(shape):11s} singular values in [{sv.min():.6f}, {sv.max():.6f}], "
          f"max |Gram - I| = {np.abs(gram - np.eye(len(gram))).max():.1e}")
(64, 64)    singular values in [1.000000, 1.000000], max |Gram - I| = 2.9e-07
(32, 128)   singular values in [1.000000, 1.000000], max |Gram - I| = 2.7e-07
(128, 32)   singular values in [1.000000, 1.000000], max |Gram - I| = 2.8e-07

In a recurrent network the hidden state is multiplied by the same matrix at every step, so the eigenvalues and singular values of \(W\) decide whether signals survive long sequences. A square orthogonal matrix is an exact isometry. A Gaussian matrix with the same per-entry variance \(1/n\) has spectral radius close to 1 but singular values spread from almost 0 up to about 2, so repeated multiplication amplifies the largest directions. Below is a linear recurrence \(h_{t+1} = h_t W\) started from a unit vector.

n, steps = 100, 200
mats = {
    "Orthogonal": init.Orthogonal()((n, n), rng=np.random.default_rng(1)),
    "Normal(0, 1/sqrt(n))": init.Normal(0.0, 1.0 / np.sqrt(n))((n, n), rng=np.random.default_rng(1)),
}

fig, axes = plt.subplots(1, 2, figsize=(11, 3.8))
for name, W in mats.items():
    W = np.asarray(W, dtype=np.float64)
    h = np.random.default_rng(2).standard_normal(n)
    h /= np.linalg.norm(h)
    norms = []
    for _ in range(steps):
        h = h @ W
        norms.append(np.linalg.norm(h))
    axes[0].semilogy(norms, label=name)
    eig = np.linalg.eigvals(W)
    axes[1].scatter(eig.real, eig.imag, s=8, label=name)
    sv = np.linalg.svd(W, compute_uv=False)
    print(f"{name:21s} spectral radius {np.abs(eig).max():.3f}, singular values [{sv.min():.3f}, {sv.max():.3f}], "
          f"|h| after {steps} steps {norms[-1]:.3g}")

theta = np.linspace(0, 2 * np.pi, 200)
axes[0].set_xlabel("step")
axes[0].set_ylabel("|h|")
axes[0].legend()
axes[1].plot(np.cos(theta), np.sin(theta), "k--", lw=0.8)
axes[1].set_aspect("equal")
axes[1].set_title("eigenvalues")
axes[1].legend(loc="lower right", fontsize=8)
fig.tight_layout()
plt.show()
Orthogonal            spectral radius 1.000, singular values [1.000, 1.000], |h| after 200 steps 1
Normal(0, 1/sqrt(n))  spectral radius 1.101, singular values [0.017, 1.952], |h| after 200 steps 5.52e+07
../_images/7e8b40bbea6687a9abf646c4dd5b174dca1d8d68336b1ee9d058389fceaac694.png

With a nonlinearity and inputs the picture is less clean, but starting from an isometry is why Orthogonal is the usual choice for recurrent weight matrices. Use scale slightly below 1 if you want a contracting recurrence.

5. DeltaOrthogonal for Convolutions#

DeltaOrthogonal builds a convolution kernel of shape (*spatial, in, out) that is zero everywhere except at the spatial centre, where it holds an orthogonal (in, out) matrix. With SAME padding the convolution then acts like a norm-preserving 1x1 convolution at initialization (when in <= out), which is what makes very deep plain CNNs trainable (Xiao et al., 2018).

brainstate.random.seed(0)
conv = brainstate.nn.Conv2d((16, 16, 8), 32, kernel_size=3, padding="SAME",
                            w_init=init.DeltaOrthogonal(), b_init=None)
kernel = np.asarray(conv.weight.value["weight"])
print(f"kernel shape {kernel.shape}")
print(f"non-zero spatial positions: {[(int(i), int(j)) for i, j in zip(*np.nonzero(np.abs(kernel).sum(axis=(2, 3))))]}")

centre = kernel[1, 1].astype(np.float64)
print(f"centre @ centre.T == I: {np.allclose(centre @ centre.T, np.eye(8), atol=1e-6)}")

x = brainstate.random.normal(size=(4, 16, 16, 8))
y = conv(x)
print(f"|x| = {float(jnp.linalg.norm(x)):.4f}, |conv(x)| = {float(jnp.linalg.norm(y)):.4f}")
kernel shape (3, 3, 8, 32)
non-zero spatial positions: [(1, 1)]
centre @ centre.T == I: True
|x| = 90.1426, |conv(x)| = 90.1426

6. Identity#

Identity(scale) returns scale * eye(rows, cols) for 2-D shapes (a rectangular identity when the shape is not square) and puts the identity on the last two axes for higher-dimensional shapes. Combined with a small random perturbation it gives a recurrent matrix that initially copies its state, as in identity-initialized ReLU RNNs (Le et al., 2015). Initializers can be added, so no manual array arithmetic is needed:

print(np.asarray(init.Identity()((3, 5))))

near_identity = init.Identity() + init.Normal(0.0, 0.01)
W = np.asarray(near_identity((100, 100), rng=np.random.default_rng(0)))
print(f"diagonal mean {np.diag(W).mean():.4f}, off-diagonal std {W[~np.eye(100, dtype=bool)].std():.4f}")
[[1. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0.]
 [0. 0. 1. 0. 0.]]
diagonal mean 0.9993, off-diagonal std 0.0100

7. Choosing an Initializer#

Layer

Common choice

Dense or conv layer followed by ReLU

KaimingNormal() / KaimingUniform()

… followed by leaky ReLU

KaimingNormal(nonlinearity='leaky_relu', negative_slope=a)

… followed by tanh, sigmoid, or nothing

XavierNormal() / XavierUniform()

SELU (self-normalizing) networks

LecunNormal()

Recurrent weight matrix

Orthogonal(), or Identity() + Normal(0, small) for ReLU RNNs

Very deep plain CNN

DeltaOrthogonal()

Summary#

  • The variance-scaling family sets variance = scale / fan; Kaiming, Xavier, and LeCun are presets of scale and mode.

  • Fans follow the brainstate layout: (in, out) for dense weights, (*spatial, in, out) for conv kernels; use in_axis / out_axis for other layouts.

  • Kaiming keeps ReLU activations at a constant scale; LeCun halves the signal power per ReLU layer.

  • Orthogonal gives unit singular values, so repeated multiplication preserves norms; DeltaOrthogonal extends this to conv kernels.