DeltaOrthogonal

Contents

DeltaOrthogonal#

class braintools.init.DeltaOrthogonal(scale=1.0, unit=None)#

Delta-orthogonal initialization for deep CNNs.

Initializes convolution kernels to be delta functions in spatial dimensions combined with orthogonal initialization in the channel dimensions. This is particularly useful for very deep convolutional networks.

Reference: Xiao et al., “Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks”, ICML 2018.

Notes

The kernel uses the channels-last convention (*spatial, in_channels, out_channels) of brainstate convolution layers (e.g. (3, 3, 64, 128) for a 3x3 kernel). The (in_channels, out_channels) matrix placed at the spatial centre is (semi-)orthogonal for any channel counts: it has orthonormal rows when in_channels <= out_channels (W @ W.T == I, the norm-preserving case for y = x @ W) and orthonormal columns otherwise. Both yield unit singular values. Every other spatial position is zero.

Parameters:

scale (float) – Multiplicative factor to apply to the orthogonal matrix (default: 1.0).

Examples

>>> import numpy as np
>>> from braintools.init import DeltaOrthogonal
>>>
>>> # For a 3x3 convolutional kernel with 64 input and 128 output channels
>>> init = DeltaOrthogonal(scale=1.0)
>>> rng = np.random.default_rng(0)
>>> weights = init((3, 3, 64, 128), rng=rng)