DeltaOrthogonal

Contents

DeltaOrthogonal#

class braintools.init.DeltaOrthogonal(scale=1.0, unit=None)#

Delta-orthogonal initialization for deep CNNs.

Initializes convolution kernels to be delta functions in spatial dimensions combined with orthogonal initialization in the channel dimensions. This is particularly useful for very deep convolutional networks.

Reference: Xiao et al., “Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks”, ICML 2018.

Notes

The kernel uses the channels-first convention (out_channels, in_channels, *spatial). The channel matrix placed at the spatial centre is (semi-)orthogonal for any channel counts: when out_channels >= in_channels it has orthonormal columns (W.T @ W == I, the norm-preserving expanding case), and when out_channels < in_channels it has orthonormal rows (W @ W.T == I). Both yield unit singular values, so the convolution is an isometry on its column/row space.

Parameters:

scale (float) – Multiplicative factor to apply to the orthogonal matrix (default: 1.0).

Examples

>>> import numpy as np
>>> from braintools.init import DeltaOrthogonal
>>>
>>> # For a 3x3 convolutional kernel with 64 input and 128 output channels
>>> init = DeltaOrthogonal(scale=1.0)
>>> rng = np.random.default_rng(0)
>>> weights = init((128, 64, 3, 3), rng=rng)
Contents