DeltaOrthogonal

Contents

DeltaOrthogonal#

class braintools.init.DeltaOrthogonal(scale=1.0, unit=None)#

Delta-orthogonal initialization for deep CNNs.

Initializes convolution kernels to be delta functions in spatial dimensions combined with orthogonal initialization in the channel dimensions. This is particularly useful for very deep convolutional networks.

Reference: Xiao et al., “Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks”, ICML 2018.

Parameters:

scale (float) – Multiplicative factor to apply to the orthogonal matrix (default: 1.0).

Examples

>>> import numpy as np
>>> from braintools.init import DeltaOrthogonal
>>>
>>> # For a 3x3 convolutional kernel with 64 input and 128 output channels
>>> init = DeltaOrthogonal(scale=1.0)
>>> rng = np.random.default_rng(0)
>>> weights = init((128, 64, 3, 3), rng=rng)
Contents