Orthogonal

class braintools.init.Orthogonal(scale=1.0, unit=None)

Orthogonal matrix initialization.

Generates a random orthogonal matrix using QR decomposition. For non-square matrices, generates an orthogonal matrix of the appropriate shape.

This initialization helps preserve the norm of gradients during backpropagation and is particularly useful for recurrent networks.

Reference: Saxe et al., “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks”, ICLR 2014.

Parameters:

scale (float) – Multiplicative factor to apply to the orthogonal matrix (default: 1.0).

Examples

>>> import numpy as np
>>> from braintools.init import Orthogonal
>>>
>>> init = Orthogonal(scale=1.0)
>>> rng = np.random.default_rng(0)
>>> weights = init((100, 100), rng=rng)
>>>
>>> # For recurrent networks, often use sqrt(2) scale
>>> init_recurrent = Orthogonal(scale=np.sqrt(2))
>>> weights = init_recurrent((50, 50), rng=rng)