Linear

Linear#

class braintrace.nn.Linear#

Linear transformation layer.

Applies a linear transformation to the incoming data: \(y = xW + b\)

Parameters:

  • in_size (int or tuple of int) – The input feature size.

  • out_size (int or tuple of int) – The output feature size.

  • w_init (Callable or ArrayLike, optional) – Weight initializer. Default is KaimingNormal().

  • b_init (Callable, ArrayLike, or None, optional) – Bias initializer. If None, no bias is added. Default is ZeroInit().

  • w_mask (ArrayLike, Callable, or None, optional) – Optional mask for the weights. If provided, weights will be element-wise multiplied by this mask.

  • name (str, optional) – Name of the module.

  • param_type (type, optional) – Type of parameter state. Default is ParamState.

Variables:

  • in_size (tuple) – Input feature size.

  • out_size (tuple) – Output feature size.

  • w_mask (ArrayLike or None) – Weight mask if provided.

  • weight (ParamState) – Parameter state containing ‘weight’ and optionally ‘bias’.

Examples

>>> import braintrace as braintrace
>>> import jax.numpy as jnp
>>>
>>> # Create a linear layer
>>> layer = braintrace.nn.Linear((10,), (5,))
>>> x = jnp.ones((32, 10))
>>> y = layer(x)
>>> y.shape
(32, 5)
>>>
>>> # Linear layer without bias
>>> layer = braintrace.nn.Linear((10,), (5,), b_init=None)
>>> y = layer(x)
>>> y.shape
(32, 5)
update(x)#

Apply the linear transform through the ETP matmul primitive.

Routing the matrix multiplication through braintrace.matmul() (instead of a plain JAX dot) is what makes weight eligible for online-learning trace computation.

Parameters:

x (ArrayLike) – Input array, of shape (..., in_size).

Returns:

ArrayLike – The transformed output, of shape (..., out_size).

Linear.__init__(in_size, out_size, w_init=KaimingNormal(   scale=2.0, mode='fan_in', in_axis=-2, out_axis=-1, distribution='truncated_normal', rng=RandomState(Array((), dtype=key<fry>) overlaying:   [2233333421 2029709265]), unit=Unit("1") ), b_init=ZeroInit(   unit=Unit("1") ), w_mask=None, name=None, param_type=<class 'brainstate.ParamState'>)#