braintools.trainer.all_reduce

Contents

braintools.trainer.all_reduce#

braintools.trainer.all_reduce(tensor, op='mean', axis_name='batch')[source]#

All-reduce tensor across devices.

Parameters:
  • tensor (Any) – Tensor to reduce.

  • op (str) – Reduction operation (‘mean’, ‘sum’, ‘min’, ‘max’).

  • axis_name (str) – Name of the axis for reduction (used with pmap).

Returns:

Reduced tensor.

Return type:

Any

Examples

Collectives need a named mapped axis. jax.pmap provides one across devices; jax.vmap provides one on a single device:

>>> import jax
>>> import jax.numpy as jnp
>>> import braintools
>>> x = jnp.arange(4.0)
>>> jax.vmap(lambda v: braintools.trainer.all_reduce(v, op='mean'),
...          axis_name='batch')(x)
Array([1.5, 1.5, 1.5, 1.5], dtype=float32)
>>> jax.vmap(lambda v: braintools.trainer.all_reduce(v, op='max'),
...          axis_name='batch')(x)
Array([3., 3., 3., 3.], dtype=float32)