braintools.trainer.all_reduce#
- braintools.trainer.all_reduce(tensor, op='mean', axis_name='batch')[source]#
All-reduce tensor across devices.
- Parameters:
- Returns:
Reduced tensor.
- Return type:
Examples
Collectives need a named mapped axis.
jax.pmapprovides one across devices;jax.vmapprovides one on a single device:>>> import jax >>> import jax.numpy as jnp >>> import braintools >>> x = jnp.arange(4.0) >>> jax.vmap(lambda v: braintools.trainer.all_reduce(v, op='mean'), ... axis_name='batch')(x) Array([1.5, 1.5, 1.5, 1.5], dtype=float32) >>> jax.vmap(lambda v: braintools.trainer.all_reduce(v, op='max'), ... axis_name='batch')(x) Array([3., 3., 3., 3.], dtype=float32)