DistributedSampler#

class braintools.trainer.DistributedSampler(data_source, num_replicas=None, rank=None, shuffle=True, seed=0, drop_last=False)#

Sampler for distributed training.

Splits data across multiple replicas (devices/processes).

Parameters:
  • data_source (Dataset) – Dataset to sample from.

  • num_replicas (int | None) – Number of distributed replicas. Default is number of JAX devices.

  • rank (int | None) – Rank of current replica. Default is 0.

  • shuffle (bool) – Whether to shuffle the data.

  • seed (int) – Random seed for shuffling.

  • drop_last (bool) – Whether to drop samples that don’t divide evenly.

Examples

Each replica gets an interleaved 1/4 of the data:

>>> import jax.numpy as jnp
>>> import braintools
>>> dataset = braintools.trainer.ArrayDataset(jnp.arange(8))
>>> sampler = braintools.trainer.DistributedSampler(
...     dataset, num_replicas=4, rank=1, shuffle=False)
>>> list(sampler)
[1, 5]
set_epoch(epoch)[source]#

Set the epoch for deterministic shuffling.