DistributedDataLoader#

class braintools.trainer.DistributedDataLoader(dataset, batch_size=32, num_replicas=None, rank=None, shuffle=True, seed=0, drop_last=False, **kwargs)#

DataLoader for distributed training.

Automatically shards data across multiple devices/processes.

Parameters:
  • dataset (Dataset | Any) – Dataset to load from.

  • batch_size (int) – Batch size per replica.

  • num_replicas (int | None) – Number of replicas. Default is number of JAX devices.

  • rank (int | None) – Rank of this replica.

  • shuffle (bool) – Whether to shuffle data.

  • seed (int) – Random seed for shuffling.

  • drop_last (bool) – Whether to drop incomplete batches.

  • **kwargs – Additional arguments passed to DataLoader.

Examples

Each replica iterates over its own shard of the data:

>>> import jax.numpy as jnp
>>> import braintools
>>> X = jnp.arange(16.0).reshape(8, 2)
>>> loader = braintools.trainer.DistributedDataLoader(
...     X, batch_size=2, num_replicas=2, rank=0, shuffle=False)
>>> len(loader)
2
>>> [batch[0][:, 0].tolist() for batch in loader]
[[0.0, 4.0], [8.0, 12.0]]
set_epoch(epoch)[source]#

Set the epoch for the distributed sampler.