OptaxOptimizer#

class braintools.optim.OptaxOptimizer(tx=None, lr=0.001, weight_decay=0.0, grad_clip_norm=None, grad_clip_value=None)#

Base class for Optax-based optimizers with PyTorch-like interface.

This class provides a unified interface for JAX/Optax optimizers with advanced features including learning rate scheduling, parameter groups, gradient clipping, and weight decay.

Parameters:
  • tx (GradientTransformation | None) – An Optax gradient transformation. If None, will be created based on optimizer-specific parameters via _create_default_tx() method.

  • lr (float | LRScheduler) – Learning rate. Can be a float (automatically converted to ConstantLR) or an LRScheduler instance for advanced scheduling.

  • weight_decay (float) – Weight decay coefficient (L2 penalty).

  • grad_clip_norm (float | None) – Maximum gradient norm for clipping. If provided, gradients will be clipped using optax.clip_by_global_norm.

  • grad_clip_value (float | None) – Maximum gradient value for clipping. If provided, gradients will be clipped element-wise using optax.clip.

Variables:
  • param_states (UniqueStateManager) – Container for PyTree of brainstate.State objects representing trainable parameters.

  • opt_state (OptimState) – Optimizer state containing momentum, variance, and other optimizer-specific values.

  • step_count (OptimState) – Number of optimization steps taken.

  • param_groups (list of dict) – List of parameter groups with their own hyperparameters. Each group is a dictionary with keys ‘params’, ‘lr’, ‘weight_decay’, etc.

  • base_lr (float) – Base learning rate (read-only property).

  • lr (float) – Current learning rate (can be modified by schedulers).

Notes

This base class implements the unified learning rate handling where all learning rates (whether float or LRScheduler) are internally managed through a scheduler. Float learning rates are automatically converted to ConstantLR for consistent handling.

Examples

Basic usage with float learning rate:

>>> import brainstate
>>> import braintools
>>>
>>> # Define a simple model
>>> class Model(brainstate.nn.Module):
...     def __init__(self):
...         super().__init__()
...         self.linear = brainstate.nn.Linear(10, 5)
...
>>> model = Model()
>>> optimizer = braintools.optim.Adam(lr=0.001)
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))

Using learning rate scheduler:

>>> import brainstate
>>> import braintools
>>>
>>> model = Model()
>>> scheduler = braintools.optim.StepLR(base_lr=0.01, step_size=10, gamma=0.1)
>>> optimizer = braintools.optim.Adam(lr=scheduler)
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
>>>
>>> # Gradients of a toy loss w.r.t. the registered weights
>>> import jax.numpy as jnp
>>> x = jnp.ones((4, 10))
>>> def loss_fn():
...     return jnp.sum(model.linear(x) ** 2)
>>> grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))()
>>>
>>> # Training loop
>>> for epoch in range(100):
...     optimizer.step(grads)
...     scheduler.step()
>>> print(f"{optimizer.current_lr:.0e}")
1e-12

Multiple parameter groups with different learning rates:

>>> import brainstate
>>> import braintools
>>> import jax.numpy as jnp
>>>
>>> model = Model()
>>> optimizer = braintools.optim.Adam(lr=0.001)
>>>
>>> # Register main parameters
>>> _ = optimizer.register_trainable_weights(model.linear.states(brainstate.ParamState))
>>>
>>> # Add another group with different lr
>>> special_params = {'special': brainstate.ParamState(jnp.zeros(5))}
>>> _ = optimizer.add_param_group(special_params, lr=0.0001)
>>> len(optimizer.param_groups)
2

See also

Adam

Adam optimizer with adaptive learning rates

SGD

Stochastic gradient descent optimizer

AdamW

AdamW optimizer with decoupled weight decay

StepLR

Learning rate scheduler with step decay

add_param_group(params, **kwargs)[source]#

Add a parameter group with specific hyperparameters.

Parameters:
  • params (State]) – A pytree (dict) of brainstate.State objects.

  • **kwargs – Additional hyperparameters for this group (e.g. lr, weight_decay).

Notes

Added parameter groups always use Adam-style adaptive scaling (optax.scale_by_adam) regardless of the parent optimizer’s algorithm; only lr and weight_decay are honored per group. To optimize a group with a different algorithm, construct a separate optimizer instead.

add_scheduler(scheduler)[source]#

Add a learning rate scheduler.

property base_lr: float#

Get base learning rate.

property current_lr: float#

Get current learning rate.

default_tx()[source]#

Create default gradient transformation with clipping and weight decay.

get_last_lr()[source]#

Get last computed learning rates from schedulers.

Return type:

List[float]

load_state_dict(state_dict)[source]#

Load optimizer state from a dictionary.

Parameters:

state_dict (Dict[str, Any]) – A dictionary returned by state_dict(), in memory or after a msgpack round trip. The scheduler states ('lr_scheduler', 'schedulers') are loaded into this optimizer’s schedulers when present; a dictionary without them (written before they were saved) loads as before and leaves the schedulers where they are.

Raises:

ValueError – If state_dict holds a different number of add_scheduler() schedulers than this optimizer. Nothing is changed in that case.

lr_apply(apply_fn)[source]#

Apply a function to modify the current learning rate.

register_trainable_weights(param_states)[source]#

Register trainable weights and initialize optimizer state.

Parameters:

param_states (State]) – A pytree (dict) of brainstate.State objects representing parameters.

state_dict()[source]#

Return the state of the optimizer as a dictionary.

Returns:

The step count, the current and base learning rates, the per-group hyperparameters and optax states, the optax state (None before register_trainable_weights), and the learning-rate schedule: 'lr_scheduler' holds the state_dict() of the scheduler passed as lr (a ConstantLR for a float lr) and 'schedulers' the state_dict() of each scheduler registered with add_scheduler(), in order. All values are numbers, arrays, strings, lists and dicts.

Return type:

dict

step(grads, **extra_args)[source]#

Perform a single optimization step.

Parameters:
  • grads (Dict[Hashable, Any]) – Dictionary mapping parameter keys to their gradients.

  • **extra_args – Extra keyword arguments forwarded to the underlying optax transformation’s update (e.g. value, value_fn and grad required by LBFGS line searches). Ignored by transformations that do not accept extra arguments.

update(grads, **extra_args)[source]#

Update the model states with gradients (backward compatibility).