OptaxOptimizer#
- class braintools.optim.OptaxOptimizer(tx=None, lr=0.001, weight_decay=0.0, grad_clip_norm=None, grad_clip_value=None)#
Base class for Optax-based optimizers with PyTorch-like interface.
This class provides a unified interface for JAX/Optax optimizers with advanced features including learning rate scheduling, parameter groups, gradient clipping, and weight decay.
- Parameters:
tx (
GradientTransformation|None) – An Optax gradient transformation. If None, will be created based on optimizer-specific parameters via_create_default_tx()method.lr (
float|LRScheduler) – Learning rate. Can be a float (automatically converted to ConstantLR) or an LRScheduler instance for advanced scheduling.weight_decay (
float) – Weight decay coefficient (L2 penalty).grad_clip_norm (
float|None) – Maximum gradient norm for clipping. If provided, gradients will be clipped usingoptax.clip_by_global_norm.grad_clip_value (
float|None) – Maximum gradient value for clipping. If provided, gradients will be clipped element-wise usingoptax.clip.
- Variables:
param_states (UniqueStateManager) – Container for PyTree of brainstate.State objects representing trainable parameters.
opt_state (OptimState) – Optimizer state containing momentum, variance, and other optimizer-specific values.
step_count (OptimState) – Number of optimization steps taken.
param_groups (list of dict) – List of parameter groups with their own hyperparameters. Each group is a dictionary with keys ‘params’, ‘lr’, ‘weight_decay’, etc.
base_lr (float) – Base learning rate (read-only property).
lr (float) – Current learning rate (can be modified by schedulers).
Notes
This base class implements the unified learning rate handling where all learning rates (whether float or LRScheduler) are internally managed through a scheduler. Float learning rates are automatically converted to
ConstantLRfor consistent handling.Examples
Basic usage with float learning rate:
>>> import brainstate >>> import braintools >>> >>> # Define a simple model >>> class Model(brainstate.nn.Module): ... def __init__(self): ... super().__init__() ... self.linear = brainstate.nn.Linear(10, 5) ... >>> model = Model() >>> optimizer = braintools.optim.Adam(lr=0.001) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
Using learning rate scheduler:
>>> import brainstate >>> import braintools >>> >>> model = Model() >>> scheduler = braintools.optim.StepLR(base_lr=0.01, step_size=10, gamma=0.1) >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Gradients of a toy loss w.r.t. the registered weights >>> import jax.numpy as jnp >>> x = jnp.ones((4, 10)) >>> def loss_fn(): ... return jnp.sum(model.linear(x) ** 2) >>> grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))() >>> >>> # Training loop >>> for epoch in range(100): ... optimizer.step(grads) ... scheduler.step() >>> print(f"{optimizer.current_lr:.0e}") 1e-12
Multiple parameter groups with different learning rates:
>>> import brainstate >>> import braintools >>> import jax.numpy as jnp >>> >>> model = Model() >>> optimizer = braintools.optim.Adam(lr=0.001) >>> >>> # Register main parameters >>> _ = optimizer.register_trainable_weights(model.linear.states(brainstate.ParamState)) >>> >>> # Add another group with different lr >>> special_params = {'special': brainstate.ParamState(jnp.zeros(5))} >>> _ = optimizer.add_param_group(special_params, lr=0.0001) >>> len(optimizer.param_groups) 2
See also
- add_param_group(params, **kwargs)[source]#
Add a parameter group with specific hyperparameters.
- Parameters:
params (
State]) – A pytree (dict) of brainstate.State objects.**kwargs – Additional hyperparameters for this group (e.g.
lr,weight_decay).
Notes
Added parameter groups always use Adam-style adaptive scaling (
optax.scale_by_adam) regardless of the parent optimizer’s algorithm; onlylrandweight_decayare honored per group. To optimize a group with a different algorithm, construct a separate optimizer instead.
- load_state_dict(state_dict)[source]#
Load optimizer state from a dictionary.
- Parameters:
state_dict (
Dict[str,Any]) – A dictionary returned bystate_dict(), in memory or after a msgpack round trip. The scheduler states ('lr_scheduler','schedulers') are loaded into this optimizer’s schedulers when present; a dictionary without them (written before they were saved) loads as before and leaves the schedulers where they are.- Raises:
ValueError – If
state_dictholds a different number ofadd_scheduler()schedulers than this optimizer. Nothing is changed in that case.
- register_trainable_weights(param_states)[source]#
Register trainable weights and initialize optimizer state.
- Parameters:
param_states (
State]) – A pytree (dict) of brainstate.State objects representing parameters.
- state_dict()[source]#
Return the state of the optimizer as a dictionary.
- Returns:
The step count, the current and base learning rates, the per-group hyperparameters and optax states, the optax state (
Nonebeforeregister_trainable_weights), and the learning-rate schedule:'lr_scheduler'holds thestate_dict()of the scheduler passed aslr(aConstantLRfor a floatlr) and'schedulers'thestate_dict()of each scheduler registered withadd_scheduler(), in order. All values are numbers, arrays, strings, lists and dicts.- Return type:
- step(grads, **extra_args)[source]#
Perform a single optimization step.
- Parameters:
grads (
Dict[Hashable,Any]) – Dictionary mapping parameter keys to their gradients.**extra_args – Extra keyword arguments forwarded to the underlying
optaxtransformation’supdate(e.g.value,value_fnandgradrequired by LBFGS line searches). Ignored by transformations that do not accept extra arguments.