ExponentialLR#
- class braintools.optim.ExponentialLR(base_lr=0.001, gamma=0.95, last_epoch=0)#
Exponential learning rate scheduler - Decays learning rate exponentially.
ExponentialLR multiplies the learning rate by gamma at every epoch, creating a smooth exponential decay. This scheduler is useful when you want a continuous and predictable decrease in the learning rate throughout training.
- Parameters:
base_lr (
float|List[float]) – Initial learning rate(s). Can be a single float or a list of floats for multiple parameter groups. Default: 1e-3.gamma (
float) – Multiplicative factor of learning rate decay per epoch. Must be in range (0, 1). Typical values: 0.95-0.99 for slow decay, 0.9-0.95 for moderate decay.last_epoch (
int) – The index of the last epoch. Used for resuming training. Default: 0.
Notes
The learning rate at epoch \(t\) is computed as:
\[\eta_t = \eta_0 \cdot \gamma^t\]where \(\eta_0\) is the initial learning rate (base_lr) and \(t\) is the current epoch number.
Key characteristics:
Smooth exponential decay every epoch
Learning rate decreases continuously
Simple one-parameter control (gamma)
Decay rate is constant in logarithmic scale
Gamma selection guidelines:
gamma=0.95: Moderate decay, lr halves every ~14 epochs
gamma=0.96: Gentle decay, lr halves every ~17 epochs
gamma=0.98: Slow decay, lr halves every ~35 epochs
gamma=0.99: Very slow decay, lr halves every ~69 epochs
When to use:
When you want smooth, continuous learning rate reduction
For fine-tuning with gradual decay
When step-based schedules are too abrupt
For long training runs with gradual convergence
Practical guidance on decaying the learning rate during stochastic gradient training is given in [1] and [2].
Examples
Basic exponential decay:
>>> import braintools >>> import brainstate >>> >>> model = brainstate.nn.Linear(10, 5) >>> # Decay by 0.95 each epoch >>> scheduler = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.95) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> for epoch in range(20): ... # Training code ... if epoch % 5 == 0: ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.6f}") ... scheduler.step() Epoch 0: lr = 0.100000 Epoch 5: lr = 0.077378 Epoch 10: lr = 0.059874 Epoch 15: lr = 0.046329
Slow decay for fine-tuning:
>>> # Very gentle decay with gamma=0.99 >>> scheduler = braintools.optim.ExponentialLR(base_lr=0.001, gamma=0.99) >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Gradients of a toy loss w.r.t. the registered weights >>> import jax.numpy as jnp >>> x = jnp.ones((4, 10)) >>> def loss_fn(): ... return jnp.sum(model(x) ** 2) >>> grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))() >>> for epoch in range(100): ... optimizer.step(grads) ... scheduler.step() >>> # After 100 epochs: lr = 0.001 * 0.99^100 >>> print(f"{optimizer.current_lr:.6f}") 0.000366
Moderate decay for standard training:
>>> # Moderate decay with gamma=0.96 >>> scheduler = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.96) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9, weight_decay=1e-4) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> for epoch in range(50): ... optimizer.step(grads) ... scheduler.step() >>> # lr smoothly decreases from 0.1 to 0.1 * 0.96^50 >>> print(f"{optimizer.current_lr:.4f}") 0.0130
Combining with warmup:
>>> # Warmup followed by exponential decay >>> warmup = braintools.optim.LinearLR( ... base_lr=0.01, ... start_factor=0.1, ... end_factor=1.0, ... total_iters=5 ... ) >>> decay = braintools.optim.ExponentialLR(base_lr=0.01, gamma=0.95) >>> scheduler = braintools.optim.ChainedScheduler([warmup, decay]) >>> >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> for epoch in range(100): ... optimizer.step(grads) ... scheduler.step()
Using with different optimizers:
>>> # Works with any optimizer >>> scheduler = braintools.optim.ExponentialLR(base_lr=0.001, gamma=0.98) >>> >>> # With Adam >>> adam_opt = braintools.optim.Adam(lr=scheduler) >>> _ = adam_opt.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Or with RMSprop >>> model2 = brainstate.nn.Linear(10, 5) >>> scheduler2 = braintools.optim.ExponentialLR(base_lr=0.001, gamma=0.98) >>> rmsprop_opt = braintools.optim.RMSprop(lr=scheduler2) >>> _ = rmsprop_opt.register_trainable_weights(model2.states(brainstate.ParamState))
Saving and loading state:
>>> scheduler = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.95) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Train for some epochs >>> for epoch in range(50): ... scheduler.step() >>> >>> # Save checkpoint >>> checkpoint = { ... 'epoch': 50, ... 'scheduler': scheduler.state_dict(), ... } >>> >>> # Resume training >>> new_scheduler = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.95) >>> new_scheduler.load_state_dict(checkpoint['scheduler']) >>> # lr is restored to 0.1 * 0.95^50 >>> print(f"{new_scheduler.get_last_lr()[0]:.6f}") 0.007694
Aggressive decay:
>>> # Fast decay with gamma=0.9 >>> scheduler = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.9) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> for epoch in range(30): ... optimizer.step(grads) ... scheduler.step() >>> # After 30 epochs: lr = 0.1 * 0.9^30 >>> print(f"{optimizer.current_lr:.5f}") 0.00424
See also
StepLRStep-wise learning rate decay
CosineAnnealingLRCosine annealing schedule
MultiStepLRMulti-step learning rate decay
References