StepLR#
- class braintools.optim.StepLR(base_lr=0.001, step_size=30, gamma=0.1, last_epoch=0)#
Step learning rate scheduler - Decays learning rate by gamma every step_size epochs.
StepLR multiplies the learning rate by gamma at regular intervals (every step_size epochs), creating a staircase decay pattern. This is one of the most commonly used learning rate schedules for training deep neural networks.
- Parameters:
base_lr (
float|List[float]) – Initial learning rate(s). Can be a single float or a list of floats for multiple parameter groups. Default: 1e-3.step_size (
int) – Period of learning rate decay in epochs. The learning rate will be multiplied by gamma every step_size epochs. Default: 30.gamma (
float) – Multiplicative factor of learning rate decay. Must be in range (0, 1]. Default: 0.1.last_epoch (
int) – The index of the last epoch. Used for resuming training. Default: 0.
Notes
The learning rate at epoch \(t\) is computed as:
\[\eta_t = \eta_0 \cdot \gamma^{\lfloor t / \text{step_size} \rfloor}\]where \(\eta_0\) is the initial learning rate (base_lr), and \(\lfloor \cdot \rfloor\) denotes the floor function.
Key characteristics:
Creates discrete “steps” in the learning rate schedule
Widely used for training image classification models
Simple to tune with only two hyperparameters
Works well when combined with momentum-based optimizers
Common step_size values:
ImageNet training: step_size=30, total_epochs=90 (decay at epochs 30, 60)
CIFAR training: step_size=50, total_epochs=150 (decay at epochs 50, 100)
Step decay schedules of this kind were used to train AlexNet [1] and ResNet [2].
Examples
Basic usage with SGD:
>>> import braintools >>> import brainstate >>> >>> # Create model and scheduler >>> model = brainstate.nn.Linear(10, 5) >>> scheduler = braintools.optim.StepLR(base_lr=0.1, step_size=30, gamma=0.1) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Training loop (print the LR for the epoch, then advance the schedule) >>> for epoch in range(90): ... # ... training code ... ... if epoch in [0, 29, 30, 59, 60, 89]: ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.6f}") ... scheduler.step() Epoch 0: lr = 0.100000 Epoch 29: lr = 0.100000 Epoch 30: lr = 0.010000 Epoch 59: lr = 0.010000 Epoch 60: lr = 0.001000 Epoch 89: lr = 0.001000
Using with Adam optimizer:
>>> scheduler = braintools.optim.StepLR(base_lr=0.001, step_size=10, gamma=0.5) >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> for epoch in range(25): ... if epoch in [0, 10, 20]: ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.6f}") ... scheduler.step() Epoch 0: lr = 0.001000 Epoch 10: lr = 0.000500 Epoch 20: lr = 0.000250
Custom decay schedule:
>>> # Aggressive decay every 5 epochs >>> scheduler = braintools.optim.StepLR(base_lr=0.1, step_size=5, gamma=0.5) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # After 15 epochs: lr = 0.1 * 0.5^3 = 0.0125 >>> for epoch in range(15): ... scheduler.step() >>> print(f"{optimizer.current_lr:.4f}") 0.0125
Saving and loading scheduler state:
>>> scheduler = braintools.optim.StepLR(base_lr=0.1, step_size=30, gamma=0.1) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Train for some epochs >>> for epoch in range(50): ... scheduler.step() >>> >>> # Save checkpoint >>> checkpoint = { ... 'epoch': 50, ... 'optimizer': optimizer.state_dict(), ... 'scheduler': scheduler.state_dict(), ... } >>> >>> # Later, resume training >>> new_scheduler = braintools.optim.StepLR(base_lr=0.1, step_size=30, gamma=0.1) >>> new_scheduler.load_state_dict(checkpoint['scheduler']) >>> # Continue from epoch 50 >>> int(new_scheduler.last_epoch.value) 50
Multiple parameter groups:
>>> # Different learning rates for different layers >>> scheduler = braintools.optim.StepLR( ... base_lr=[0.1, 0.01], # Different base lr for each group ... step_size=30, ... gamma=0.1 ... ) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> # Both groups decay by gamma every step_size epochs
Complete training example:
>>> import jax.numpy as jnp >>> >>> brainstate.random.seed(0) >>> model = brainstate.nn.Linear(10, 5) >>> scheduler = braintools.optim.StepLR(base_lr=0.1, step_size=30, gamma=0.1) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> x = brainstate.random.randn(32, 10) >>> y = brainstate.random.randn(32, 5) >>> def loss_fn(): ... return jnp.mean((model(x) - y) ** 2) >>> >>> @brainstate.transform.jit ... def train_epoch(): ... grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))() ... optimizer.step(grads) >>> >>> for epoch in range(90): ... train_epoch() ... scheduler.step() >>> print(f"lr after 90 epochs = {optimizer.current_lr:.4f}") lr after 90 epochs = 0.0001
See also
MultiStepLRDecay learning rate at specific milestone epochs
ExponentialLRExponential decay of learning rate
CosineAnnealingLRCosine annealing schedule
References