ConstantLR#
- class braintools.optim.ConstantLR(base_lr=0.001, factor=0.3333333333333333, total_iters=5, last_epoch=0)#
Constant learning rate scheduler - Multiplies learning rate by a constant factor.
ConstantLR multiplies the base learning rate by a constant factor for a specified number of epochs (total_iters), then returns to the original base learning rate. This is useful for implementing warmup phases or temporary learning rate adjustments.
- Parameters:
base_lr (
float|List[float]) – Initial learning rate(s). Can be a single float or a list of floats for multiple parameter groups. Default: 1e-3.factor (
float) – Multiplicative factor applied to base_lr for the first total_iters epochs. Must be in range (0, 1]. Default: 1/3.total_iters (
int) – Number of epochs to apply the factor. After total_iters epochs, the learning rate returns to base_lr. Default: 5.last_epoch (
int) – The index of the last epoch. Used for resuming training. Default: 0.
Notes
The learning rate at epoch \(t\) is computed as:
\[\begin{split}\eta_t = \begin{cases} \eta_0 \cdot \text{factor} & \text{if } t < \text{total_iters} \\ \eta_0 & \text{otherwise} \end{cases}\end{split}\]where \(\eta_0\) is the base learning rate.
Key characteristics:
Simple two-phase learning rate schedule
Commonly used for warmup with constant reduced lr (constant warmup in [1])
Automatically returns to base_lr after warmup period
No gradual transition (step change at total_iters)
Comparison with LinearLR:
ConstantLR: Instant jump from (factor * base_lr) to base_lr at total_iters
LinearLR: Smooth linear transition from start_factor to end_factor
For learning-rate range exploration beyond simple two-phase schedules, see cyclical learning rates [2].
Examples
Basic constant warmup:
>>> import braintools >>> import brainstate >>> >>> model = brainstate.nn.Linear(10, 5) >>> # Use 0.5 * base_lr for first 10 epochs, then full base_lr >>> scheduler = braintools.optim.ConstantLR( ... base_lr=0.001, ... factor=0.5, ... total_iters=10 ... ) >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Epochs 0-9: lr = 0.0005 >>> # Epochs 10+: lr = 0.001 >>> for epoch in range(12): ... if epoch in [0, 9, 10]: ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.4f}") ... scheduler.step() Epoch 0: lr = 0.0005 Epoch 9: lr = 0.0005 Epoch 10: lr = 0.0010
Default warmup configuration:
>>> # Default: lr = base_lr/3 for 5 epochs, then lr = base_lr >>> scheduler = braintools.optim.ConstantLR() >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Gradients of a toy loss w.r.t. the registered weights >>> import jax.numpy as jnp >>> x = jnp.ones((4, 10)) >>> def loss_fn(): ... return jnp.sum(model(x) ** 2) >>> grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))() >>> # First 5 epochs: lr ≈ 0.000333; remaining epochs: lr = 0.001 >>> for epoch in range(7): ... optimizer.step(grads) ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.6f}") ... scheduler.step() Epoch 0: lr = 0.000333 Epoch 1: lr = 0.000333 Epoch 2: lr = 0.000333 Epoch 3: lr = 0.000333 Epoch 4: lr = 0.000333 Epoch 5: lr = 0.001000 Epoch 6: lr = 0.001000
Short warmup for fine-tuning:
>>> # Use 20% of base_lr for first 3 epochs >>> scheduler = braintools.optim.ConstantLR( ... base_lr=0.0001, ... factor=0.2, ... total_iters=3 ... ) >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Epochs 0-2: lr = 0.00002 >>> # Epochs 3+: lr = 0.0001 >>> print(f"{optimizer.current_lr:.5f}") 0.00002
Combining with StepLR:
>>> # Warmup, then step decay >>> warmup = braintools.optim.ConstantLR( ... base_lr=0.1, ... factor=0.1, ... total_iters=5 ... ) >>> decay = braintools.optim.StepLR(base_lr=0.1, step_size=30, gamma=0.1) >>> scheduler = braintools.optim.ChainedScheduler([warmup, decay]) >>> >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Epochs 0-4: warmup at 0.01; 5-29: 0.1; 30-59: 0.01; 60+: 0.001 >>> for epoch in range(90): ... optimizer.step(grads) ... if epoch in [0, 5, 30, 60]: ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.3f}") ... scheduler.step() Epoch 0: lr = 0.010 Epoch 5: lr = 0.100 Epoch 30: lr = 0.010 Epoch 60: lr = 0.001
Conservative start for transfer learning:
>>> # Start with very low lr for stability >>> scheduler = braintools.optim.ConstantLR( ... base_lr=0.001, ... factor=0.01, ... total_iters=10 ... ) >>> optimizer = braintools.optim.Adam(lr=scheduler, weight_decay=1e-5) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # First 10 epochs: lr = 0.00001 (conservative) >>> # Remaining epochs: lr = 0.001 (normal training) >>> print(f"{optimizer.current_lr:.5f}") 0.00001
Multiple parameter groups:
>>> # Different base_lr for different layers >>> scheduler = braintools.optim.ConstantLR( ... base_lr=[0.1, 0.01], ... factor=0.1, ... total_iters=5 ... ) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> # Both groups use factor=0.1 for first 5 epochs
Complete training workflow:
>>> import jax.numpy as jnp >>> >>> brainstate.random.seed(0) >>> model = brainstate.nn.Linear(10, 5) >>> scheduler = braintools.optim.ConstantLR( ... base_lr=0.01, ... factor=0.1, ... total_iters=5 ... ) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> x = brainstate.random.randn(32, 10) >>> y = brainstate.random.randn(32, 5) >>> def loss_fn(): ... return jnp.mean((model(x) - y) ** 2) >>> >>> @brainstate.transform.jit ... def train_step(): ... grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))() ... optimizer.step(grads) >>> >>> for epoch in range(50): ... train_step() ... if epoch in [0, 4, 5, 10]: ... print(f"Epoch {epoch}: lr = {optimizer.current_lr:.3f}") ... scheduler.step() Epoch 0: lr = 0.001 Epoch 4: lr = 0.001 Epoch 5: lr = 0.010 Epoch 10: lr = 0.010
See also
LinearLRLinearly scale learning rate (smooth transition)
WarmupSchedulerAlternative warmup implementation
ChainedSchedulerCombine multiple schedulers
References