SequentialLR#
- class braintools.optim.SequentialLR(schedulers, milestones, last_epoch=0)#
Sequential learning rate scheduler - Chains multiple schedulers based on epoch milestones.
SequentialLR allows you to chain multiple learning rate schedulers, with each scheduler being active during specific epoch ranges defined by milestones. This is particularly useful for complex training strategies that require different learning rate policies at different stages of training, such as the warmup phases used for large-batch SGD [1], adaptive optimizers [2], and large-batch BERT training [3].
- Parameters:
schedulers (
List[LRScheduler]) – List of schedulers to be sequentially applied. The number of schedulers should belen(milestones) + 1.milestones (
List[int]) – List of epoch indices that define when to switch schedulers. Must be in ascending order. For n milestones, you need n+1 schedulers.last_epoch (
int) – The index of the last epoch. Default: 0.
Notes
Scheduler Switching Logic:
Given milestones [m1, m2, …, mn] and schedulers [s0, s1, …, sn]:
Epochs [0, m1): uses scheduler s0
Epochs [m1, m2): uses scheduler s1
…
Epochs [mn, ∞): uses scheduler sn
JIT Compatibility:
This implementation is JIT-compatible through the use of JAX operations for scheduler selection. The scheduler index is computed using
jnp.searchsortedfor efficient milestone-based switching.Important Considerations:
Each scheduler should be initialized with the appropriate
base_lrthat matches your intended learning rate at the transition point.The
last_epochparameter of individual schedulers is managed internally.When saving/loading state, all schedulers’ states are preserved.
The contained schedulers cannot be a
ReduceLROnPlateau, whosestep(metric)signature is incompatible with sequential stepping.
Examples
Basic usage with warmup and decay:
>>> import braintools >>> import brainstate >>> >>> model = brainstate.nn.Linear(10, 5) >>> >>> # Warmup for 5 epochs, then exponential decay >>> warmup = braintools.optim.LinearLR( ... base_lr=0.1, ... start_factor=0.01, ... end_factor=1.0, ... total_iters=5 ... ) >>> decay = braintools.optim.ExponentialLR( ... base_lr=0.1, ... gamma=0.95 ... ) >>> scheduler = braintools.optim.SequentialLR( ... schedulers=[warmup, decay], ... milestones=[5] ... ) >>> >>> optimizer = braintools.optim.Adam(lr=scheduler) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> for epoch in range(8): ... print(f"Epoch {epoch}: lr = {scheduler.get_last_lr()[0]:.5f}") ... scheduler.step() Epoch 0: lr = 0.00100 Epoch 1: lr = 0.02080 Epoch 2: lr = 0.04060 Epoch 3: lr = 0.06040 Epoch 4: lr = 0.08020 Epoch 5: lr = 0.10000 Epoch 6: lr = 0.09500 Epoch 7: lr = 0.09025
Three-phase training (warmup → training → fine-tuning):
>>> import braintools >>> >>> # Phase 1: Warmup (epochs 0-5) >>> warmup = braintools.optim.LinearLR( ... base_lr=0.001, ... start_factor=0.1, ... end_factor=1.0, ... total_iters=5 ... ) >>> >>> # Phase 2: Main training (epochs 5-80) >>> main_training = braintools.optim.CosineAnnealingLR( ... base_lr=0.001, ... T_max=75, ... eta_min=0.0001 ... ) >>> >>> # Phase 3: Fine-tuning (epochs 80+) >>> fine_tuning = braintools.optim.ConstantLR( ... base_lr=0.0001, ... factor=1.0 ... ) >>> >>> scheduler = braintools.optim.SequentialLR( ... schedulers=[warmup, main_training, fine_tuning], ... milestones=[5, 80] ... ) >>> for epoch in range(100): ... if epoch in (0, 5, 40, 79, 80, 99): ... print(f"Epoch {epoch}: lr = {scheduler.get_last_lr()[0]:.6f}") ... scheduler.step() Epoch 0: lr = 0.000100 Epoch 5: lr = 0.001000 Epoch 40: lr = 0.000597 Epoch 79: lr = 0.000100 Epoch 80: lr = 0.000100 Epoch 99: lr = 0.000100
Complex schedule for transformer training:
>>> import braintools >>> >>> # Transformer training schedule >>> # 1. Linear warmup >>> warmup = braintools.optim.LinearLR( ... base_lr=0.0005, ... start_factor=0.0, ... end_factor=1.0, ... total_iters=4000 # 4000 steps ... ) >>> >>> # 2. Constant learning rate >>> constant = braintools.optim.ConstantLR( ... base_lr=0.0005, ... factor=1.0 ... ) >>> >>> # 3. Cosine decay to near zero >>> cosine_decay = braintools.optim.CosineAnnealingLR( ... base_lr=0.0005, ... T_max=20000, ... eta_min=1e-6 ... ) >>> >>> scheduler = braintools.optim.SequentialLR( ... schedulers=[warmup, constant, cosine_decay], ... milestones=[4000, 10000] ... ) >>> # Jump straight to a few representative steps >>> for step in (0, 2000, 4000, 10000, 20000, 30000): ... scheduler.step(step) ... print(f"Step {step}: lr = {scheduler.get_last_lr()[0]:.6f}") Step 0: lr = 0.000000 Step 2000: lr = 0.000250 Step 4000: lr = 0.000500 Step 10000: lr = 0.000500 Step 20000: lr = 0.000250 Step 30000: lr = 0.000001
State persistence across training sessions:
>>> import braintools >>> >>> def make_scheduler(): ... scheduler1 = braintools.optim.LinearLR( ... base_lr=0.1, start_factor=0.1, total_iters=50) ... scheduler2 = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.9) ... return braintools.optim.SequentialLR( ... schedulers=[scheduler1, scheduler2], ... milestones=[50] ... ) >>> >>> # Train for some epochs, then save the scheduler state >>> scheduler = make_scheduler() >>> for epoch in range(60): ... scheduler.step() >>> checkpoint = { ... 'epoch': 60, ... 'scheduler': scheduler.state_dict(), ... } >>> print(f"{scheduler.get_last_lr()[0]:.6f}") 0.034868 >>> >>> # Resume training with freshly built schedulers >>> resumed = make_scheduler() >>> resumed.load_state_dict(checkpoint['scheduler']) >>> print(f"{resumed.get_last_lr()[0]:.6f}") 0.034868
Using with different optimizers:
>>> import braintools >>> >>> warmup_sched = braintools.optim.LinearLR( ... base_lr=0.01, start_factor=0.1, total_iters=10) >>> main_sched = braintools.optim.StepLR(base_lr=0.01, step_size=30, gamma=0.1) >>> >>> # Works with any optimizer >>> scheduler = braintools.optim.SequentialLR( ... schedulers=[warmup_sched, main_sched], ... milestones=[10] ... ) >>> >>> # With SGD >>> sgd_opt = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> >>> # With Adam >>> adam_opt = braintools.optim.Adam(lr=scheduler) >>> >>> # With LAMB for large batch training >>> lamb_opt = braintools.optim.Lamb(lr=scheduler) >>> type(lamb_opt).__name__ 'Lamb'
Monitoring scheduler transitions:
>>> import braintools >>> >>> sched1 = braintools.optim.LinearLR(base_lr=0.1, start_factor=0.1, total_iters=10) >>> sched2 = braintools.optim.ConstantLR(base_lr=0.1, factor=1.0) >>> sched3 = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.9) >>> scheduler = braintools.optim.SequentialLR( ... schedulers=[sched1, sched2, sched3], ... milestones=[10, 20] ... ) >>> >>> previous = None >>> for epoch in range(30): ... scheduler.step(epoch) ... active_scheduler = int(scheduler.current_scheduler_idx) ... # Report only the epochs where a new scheduler takes over ... if active_scheduler != previous: ... print(f"Epoch {epoch}: LR={scheduler.get_last_lr()[0]:.6f}, " ... f"active scheduler: {active_scheduler}") ... previous = active_scheduler Epoch 0: LR=0.010000, active scheduler: 0 Epoch 10: LR=0.100000, active scheduler: 1 Epoch 20: LR=0.100000, active scheduler: 2
See also
ChainedSchedulerApplies multiple schedulers simultaneously
LinearLRLinear learning rate schedule (good for warmup)
CosineAnnealingLRCosine annealing schedule
ExponentialLRExponential decay schedule
StepLRStep-wise decay schedule
References