SequentialLR#

class braintools.optim.SequentialLR(schedulers, milestones, last_epoch=0)#

Sequential learning rate scheduler - Chains multiple schedulers based on epoch milestones.

SequentialLR allows you to chain multiple learning rate schedulers, with each scheduler being active during specific epoch ranges defined by milestones. This is particularly useful for complex training strategies that require different learning rate policies at different stages of training, such as the warmup phases used for large-batch SGD [1], adaptive optimizers [2], and large-batch BERT training [3].

Parameters:
  • schedulers (List[LRScheduler]) – List of schedulers to be sequentially applied. The number of schedulers should be len(milestones) + 1.

  • milestones (List[int]) – List of epoch indices that define when to switch schedulers. Must be in ascending order. For n milestones, you need n+1 schedulers.

  • last_epoch (int) – The index of the last epoch. Default: 0.

Notes

Scheduler Switching Logic:

Given milestones [m1, m2, …, mn] and schedulers [s0, s1, …, sn]:

  • Epochs [0, m1): uses scheduler s0

  • Epochs [m1, m2): uses scheduler s1

  • …

  • Epochs [mn, ∞): uses scheduler sn

JIT Compatibility:

This implementation is JIT-compatible through the use of JAX operations for scheduler selection. The scheduler index is computed using jnp.searchsorted for efficient milestone-based switching.

Important Considerations:

  1. Each scheduler should be initialized with the appropriate base_lr that matches your intended learning rate at the transition point.

  2. The last_epoch parameter of individual schedulers is managed internally.

  3. When saving/loading state, all schedulers’ states are preserved.

  4. The contained schedulers cannot be a ReduceLROnPlateau, whose step(metric) signature is incompatible with sequential stepping.

Examples

Basic usage with warmup and decay:

>>> import braintools
>>> import brainstate
>>>
>>> model = brainstate.nn.Linear(10, 5)
>>>
>>> # Warmup for 5 epochs, then exponential decay
>>> warmup = braintools.optim.LinearLR(
...     base_lr=0.1,
...     start_factor=0.01,
...     end_factor=1.0,
...     total_iters=5
... )
>>> decay = braintools.optim.ExponentialLR(
...     base_lr=0.1,
...     gamma=0.95
... )
>>> scheduler = braintools.optim.SequentialLR(
...     schedulers=[warmup, decay],
...     milestones=[5]
... )
>>>
>>> optimizer = braintools.optim.Adam(lr=scheduler)
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
>>>
>>> for epoch in range(8):
...     print(f"Epoch {epoch}: lr = {scheduler.get_last_lr()[0]:.5f}")
...     scheduler.step()
Epoch 0: lr = 0.00100
Epoch 1: lr = 0.02080
Epoch 2: lr = 0.04060
Epoch 3: lr = 0.06040
Epoch 4: lr = 0.08020
Epoch 5: lr = 0.10000
Epoch 6: lr = 0.09500
Epoch 7: lr = 0.09025

Three-phase training (warmup → training → fine-tuning):

>>> import braintools
>>>
>>> # Phase 1: Warmup (epochs 0-5)
>>> warmup = braintools.optim.LinearLR(
...     base_lr=0.001,
...     start_factor=0.1,
...     end_factor=1.0,
...     total_iters=5
... )
>>>
>>> # Phase 2: Main training (epochs 5-80)
>>> main_training = braintools.optim.CosineAnnealingLR(
...     base_lr=0.001,
...     T_max=75,
...     eta_min=0.0001
... )
>>>
>>> # Phase 3: Fine-tuning (epochs 80+)
>>> fine_tuning = braintools.optim.ConstantLR(
...     base_lr=0.0001,
...     factor=1.0
... )
>>>
>>> scheduler = braintools.optim.SequentialLR(
...     schedulers=[warmup, main_training, fine_tuning],
...     milestones=[5, 80]
... )
>>> for epoch in range(100):
...     if epoch in (0, 5, 40, 79, 80, 99):
...         print(f"Epoch {epoch}: lr = {scheduler.get_last_lr()[0]:.6f}")
...     scheduler.step()
Epoch 0: lr = 0.000100
Epoch 5: lr = 0.001000
Epoch 40: lr = 0.000597
Epoch 79: lr = 0.000100
Epoch 80: lr = 0.000100
Epoch 99: lr = 0.000100

Complex schedule for transformer training:

>>> import braintools
>>>
>>> # Transformer training schedule
>>> # 1. Linear warmup
>>> warmup = braintools.optim.LinearLR(
...     base_lr=0.0005,
...     start_factor=0.0,
...     end_factor=1.0,
...     total_iters=4000  # 4000 steps
... )
>>>
>>> # 2. Constant learning rate
>>> constant = braintools.optim.ConstantLR(
...     base_lr=0.0005,
...     factor=1.0
... )
>>>
>>> # 3. Cosine decay to near zero
>>> cosine_decay = braintools.optim.CosineAnnealingLR(
...     base_lr=0.0005,
...     T_max=20000,
...     eta_min=1e-6
... )
>>>
>>> scheduler = braintools.optim.SequentialLR(
...     schedulers=[warmup, constant, cosine_decay],
...     milestones=[4000, 10000]
... )
>>> # Jump straight to a few representative steps
>>> for step in (0, 2000, 4000, 10000, 20000, 30000):
...     scheduler.step(step)
...     print(f"Step {step}: lr = {scheduler.get_last_lr()[0]:.6f}")
Step 0: lr = 0.000000
Step 2000: lr = 0.000250
Step 4000: lr = 0.000500
Step 10000: lr = 0.000500
Step 20000: lr = 0.000250
Step 30000: lr = 0.000001

State persistence across training sessions:

>>> import braintools
>>>
>>> def make_scheduler():
...     scheduler1 = braintools.optim.LinearLR(
...         base_lr=0.1, start_factor=0.1, total_iters=50)
...     scheduler2 = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.9)
...     return braintools.optim.SequentialLR(
...         schedulers=[scheduler1, scheduler2],
...         milestones=[50]
...     )
>>>
>>> # Train for some epochs, then save the scheduler state
>>> scheduler = make_scheduler()
>>> for epoch in range(60):
...     scheduler.step()
>>> checkpoint = {
...     'epoch': 60,
...     'scheduler': scheduler.state_dict(),
... }
>>> print(f"{scheduler.get_last_lr()[0]:.6f}")
0.034868
>>>
>>> # Resume training with freshly built schedulers
>>> resumed = make_scheduler()
>>> resumed.load_state_dict(checkpoint['scheduler'])
>>> print(f"{resumed.get_last_lr()[0]:.6f}")
0.034868

Using with different optimizers:

>>> import braintools
>>>
>>> warmup_sched = braintools.optim.LinearLR(
...     base_lr=0.01, start_factor=0.1, total_iters=10)
>>> main_sched = braintools.optim.StepLR(base_lr=0.01, step_size=30, gamma=0.1)
>>>
>>> # Works with any optimizer
>>> scheduler = braintools.optim.SequentialLR(
...     schedulers=[warmup_sched, main_sched],
...     milestones=[10]
... )
>>>
>>> # With SGD
>>> sgd_opt = braintools.optim.SGD(lr=scheduler, momentum=0.9)
>>>
>>> # With Adam
>>> adam_opt = braintools.optim.Adam(lr=scheduler)
>>>
>>> # With LAMB for large batch training
>>> lamb_opt = braintools.optim.Lamb(lr=scheduler)
>>> type(lamb_opt).__name__
'Lamb'

Monitoring scheduler transitions:

>>> import braintools
>>>
>>> sched1 = braintools.optim.LinearLR(base_lr=0.1, start_factor=0.1, total_iters=10)
>>> sched2 = braintools.optim.ConstantLR(base_lr=0.1, factor=1.0)
>>> sched3 = braintools.optim.ExponentialLR(base_lr=0.1, gamma=0.9)
>>> scheduler = braintools.optim.SequentialLR(
...     schedulers=[sched1, sched2, sched3],
...     milestones=[10, 20]
... )
>>>
>>> previous = None
>>> for epoch in range(30):
...     scheduler.step(epoch)
...     active_scheduler = int(scheduler.current_scheduler_idx)
...     # Report only the epochs where a new scheduler takes over
...     if active_scheduler != previous:
...         print(f"Epoch {epoch}: LR={scheduler.get_last_lr()[0]:.6f}, "
...               f"active scheduler: {active_scheduler}")
...     previous = active_scheduler
Epoch 0: LR=0.010000, active scheduler: 0
Epoch 10: LR=0.100000, active scheduler: 1
Epoch 20: LR=0.100000, active scheduler: 2

See also

ChainedScheduler

Applies multiple schedulers simultaneously

LinearLR

Linear learning rate schedule (good for warmup)

CosineAnnealingLR

Cosine annealing schedule

ExponentialLR

Exponential decay schedule

StepLR

Step-wise decay schedule

References

get_lr()[source]#

Get learning rate from the current (already milestone-offset) scheduler.

load_state_dict(state_dict)[source]#

Load scheduler state from dictionary.

state_dict()[source]#

Return scheduler state as dictionary.

step(epoch=None)[source]#

Update learning rate.