PiecewiseConstantSchedule#

class braintools.optim.PiecewiseConstantSchedule(base_lr=0.001, boundaries=None, values=None, last_epoch=0)#

Piecewise constant learning rate schedule with step-wise transitions.

PiecewiseConstantSchedule implements a learning rate schedule where the learning rate remains constant within specified intervals and changes abruptly at predefined boundaries. This creates a step function that’s useful for stage-based training where different phases require different learning rates.

This scheduler is particularly effective for: - Multi-stage training pipelines - Transfer learning with progressive unfreezing - Training with curriculum learning - Reproducing specific research papers with fixed schedules (e.g., [1], [2]) - Budget-constrained training with predetermined phases

Parameters:
  • base_lr (float | List[float]) – Reference learning rate(s). Only the number of entries is used (to set the number of parameter groups); the magnitude is ignored because values are absolute learning rates, not multipliers. Pass a list to configure multiple parameter groups. Default: 1e-3.

  • boundaries (List[int] | None) – Step indices where the learning rate changes. Must be sorted in ascending order. The schedule will have len(boundaries) + 1 distinct phases. Default: [1000, 2000].

  • values (List[float] | None) – Absolute learning rate applied in each phase (not multipliers of base_lr). Must have exactly len(boundaries) + 1 elements. The i-th value applies from boundary[i-1] to boundary[i]. Default: [1.0, 0.1, 0.01].

  • last_epoch (int) – The index of the last epoch. Used when resuming training. Default: 0.

Notes

Mathematical Formulation:

The learning rate at step t is the (absolute) phase value:

\[\eta_t = v_i\]

where \(v_i\) is determined by:

\[\begin{split}v_i = \begin{cases} \text{values}[0] & \text{if } t < \text{boundaries}[0] \\ \text{values}[1] & \text{if } \text{boundaries}[0] \leq t < \text{boundaries}[1] \\ ... & ... \\ \text{values}[n] & \text{if } t \geq \text{boundaries}[n-1] \end{cases}\end{split}\]

Schedule Structure:

Given boundaries [b1, b2, …, bn] and values [v0, v1, …, vn]:

  • Steps [0, b1): learning_rate = v0

  • Steps [b1, b2): learning_rate = v1

  • Steps [b2, b3): learning_rate = v2

  • …

  • Steps [bn, ∞): learning_rate = vn

JIT Compatibility:

This implementation is JIT-compatible through the use of jnp.searchsorted for efficient boundary-based value selection without Python conditionals.

For smoother alternatives to step-wise schedules, see cyclical learning rates [3] and the training refinements surveyed in [4].

Examples

Classic ImageNet training schedule:

>>> import braintools
>>> import brainstate
>>>
>>> model = brainstate.nn.Linear(10, 5)
>>>
>>> # ResNet50 on ImageNet schedule
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=[30, 60, 80],  # Epochs to decrease LR
...     values=[0.1, 0.01, 0.001, 0.0001]  # absolute LRs (not multipliers of base_lr)
... )
>>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9)
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
>>>
>>> # Train for 90 epochs total
>>> # LR: 0.1 (epochs 0-29), 0.01 (30-59), 0.001 (60-79), 0.0001 (80-89)
>>> for epoch in range(90):
...     if epoch in (0, 29, 30, 60, 80):
...         print(f"Epoch {epoch}: LR = {scheduler.get_last_lr()[0]:.6f}")
...     scheduler.step()
Epoch 0: LR = 0.100000
Epoch 29: LR = 0.100000
Epoch 30: LR = 0.010000
Epoch 60: LR = 0.001000
Epoch 80: LR = 0.000100

Transfer learning with progressive unfreezing:

>>> import braintools
>>>
>>> # Unfreeze layers progressively with different LRs
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=1e-3,
...     boundaries=[5, 10, 15],  # Unfreeze stages
...     values=[1e-5, 1e-4, 5e-4, 1e-3]  # Gradual increase
... )
>>>
>>> # Stage 1 (0-4): Only train head, very low LR
>>> # Stage 2 (5-9): Unfreeze top layers
>>> # Stage 3 (10-14): Unfreeze middle layers
>>> # Stage 4 (15+): Full model training
>>> stages = {5: 'top layers', 10: 'middle layers', 15: 'all layers'}
>>> for epoch in range(20):
...     if epoch in stages:
...         # (the corresponding layers would be unfrozen here)
...         print(f"Epoch {epoch}: unfreeze {stages[epoch]}, "
...               f"LR = {scheduler.get_last_lr()[0]:.0e}")
...     scheduler.step()
Epoch 5: unfreeze top layers, LR = 1e-04
Epoch 10: unfreeze middle layers, LR = 5e-04
Epoch 15: unfreeze all layers, LR = 1e-03

Budget-aware training schedule:

>>> import braintools
>>>
>>> # Training with computational budget constraints
>>> # Fast initial training, then careful fine-tuning
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.01,
...     boundaries=[100, 500, 800],
...     values=[0.1, 0.01, 0.001, 0.0001]  # Aggressive start, careful end
... )
>>>
>>> # Steps 0-99: Fast exploration (LR=0.1)
>>> # Steps 100-499: Normal training (LR=0.01)
>>> # Steps 500-799: Fine-tuning (LR=0.001)
>>> # Steps 800+: Final refinement (LR=0.0001)
>>> for step in (0, 100, 500, 800):
...     scheduler.step(step)
...     print(f"Step {step}: LR = {scheduler.get_last_lr()[0]:.4f}")
Step 0: LR = 0.1000
Step 100: LR = 0.0100
Step 500: LR = 0.0010
Step 800: LR = 0.0001

Multi-phase curriculum learning:

>>> import braintools
>>>
>>> # Different learning rates for different curriculum stages
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=1e-3,
...     boundaries=[1000, 3000, 6000, 9000],
...     values=[1e-4, 5e-4, 1e-3, 5e-4, 1e-4]
... )
>>>
>>> curriculum_difficulties = [0.2, 0.4, 0.6, 0.8, 1.0]
>>>
>>> for step in (0, 1000, 3000, 6000, 9000):
...     # Determine curriculum difficulty
...     stage = sum(step >= b for b in scheduler.boundaries)
...     difficulty = curriculum_difficulties[stage]
...     # (a batch of this difficulty would be trained on here)
...     scheduler.step(step)
...     print(f"Step {step}: difficulty={difficulty}, "
...           f"LR = {scheduler.get_last_lr()[0]:.4f}")
Step 0: difficulty=0.2, LR = 0.0001
Step 1000: difficulty=0.4, LR = 0.0005
Step 3000: difficulty=0.6, LR = 0.0010
Step 6000: difficulty=0.8, LR = 0.0005
Step 9000: difficulty=1.0, LR = 0.0001

Reproducing paper schedules:

>>> import braintools
>>>
>>> # WideResNet schedule from paper
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=[60, 120, 160],  # Specific to WideResNet
...     values=[0.1, 0.02, 0.004, 0.0008]
... )
>>>
>>> # CIFAR training schedule from "Bag of Tricks"
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=[150, 250],
...     values=[0.1, 0.01, 0.001]
... )

Step-based (not epoch-based) scheduling:

>>> import braintools
>>>
>>> # Define boundaries in terms of training steps
>>> steps_per_epoch = 100  # e.g. len(train_loader)
>>> epoch_boundaries = [30, 60, 80]  # Desired epoch boundaries
>>> step_boundaries = [e * steps_per_epoch for e in epoch_boundaries]
>>> step_boundaries
[3000, 6000, 8000]
>>>
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=step_boundaries,
...     values=[0.1, 0.01, 0.001, 0.0001]
... )
>>>
>>> # A training loop calls ``scheduler.step()`` once per batch;
>>> # here we jump to the first step of epoch 30.
>>> global_step = 30 * steps_per_epoch
>>> scheduler.step(global_step)
>>> print(f"{scheduler.get_last_lr()[0]:.4f}")
0.0100

Combining with warmup:

>>> import braintools
>>>
>>> # Add warmup to piecewise schedule
>>> warmup_steps = 500
>>> main_boundaries = [5000, 10000, 15000]
>>>
>>> # Combine warmup with main schedule
>>> all_boundaries = [warmup_steps] + main_boundaries
>>> all_values = [0.001, 0.1, 0.01, 0.001, 0.0001]  # Low start for warmup
>>>
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=all_boundaries,
...     values=all_values
... )

Dynamic monitoring and adjustment:

>>> import braintools
>>>
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=[1000, 2000, 3000],
...     values=[0.1, 0.05, 0.01, 0.001]
... )
>>>
>>> for step in range(1, 4000):
...     old_lr = float(scheduler.get_last_lr()[0])
...     scheduler.step()
...     new_lr = float(scheduler.get_last_lr()[0])
...
...     # Detect LR changes (momentum or other optimizer
...     # states could optionally be reset here)
...     if old_lr != new_lr:
...         print(f"Step {step}: LR changed from {old_lr:.6f} to {new_lr:.6f}")
Step 1000: LR changed from 0.100000 to 0.050000
Step 2000: LR changed from 0.050000 to 0.010000
Step 3000: LR changed from 0.010000 to 0.001000

Research experimentation with multiple schedules:

>>> import braintools
>>>
>>> # Compare different decay strategies
>>> schedules = {
...     'aggressive': braintools.optim.PiecewiseConstantSchedule(
...         base_lr=0.1,
...         boundaries=[10, 20],
...         values=[0.1, 0.001, 0.00001]
...     ),
...     'conservative': braintools.optim.PiecewiseConstantSchedule(
...         base_lr=0.1,
...         boundaries=[30, 60],
...         values=[0.1, 0.05, 0.01]
...     ),
...     'multi_stage': braintools.optim.PiecewiseConstantSchedule(
...         base_lr=0.1,
...         boundaries=[10, 20, 30, 40],
...         values=[0.1, 0.08, 0.04, 0.01, 0.001]
...     )
... }
>>>
>>> # Compare the learning rate each schedule reaches at epoch 25
>>> for name, scheduler in schedules.items():
...     scheduler.step(25)
...     print(f"{name}: LR at epoch 25 = {scheduler.get_last_lr()[0]:.5f}")
aggressive: LR at epoch 25 = 0.00001
conservative: LR at epoch 25 = 0.10000
multi_stage: LR at epoch 25 = 0.04000

State persistence and checkpointing:

>>> import braintools
>>>
>>> # Save and restore schedule state
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=[1000, 2000],
...     values=[0.1, 0.01, 0.001]
... )
>>>
>>> # Train for some steps (jump straight to step 1500 here)
>>> scheduler.step(1500)
>>>
>>> # Save checkpoint
>>> checkpoint = {
...     'step': 1500,
...     'scheduler_state': scheduler.state_dict(),
... }
>>>
>>> # Later: restore and continue
>>> scheduler = braintools.optim.PiecewiseConstantSchedule(
...     base_lr=0.1,
...     boundaries=[1000, 2000],
...     values=[0.1, 0.01, 0.001]
... )
>>> scheduler.load_state_dict(checkpoint['scheduler_state'])
>>> # Continue from step 1500 with correct LR
>>> print(f"{scheduler.get_last_lr()[0]:.4f}")
0.0100

See also

StepLR

Exponential decay at regular intervals

MultiStepLR

Similar concept with multiplicative decay

CosineAnnealingLR

Smooth transitions instead of step changes

SequentialLR

Chain multiple schedulers sequentially

References

get_lr()[source]#

Calculate learning rate.