PiecewiseConstantSchedule#
- class braintools.optim.PiecewiseConstantSchedule(base_lr=0.001, boundaries=None, values=None, last_epoch=0)#
Piecewise constant learning rate schedule with step-wise transitions.
PiecewiseConstantSchedule implements a learning rate schedule where the learning rate remains constant within specified intervals and changes abruptly at predefined boundaries. This creates a step function that’s useful for stage-based training where different phases require different learning rates.
This scheduler is particularly effective for: - Multi-stage training pipelines - Transfer learning with progressive unfreezing - Training with curriculum learning - Reproducing specific research papers with fixed schedules (e.g., [1], [2]) - Budget-constrained training with predetermined phases
- Parameters:
base_lr (
float|List[float]) – Reference learning rate(s). Only the number of entries is used (to set the number of parameter groups); the magnitude is ignored becausevaluesare absolute learning rates, not multipliers. Pass a list to configure multiple parameter groups. Default: 1e-3.boundaries (
List[int] |None) – Step indices where the learning rate changes. Must be sorted in ascending order. The schedule will have len(boundaries) + 1 distinct phases. Default: [1000, 2000].values (
List[float] |None) – Absolute learning rate applied in each phase (not multipliers ofbase_lr). Must have exactly len(boundaries) + 1 elements. The i-th value applies from boundary[i-1] to boundary[i]. Default: [1.0, 0.1, 0.01].last_epoch (
int) – The index of the last epoch. Used when resuming training. Default: 0.
Notes
Mathematical Formulation:
The learning rate at step t is the (absolute) phase value:
\[\eta_t = v_i\]where \(v_i\) is determined by:
\[\begin{split}v_i = \begin{cases} \text{values}[0] & \text{if } t < \text{boundaries}[0] \\ \text{values}[1] & \text{if } \text{boundaries}[0] \leq t < \text{boundaries}[1] \\ ... & ... \\ \text{values}[n] & \text{if } t \geq \text{boundaries}[n-1] \end{cases}\end{split}\]Schedule Structure:
Given boundaries [b1, b2, …, bn] and values [v0, v1, …, vn]:
Steps [0, b1): learning_rate = v0
Steps [b1, b2): learning_rate = v1
Steps [b2, b3): learning_rate = v2
…
Steps [bn, ∞): learning_rate = vn
JIT Compatibility:
This implementation is JIT-compatible through the use of jnp.searchsorted for efficient boundary-based value selection without Python conditionals.
For smoother alternatives to step-wise schedules, see cyclical learning rates [3] and the training refinements surveyed in [4].
Examples
Classic ImageNet training schedule:
>>> import braintools >>> import brainstate >>> >>> model = brainstate.nn.Linear(10, 5) >>> >>> # ResNet50 on ImageNet schedule >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[30, 60, 80], # Epochs to decrease LR ... values=[0.1, 0.01, 0.001, 0.0001] # absolute LRs (not multipliers of base_lr) ... ) >>> optimizer = braintools.optim.SGD(lr=scheduler, momentum=0.9) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Train for 90 epochs total >>> # LR: 0.1 (epochs 0-29), 0.01 (30-59), 0.001 (60-79), 0.0001 (80-89) >>> for epoch in range(90): ... if epoch in (0, 29, 30, 60, 80): ... print(f"Epoch {epoch}: LR = {scheduler.get_last_lr()[0]:.6f}") ... scheduler.step() Epoch 0: LR = 0.100000 Epoch 29: LR = 0.100000 Epoch 30: LR = 0.010000 Epoch 60: LR = 0.001000 Epoch 80: LR = 0.000100
Transfer learning with progressive unfreezing:
>>> import braintools >>> >>> # Unfreeze layers progressively with different LRs >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=1e-3, ... boundaries=[5, 10, 15], # Unfreeze stages ... values=[1e-5, 1e-4, 5e-4, 1e-3] # Gradual increase ... ) >>> >>> # Stage 1 (0-4): Only train head, very low LR >>> # Stage 2 (5-9): Unfreeze top layers >>> # Stage 3 (10-14): Unfreeze middle layers >>> # Stage 4 (15+): Full model training >>> stages = {5: 'top layers', 10: 'middle layers', 15: 'all layers'} >>> for epoch in range(20): ... if epoch in stages: ... # (the corresponding layers would be unfrozen here) ... print(f"Epoch {epoch}: unfreeze {stages[epoch]}, " ... f"LR = {scheduler.get_last_lr()[0]:.0e}") ... scheduler.step() Epoch 5: unfreeze top layers, LR = 1e-04 Epoch 10: unfreeze middle layers, LR = 5e-04 Epoch 15: unfreeze all layers, LR = 1e-03
Budget-aware training schedule:
>>> import braintools >>> >>> # Training with computational budget constraints >>> # Fast initial training, then careful fine-tuning >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.01, ... boundaries=[100, 500, 800], ... values=[0.1, 0.01, 0.001, 0.0001] # Aggressive start, careful end ... ) >>> >>> # Steps 0-99: Fast exploration (LR=0.1) >>> # Steps 100-499: Normal training (LR=0.01) >>> # Steps 500-799: Fine-tuning (LR=0.001) >>> # Steps 800+: Final refinement (LR=0.0001) >>> for step in (0, 100, 500, 800): ... scheduler.step(step) ... print(f"Step {step}: LR = {scheduler.get_last_lr()[0]:.4f}") Step 0: LR = 0.1000 Step 100: LR = 0.0100 Step 500: LR = 0.0010 Step 800: LR = 0.0001
Multi-phase curriculum learning:
>>> import braintools >>> >>> # Different learning rates for different curriculum stages >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=1e-3, ... boundaries=[1000, 3000, 6000, 9000], ... values=[1e-4, 5e-4, 1e-3, 5e-4, 1e-4] ... ) >>> >>> curriculum_difficulties = [0.2, 0.4, 0.6, 0.8, 1.0] >>> >>> for step in (0, 1000, 3000, 6000, 9000): ... # Determine curriculum difficulty ... stage = sum(step >= b for b in scheduler.boundaries) ... difficulty = curriculum_difficulties[stage] ... # (a batch of this difficulty would be trained on here) ... scheduler.step(step) ... print(f"Step {step}: difficulty={difficulty}, " ... f"LR = {scheduler.get_last_lr()[0]:.4f}") Step 0: difficulty=0.2, LR = 0.0001 Step 1000: difficulty=0.4, LR = 0.0005 Step 3000: difficulty=0.6, LR = 0.0010 Step 6000: difficulty=0.8, LR = 0.0005 Step 9000: difficulty=1.0, LR = 0.0001
Reproducing paper schedules:
>>> import braintools >>> >>> # WideResNet schedule from paper >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[60, 120, 160], # Specific to WideResNet ... values=[0.1, 0.02, 0.004, 0.0008] ... ) >>> >>> # CIFAR training schedule from "Bag of Tricks" >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[150, 250], ... values=[0.1, 0.01, 0.001] ... )
Step-based (not epoch-based) scheduling:
>>> import braintools >>> >>> # Define boundaries in terms of training steps >>> steps_per_epoch = 100 # e.g. len(train_loader) >>> epoch_boundaries = [30, 60, 80] # Desired epoch boundaries >>> step_boundaries = [e * steps_per_epoch for e in epoch_boundaries] >>> step_boundaries [3000, 6000, 8000] >>> >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=step_boundaries, ... values=[0.1, 0.01, 0.001, 0.0001] ... ) >>> >>> # A training loop calls ``scheduler.step()`` once per batch; >>> # here we jump to the first step of epoch 30. >>> global_step = 30 * steps_per_epoch >>> scheduler.step(global_step) >>> print(f"{scheduler.get_last_lr()[0]:.4f}") 0.0100
Combining with warmup:
>>> import braintools >>> >>> # Add warmup to piecewise schedule >>> warmup_steps = 500 >>> main_boundaries = [5000, 10000, 15000] >>> >>> # Combine warmup with main schedule >>> all_boundaries = [warmup_steps] + main_boundaries >>> all_values = [0.001, 0.1, 0.01, 0.001, 0.0001] # Low start for warmup >>> >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=all_boundaries, ... values=all_values ... )
Dynamic monitoring and adjustment:
>>> import braintools >>> >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[1000, 2000, 3000], ... values=[0.1, 0.05, 0.01, 0.001] ... ) >>> >>> for step in range(1, 4000): ... old_lr = float(scheduler.get_last_lr()[0]) ... scheduler.step() ... new_lr = float(scheduler.get_last_lr()[0]) ... ... # Detect LR changes (momentum or other optimizer ... # states could optionally be reset here) ... if old_lr != new_lr: ... print(f"Step {step}: LR changed from {old_lr:.6f} to {new_lr:.6f}") Step 1000: LR changed from 0.100000 to 0.050000 Step 2000: LR changed from 0.050000 to 0.010000 Step 3000: LR changed from 0.010000 to 0.001000
Research experimentation with multiple schedules:
>>> import braintools >>> >>> # Compare different decay strategies >>> schedules = { ... 'aggressive': braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[10, 20], ... values=[0.1, 0.001, 0.00001] ... ), ... 'conservative': braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[30, 60], ... values=[0.1, 0.05, 0.01] ... ), ... 'multi_stage': braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[10, 20, 30, 40], ... values=[0.1, 0.08, 0.04, 0.01, 0.001] ... ) ... } >>> >>> # Compare the learning rate each schedule reaches at epoch 25 >>> for name, scheduler in schedules.items(): ... scheduler.step(25) ... print(f"{name}: LR at epoch 25 = {scheduler.get_last_lr()[0]:.5f}") aggressive: LR at epoch 25 = 0.00001 conservative: LR at epoch 25 = 0.10000 multi_stage: LR at epoch 25 = 0.04000
State persistence and checkpointing:
>>> import braintools >>> >>> # Save and restore schedule state >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[1000, 2000], ... values=[0.1, 0.01, 0.001] ... ) >>> >>> # Train for some steps (jump straight to step 1500 here) >>> scheduler.step(1500) >>> >>> # Save checkpoint >>> checkpoint = { ... 'step': 1500, ... 'scheduler_state': scheduler.state_dict(), ... } >>> >>> # Later: restore and continue >>> scheduler = braintools.optim.PiecewiseConstantSchedule( ... base_lr=0.1, ... boundaries=[1000, 2000], ... values=[0.1, 0.01, 0.001] ... ) >>> scheduler.load_state_dict(checkpoint['scheduler_state']) >>> # Continue from step 1500 with correct LR >>> print(f"{scheduler.get_last_lr()[0]:.4f}") 0.0100
See also
StepLRExponential decay at regular intervals
MultiStepLRSimilar concept with multiplicative decay
CosineAnnealingLRSmooth transitions instead of step changes
SequentialLRChain multiple schedulers sequentially
References