LBFGS#
- class braintools.optim.LBFGS(lr=1.0, memory_size=10, scale_init_precond=True, linesearch=None, grad_clip_norm=None, grad_clip_value=None)#
L-BFGS optimizer (Limited-memory Broyden-Fletcher-Goldfarb-Shanno).
L-BFGS is a quasi-Newton optimization method that approximates the inverse Hessian matrix using a limited amount of memory [1]. It provides superlinear convergence for smooth, unconstrained optimization problems and is widely used in scientific computing, machine learning, and numerical optimization.
This optimizer is particularly effective for: - Medium to large-scale optimization problems - Smooth, differentiable objective functions - Full-batch or deterministic gradient computations - Scientific computing and parameter estimation - Neural network fine-tuning with small datasets
- Parameters:
lr (
float|LRScheduler) – Learning rate (step size). Can be a float or any LRScheduler instance. L-BFGS typically uses lr=1.0 as it computes optimal step sizes via line search. Adjust only if line search is disabled or for specific needs.memory_size (
int) – Number of past gradient-position pairs to store for Hessian approximation. Typical values: 3-20. Larger values give better approximations but use more memory. Trade-off between accuracy and computational cost.scale_init_precond (
bool) – Whether to scale the initial Hessian approximation using gradient information. Improves convergence by adapting to problem scale. Recommended for most cases.grad_clip_norm (
float|None) – Maximum norm for gradient clipping. Gradients are scaled when their global norm exceeds this value. Useful for numerical stability.grad_clip_value (
float|None) – Maximum absolute value for element-wise gradient clipping. Each gradient component is clipped to [-grad_clip_value, grad_clip_value].
Notes
Mathematical Formulation:
L-BFGS approximates the inverse Hessian matrix \(H_k^{-1}\) using the limited-memory BFGS update formula [2]. The parameter update is:
\[\theta_{k+1} = \theta_k - \alpha_k H_k^{-1} \nabla f(\theta_k)\]The inverse Hessian approximation uses \(m\) stored pairs:
\[s_i = \theta_{i+1} - \theta_i \quad \text{(position difference)}\]\[y_i = \nabla f(\theta_{i+1}) - \nabla f(\theta_i) \quad \text{(gradient difference)}\]with curvature information:
\[\rho_i = \frac{1}{y_i^T s_i}\]Two-Loop Recursion Algorithm:
First loop (newest to oldest): Compute direction adjustments
Initial scaling: \(H_0^{-1} = \gamma_k I\) where \(\gamma_k = \frac{s_{k-1}^T y_{k-1}}{y_{k-1}^T y_{k-1}}\)
Second loop (oldest to newest): Apply BFGS corrections
Line Search with Zoom Algorithm:
This implementation includes automatic zoom line search finding step size \(\alpha_k\) satisfying the strong Wolfe conditions for robust convergence. The bound-constrained variant L-BFGS-B is described in [3] and [4].
Key Characteristics:
Superlinear convergence: Faster than first-order methods near optimum
Memory efficient: O(mn) storage for n parameters, m history size
Curvature aware: Uses second-order information without computing Hessian
Self-scaling: Adapts to problem geometry automatically
Robust line search: Ensures sufficient decrease and curvature conditions
Limitations:
Not suitable for stochastic mini-batch optimization
Requires full gradients for best performance
Memory scales with memory_size × parameter_count
Line search requires additional function evaluations
Important Usage Note:
Without a line search (
linesearch=None), update L-BFGS like any other optimizer by passing only the gradients:>>> import brainstate >>> import braintools >>> import jax.numpy as jnp >>> >>> brainstate.random.seed(0) >>> model = brainstate.nn.Linear(3, 1) >>> x = brainstate.random.randn(16, 3) >>> def loss_fn(): ... return jnp.mean(model(x) ** 2) >>> optimizer = braintools.optim.LBFGS() >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> grads = brainstate.transform.grad( ... loss_fn, model.states(brainstate.ParamState) ... )() >>> optimizer.update(grads)
With a line search (
linesearch='zoom'or'backtracking'), the line search needs the current objective value and a function that evaluates the objective at trial points. That function must be pure: it receives the parameter values as a dict keyed like the weights given toregister_trainable_weights()and returns a scalar.>>> params = {'w': brainstate.ParamState(jnp.array([-1.5, 2.0]))} >>> optimizer = braintools.optim.LBFGS(linesearch='zoom') >>> _ = optimizer.register_trainable_weights(params) >>> def objective(values): # pure function of the parameter values ... w = values['w'] ... return (1 - w[0]) ** 2 + 100 * (w[1] - w[0] ** 2) ** 2 >>> def step(i): ... grads, value = brainstate.transform.grad( ... lambda: objective({'w': params['w'].value}), ... grad_states=params, return_value=True)() ... optimizer.update(grads, value=value, value_fn=objective) >>> _ = brainstate.transform.for_loop(step, jnp.arange(40)) # 40 compiled iterations >>> print(jnp.round(params['w'].value, 3)) # the Rosenbrock minimum [1. 1.]
References
Examples
Basic usage for batch optimization:
>>> import brainstate as brainstate >>> import braintools as braintools >>> >>> # Create model >>> model = brainstate.nn.Linear(10, 5) >>> >>> # Initialize L-BFGS optimizer >>> optimizer = braintools.optim.LBFGS(lr=1.0) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
With custom memory size:
>>> # Larger memory for better Hessian approximation >>> optimizer = braintools.optim.LBFGS( ... lr=1.0, ... memory_size=20 ... ) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
With smaller memory for efficiency:
>>> # Smaller memory footprint >>> optimizer = braintools.optim.LBFGS( ... lr=1.0, ... memory_size=5 ... ) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
Disabling initial Hessian scaling:
>>> # Without Hessian scaling >>> optimizer = braintools.optim.LBFGS( ... lr=1.0, ... memory_size=10, ... scale_init_precond=False ... ) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
Fine-tuning example with full-batch training:
>>> import jax >>> import jax.numpy as jnp >>> >>> brainstate.random.seed(0) >>> model = brainstate.nn.Sequential( ... brainstate.nn.Linear(20, 64), ... brainstate.nn.ReLU(), ... brainstate.nn.Linear(64, 10) ... ) >>> >>> # L-BFGS for fine-tuning with full batch >>> optimizer = braintools.optim.LBFGS( ... lr=1.0, ... memory_size=10 ... ) >>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState)) >>> >>> # Use the entire dataset (not mini-batches) >>> x_full = brainstate.random.randn(256, 20) >>> y_full = jax.nn.one_hot(brainstate.random.randint(0, 10, (256,)), 10) >>> >>> def loss_fn(): ... logits = model(x_full) ... return jnp.mean(braintools.metric.softmax_cross_entropy(logits, y_full)) >>> >>> @brainstate.transform.jit ... def train_step(): ... grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))() ... optimizer.update(grads) ... return loss_fn() >>> >>> # Run 50 full-batch steps as one compiled loop >>> losses = brainstate.transform.for_loop(lambda i: train_step(), jnp.arange(50)) >>> bool(losses[-1] < losses[0]) True
Convex optimization example:
>>> # L-BFGS excels at convex problems: least-squares linear regression >>> brainstate.random.seed(0) >>> x = brainstate.random.randn(200, 50) >>> y = x @ brainstate.random.randn(50) >>> params = {'w': brainstate.ParamState(jnp.zeros(50))} >>> optimizer = braintools.optim.LBFGS( ... lr=1.0, ... memory_size=15, ... linesearch='zoom' ... ) >>> _ = optimizer.register_trainable_weights(params) >>> >>> def objective(values): ... return jnp.mean((x @ values['w'] - y) ** 2) >>> def step(i): ... grads, value = brainstate.transform.grad( ... lambda: objective({'w': params['w'].value}), ... grad_states=params, return_value=True)() ... optimizer.update(grads, value=value, value_fn=objective) >>> >>> # Typically converges in fewer iterations than first-order methods >>> _ = brainstate.transform.for_loop(step, jnp.arange(30)) >>> bool(objective({'w': params['w'].value}) < 1e-6) True
Scientific computing - parameter fitting:
>>> # Fitting an exponential decay model A * exp(-k t) + C >>> def exponential_model(params, t): ... return params['A'] * jnp.exp(-params['k'] * t) + params['C'] >>> >>> time_points = jnp.linspace(0.0, 5.0, 50) >>> observations = exponential_model({'A': 2.0, 'k': 1.5, 'C': 0.5}, time_points) >>> def loss_fn(params): ... predictions = exponential_model(params, time_points) ... return jnp.mean((predictions - observations) ** 2) >>> >>> params = {'A': brainstate.ParamState(jnp.array(1.0)), ... 'k': brainstate.ParamState(jnp.array(1.0)), ... 'C': brainstate.ParamState(jnp.array(0.0))} >>> >>> # L-BFGS for precise parameter estimation >>> optimizer = braintools.optim.LBFGS( ... lr=1.0, ... memory_size=20, # Higher accuracy for scientific computing ... scale_init_precond=True, ... linesearch='zoom' ... ) >>> _ = optimizer.register_trainable_weights(params) >>> def fit_step(i): ... grads, value = brainstate.transform.grad( ... lambda: loss_fn({k: v.value for k, v in params.items()}), ... grad_states=params, return_value=True)() ... optimizer.update(grads, value=value, value_fn=loss_fn) >>> _ = brainstate.transform.for_loop(fit_step, jnp.arange(50)) >>> print({k: round(float(v.value), 3) for k, v in params.items()}) {'A': 2.0, 'k': 1.5, 'C': 0.5}
Hybrid optimization strategy:
>>> brainstate.random.seed(0) >>> x = brainstate.random.randn(512, 10) >>> y = x @ brainstate.random.randn(10) >>> params = {'w': brainstate.ParamState(jnp.zeros(10))} >>> def objective(values, xb, yb): ... return jnp.mean((xb @ values['w'] - yb) ** 2) >>> >>> # Stage 1: Adam for exploration on mini-batches (stochastic) >>> adam_opt = braintools.optim.Adam(lr=0.01) >>> _ = adam_opt.register_trainable_weights(params) >>> def adam_step(batch): ... xb, yb = batch ... grads = brainstate.transform.grad( ... lambda: objective({'w': params['w'].value}, xb, yb), grad_states=params)() ... adam_opt.update(grads) >>> batches = (x.reshape(8, 64, 10), y.reshape(8, 64)) >>> _ = brainstate.transform.for_loop(adam_step, batches) >>> >>> # Stage 2: L-BFGS for refinement on the full dataset (deterministic) >>> lbfgs_opt = braintools.optim.LBFGS(lr=1.0, memory_size=20, linesearch='zoom') >>> _ = lbfgs_opt.register_trainable_weights(params) >>> full_objective = lambda values: objective(values, x, y) >>> def lbfgs_step(i): ... grads, value = brainstate.transform.grad( ... lambda: full_objective({'w': params['w'].value}), ... grad_states=params, return_value=True)() ... lbfgs_opt.update(grads, value=value, value_fn=full_objective) >>> _ = brainstate.transform.for_loop(lbfgs_step, jnp.arange(20)) >>> bool(full_objective({'w': params['w'].value}) < 1e-6) True
Memory size comparison:
>>> # Small memory (fast, less accurate) >>> opt_small = braintools.optim.LBFGS(memory_size=3) >>> >>> # Medium memory (balanced) >>> opt_medium = braintools.optim.LBFGS(memory_size=10) >>> >>> # Large memory (slower, more accurate) >>> opt_large = braintools.optim.LBFGS(memory_size=30)
See also
- update(grads, value=None, value_fn=None, **kwargs)[source]#
Update parameters with LBFGS optimizer.
- Parameters:
grads (dict) – Dictionary of gradients for each parameter.
value (float, optional) – Current value of the objective function. Required for linesearch.
value_fn (callable, optional) – Pure function
value_fn(params) -> valueof the parameter values (a dict keyed like the registered weights). Required for linesearch.**kwargs – Additional arguments passed to the optimizer update.
Notes
LBFGS requires additional arguments for the linesearch: - value: current objective function value - grad: gradients (automatically passed) - value_fn: callable to evaluate objective function
valuemay also be the(grads, value)pair returned bybrainstate.transform.grad(..., return_value=True).