LBFGS#

class braintools.optim.LBFGS(lr=1.0, memory_size=10, scale_init_precond=True, linesearch=None, grad_clip_norm=None, grad_clip_value=None)#

L-BFGS optimizer (Limited-memory Broyden-Fletcher-Goldfarb-Shanno).

L-BFGS is a quasi-Newton optimization method that approximates the inverse Hessian matrix using a limited amount of memory [1]. It provides superlinear convergence for smooth, unconstrained optimization problems and is widely used in scientific computing, machine learning, and numerical optimization.

This optimizer is particularly effective for: - Medium to large-scale optimization problems - Smooth, differentiable objective functions - Full-batch or deterministic gradient computations - Scientific computing and parameter estimation - Neural network fine-tuning with small datasets

Parameters:
  • lr (float | LRScheduler) – Learning rate (step size). Can be a float or any LRScheduler instance. L-BFGS typically uses lr=1.0 as it computes optimal step sizes via line search. Adjust only if line search is disabled or for specific needs.

  • memory_size (int) – Number of past gradient-position pairs to store for Hessian approximation. Typical values: 3-20. Larger values give better approximations but use more memory. Trade-off between accuracy and computational cost.

  • scale_init_precond (bool) – Whether to scale the initial Hessian approximation using gradient information. Improves convergence by adapting to problem scale. Recommended for most cases.

  • grad_clip_norm (float | None) – Maximum norm for gradient clipping. Gradients are scaled when their global norm exceeds this value. Useful for numerical stability.

  • grad_clip_value (float | None) – Maximum absolute value for element-wise gradient clipping. Each gradient component is clipped to [-grad_clip_value, grad_clip_value].

Notes

Mathematical Formulation:

L-BFGS approximates the inverse Hessian matrix \(H_k^{-1}\) using the limited-memory BFGS update formula [2]. The parameter update is:

\[\theta_{k+1} = \theta_k - \alpha_k H_k^{-1} \nabla f(\theta_k)\]

The inverse Hessian approximation uses \(m\) stored pairs:

\[s_i = \theta_{i+1} - \theta_i \quad \text{(position difference)}\]
\[y_i = \nabla f(\theta_{i+1}) - \nabla f(\theta_i) \quad \text{(gradient difference)}\]

with curvature information:

\[\rho_i = \frac{1}{y_i^T s_i}\]

Two-Loop Recursion Algorithm:

  1. First loop (newest to oldest): Compute direction adjustments

  2. Initial scaling: \(H_0^{-1} = \gamma_k I\) where \(\gamma_k = \frac{s_{k-1}^T y_{k-1}}{y_{k-1}^T y_{k-1}}\)

  3. Second loop (oldest to newest): Apply BFGS corrections

Line Search with Zoom Algorithm:

This implementation includes automatic zoom line search finding step size \(\alpha_k\) satisfying the strong Wolfe conditions for robust convergence. The bound-constrained variant L-BFGS-B is described in [3] and [4].

Key Characteristics:

  • Superlinear convergence: Faster than first-order methods near optimum

  • Memory efficient: O(mn) storage for n parameters, m history size

  • Curvature aware: Uses second-order information without computing Hessian

  • Self-scaling: Adapts to problem geometry automatically

  • Robust line search: Ensures sufficient decrease and curvature conditions

Limitations:

  • Not suitable for stochastic mini-batch optimization

  • Requires full gradients for best performance

  • Memory scales with memory_size × parameter_count

  • Line search requires additional function evaluations

Important Usage Note:

Without a line search (linesearch=None), update L-BFGS like any other optimizer by passing only the gradients:

>>> import brainstate
>>> import braintools
>>> import jax.numpy as jnp
>>>
>>> brainstate.random.seed(0)
>>> model = brainstate.nn.Linear(3, 1)
>>> x = brainstate.random.randn(16, 3)
>>> def loss_fn():
...     return jnp.mean(model(x) ** 2)
>>> optimizer = braintools.optim.LBFGS()
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
>>> grads = brainstate.transform.grad(
...     loss_fn, model.states(brainstate.ParamState)
... )()
>>> optimizer.update(grads)

With a line search (linesearch='zoom' or 'backtracking'), the line search needs the current objective value and a function that evaluates the objective at trial points. That function must be pure: it receives the parameter values as a dict keyed like the weights given to register_trainable_weights() and returns a scalar.

>>> params = {'w': brainstate.ParamState(jnp.array([-1.5, 2.0]))}
>>> optimizer = braintools.optim.LBFGS(linesearch='zoom')
>>> _ = optimizer.register_trainable_weights(params)
>>> def objective(values):  # pure function of the parameter values
...     w = values['w']
...     return (1 - w[0]) ** 2 + 100 * (w[1] - w[0] ** 2) ** 2
>>> def step(i):
...     grads, value = brainstate.transform.grad(
...         lambda: objective({'w': params['w'].value}),
...         grad_states=params, return_value=True)()
...     optimizer.update(grads, value=value, value_fn=objective)
>>> _ = brainstate.transform.for_loop(step, jnp.arange(40))  # 40 compiled iterations
>>> print(jnp.round(params['w'].value, 3))  # the Rosenbrock minimum
[1. 1.]

References

Examples

Basic usage for batch optimization:

>>> import brainstate as brainstate
>>> import braintools as braintools
>>>
>>> # Create model
>>> model = brainstate.nn.Linear(10, 5)
>>>
>>> # Initialize L-BFGS optimizer
>>> optimizer = braintools.optim.LBFGS(lr=1.0)
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))

With custom memory size:

>>> # Larger memory for better Hessian approximation
>>> optimizer = braintools.optim.LBFGS(
...     lr=1.0,
...     memory_size=20
... )
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))

With smaller memory for efficiency:

>>> # Smaller memory footprint
>>> optimizer = braintools.optim.LBFGS(
...     lr=1.0,
...     memory_size=5
... )
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))

Disabling initial Hessian scaling:

>>> # Without Hessian scaling
>>> optimizer = braintools.optim.LBFGS(
...     lr=1.0,
...     memory_size=10,
...     scale_init_precond=False
... )
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))

Fine-tuning example with full-batch training:

>>> import jax
>>> import jax.numpy as jnp
>>>
>>> brainstate.random.seed(0)
>>> model = brainstate.nn.Sequential(
...     brainstate.nn.Linear(20, 64),
...     brainstate.nn.ReLU(),
...     brainstate.nn.Linear(64, 10)
... )
>>>
>>> # L-BFGS for fine-tuning with full batch
>>> optimizer = braintools.optim.LBFGS(
...     lr=1.0,
...     memory_size=10
... )
>>> _ = optimizer.register_trainable_weights(model.states(brainstate.ParamState))
>>>
>>> # Use the entire dataset (not mini-batches)
>>> x_full = brainstate.random.randn(256, 20)
>>> y_full = jax.nn.one_hot(brainstate.random.randint(0, 10, (256,)), 10)
>>>
>>> def loss_fn():
...     logits = model(x_full)
...     return jnp.mean(braintools.metric.softmax_cross_entropy(logits, y_full))
>>>
>>> @brainstate.transform.jit
... def train_step():
...     grads = brainstate.transform.grad(loss_fn, model.states(brainstate.ParamState))()
...     optimizer.update(grads)
...     return loss_fn()
>>>
>>> # Run 50 full-batch steps as one compiled loop
>>> losses = brainstate.transform.for_loop(lambda i: train_step(), jnp.arange(50))
>>> bool(losses[-1] < losses[0])
True

Convex optimization example:

>>> # L-BFGS excels at convex problems: least-squares linear regression
>>> brainstate.random.seed(0)
>>> x = brainstate.random.randn(200, 50)
>>> y = x @ brainstate.random.randn(50)
>>> params = {'w': brainstate.ParamState(jnp.zeros(50))}
>>> optimizer = braintools.optim.LBFGS(
...     lr=1.0,
...     memory_size=15,
...     linesearch='zoom'
... )
>>> _ = optimizer.register_trainable_weights(params)
>>>
>>> def objective(values):
...     return jnp.mean((x @ values['w'] - y) ** 2)
>>> def step(i):
...     grads, value = brainstate.transform.grad(
...         lambda: objective({'w': params['w'].value}),
...         grad_states=params, return_value=True)()
...     optimizer.update(grads, value=value, value_fn=objective)
>>>
>>> # Typically converges in fewer iterations than first-order methods
>>> _ = brainstate.transform.for_loop(step, jnp.arange(30))
>>> bool(objective({'w': params['w'].value}) < 1e-6)
True

Scientific computing - parameter fitting:

>>> # Fitting an exponential decay model A * exp(-k t) + C
>>> def exponential_model(params, t):
...     return params['A'] * jnp.exp(-params['k'] * t) + params['C']
>>>
>>> time_points = jnp.linspace(0.0, 5.0, 50)
>>> observations = exponential_model({'A': 2.0, 'k': 1.5, 'C': 0.5}, time_points)
>>> def loss_fn(params):
...     predictions = exponential_model(params, time_points)
...     return jnp.mean((predictions - observations) ** 2)
>>>
>>> params = {'A': brainstate.ParamState(jnp.array(1.0)),
...           'k': brainstate.ParamState(jnp.array(1.0)),
...           'C': brainstate.ParamState(jnp.array(0.0))}
>>>
>>> # L-BFGS for precise parameter estimation
>>> optimizer = braintools.optim.LBFGS(
...     lr=1.0,
...     memory_size=20,  # Higher accuracy for scientific computing
...     scale_init_precond=True,
...     linesearch='zoom'
... )
>>> _ = optimizer.register_trainable_weights(params)
>>> def fit_step(i):
...     grads, value = brainstate.transform.grad(
...         lambda: loss_fn({k: v.value for k, v in params.items()}),
...         grad_states=params, return_value=True)()
...     optimizer.update(grads, value=value, value_fn=loss_fn)
>>> _ = brainstate.transform.for_loop(fit_step, jnp.arange(50))
>>> print({k: round(float(v.value), 3) for k, v in params.items()})
{'A': 2.0, 'k': 1.5, 'C': 0.5}

Hybrid optimization strategy:

>>> brainstate.random.seed(0)
>>> x = brainstate.random.randn(512, 10)
>>> y = x @ brainstate.random.randn(10)
>>> params = {'w': brainstate.ParamState(jnp.zeros(10))}
>>> def objective(values, xb, yb):
...     return jnp.mean((xb @ values['w'] - yb) ** 2)
>>>
>>> # Stage 1: Adam for exploration on mini-batches (stochastic)
>>> adam_opt = braintools.optim.Adam(lr=0.01)
>>> _ = adam_opt.register_trainable_weights(params)
>>> def adam_step(batch):
...     xb, yb = batch
...     grads = brainstate.transform.grad(
...         lambda: objective({'w': params['w'].value}, xb, yb), grad_states=params)()
...     adam_opt.update(grads)
>>> batches = (x.reshape(8, 64, 10), y.reshape(8, 64))
>>> _ = brainstate.transform.for_loop(adam_step, batches)
>>>
>>> # Stage 2: L-BFGS for refinement on the full dataset (deterministic)
>>> lbfgs_opt = braintools.optim.LBFGS(lr=1.0, memory_size=20, linesearch='zoom')
>>> _ = lbfgs_opt.register_trainable_weights(params)
>>> full_objective = lambda values: objective(values, x, y)
>>> def lbfgs_step(i):
...     grads, value = brainstate.transform.grad(
...         lambda: full_objective({'w': params['w'].value}),
...         grad_states=params, return_value=True)()
...     lbfgs_opt.update(grads, value=value, value_fn=full_objective)
>>> _ = brainstate.transform.for_loop(lbfgs_step, jnp.arange(20))
>>> bool(full_objective({'w': params['w'].value}) < 1e-6)
True

Memory size comparison:

>>> # Small memory (fast, less accurate)
>>> opt_small = braintools.optim.LBFGS(memory_size=3)
>>>
>>> # Medium memory (balanced)
>>> opt_medium = braintools.optim.LBFGS(memory_size=10)
>>>
>>> # Large memory (slower, more accurate)
>>> opt_large = braintools.optim.LBFGS(memory_size=30)

See also

SGD

First-order stochastic gradient descent

Adam

Adaptive moment estimation for stochastic optimization

Rprop

Resilient propagation for batch learning

Adagrad

Adaptive gradient algorithm

default_tx()[source]#

Create default gradient transformation with clipping and weight decay.

update(grads, value=None, value_fn=None, **kwargs)[source]#

Update parameters with LBFGS optimizer.

Parameters:
  • grads (dict) – Dictionary of gradients for each parameter.

  • value (float, optional) – Current value of the objective function. Required for linesearch.

  • value_fn (callable, optional) – Pure function value_fn(params) -> value of the parameter values (a dict keyed like the registered weights). Required for linesearch.

  • **kwargs – Additional arguments passed to the optimizer update.

Notes

LBFGS requires additional arguments for the linesearch: - value: current objective function value - grad: gradients (automatically passed) - value_fn: callable to evaluate objective function

value may also be the (grads, value) pair returned by brainstate.transform.grad(..., return_value=True).