Bounded Optimization with ScipyOptimizer#

ScipyOptimizer runs scipy.optimize.minimize on a JAX loss function and supplies the gradient automatically with JAX. It suits smooth, low- to medium-dimensional problems such as fitting model parameters to data, where quasi-Newton methods ('L-BFGS-B', 'TNC', 'SLSQP') converge in a handful of iterations.

Argument

Meaning

loss_fun

returns a scalar; called as loss_fun(*params) for list bounds or loss_fun(**params) for dict bounds

bounds

(low, high) per parameter; scalars, arrays, or brainunit quantities

method

any scipy.optimize.minimize method; gradient-free ones ('Nelder-Mead', 'Powell', 'COBYLA') skip the gradient

constraints, tol, callback, options

passed through to SciPy

minimize(n_iter) runs n_iter independent minimizations from random starting points drawn uniformly within the bounds and returns the best OptimizeResult; result.x has the structure of bounds.

Two points to keep in mind:

  • units are stripped: with quantity bounds, loss_fun receives plain numbers expressed in the unit of each lower bound (as with NevergradOptimizer), and result.x is returned without units;

  • starting points come from NumPy’s global generator, so call np.random.seed(...) for reproducible restarts.

1. Setup#

import brainunit as u
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np
from scipy import optimize as sopt

import braintools

np.random.seed(0)

2. A First Example#

The Rosenbrock function has a curved valley that slows down plain gradient descent. L-BFGS-B, with exact JAX gradients, reaches the minimum at \((1, 1)\) in a few dozen evaluations. With dict bounds the loss is called by keyword.

def rosenbrock(x, y):
    return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2


opt = braintools.optim.ScipyOptimizer(rosenbrock, bounds={"x": (-2.0, 2.0), "y": (-1.0, 3.0)}, method="L-BFGS-B")
res = opt.minimize(n_iter=1)
print(f"x = {float(res.x['x']):.6f}, y = {float(res.x['y']):.6f}, loss {res.fun:.2e}, "
      f"{res.nfev} function and {res.njev} gradient evaluations, success {res.success}")
x = 1.000000, y = 0.999999, loss 1.65e-12, 20 function and 20 gradient evaluations, success True

Array-valued bounds give array-valued arguments; here 50 weights of a linear model are fitted at once:

rng = np.random.default_rng(1)
X = jnp.asarray(rng.normal(size=(200, 50)))
w_true = jnp.asarray(rng.normal(size=50))
y = X @ w_true + 0.1 * jnp.asarray(rng.normal(size=200))

opt = braintools.optim.ScipyOptimizer(lambda w: jnp.mean((X @ w - y) ** 2),
                                      bounds=[(-5 * jnp.ones(50), 5 * jnp.ones(50))], method="L-BFGS-B")
res = opt.minimize()
w_fit = res.x[0]
w_lstsq = np.linalg.lstsq(np.asarray(X), np.asarray(y), rcond=None)[0]
print(f"{res.nit} iterations; max |w - least-squares solution| = {np.max(np.abs(np.asarray(w_fit) - w_lstsq)):.1e}")
17 iterations; max |w - least-squares solution| = 5.7e-05

3. Physical Units: Fitting a Membrane Time Constant#

A neuron’s voltage relaxes after a current step as \(V(t) = V_\infty + (V_0 - V_\infty)e^{-t/\tau}\). Below, \(\tau\) and \(V_\infty\) are fitted to a noisy trace. The bounds are quantities, but the loss receives plain numbers in the bounds’ units (ms for \(\tau\), mV for \(V_\infty\)), so the model must be written in those units too.

t_ms = np.arange(0, 100, 0.5)
true_tau, true_vinf, v0 = 18.0, -55.0, -70.0
trace = true_vinf + (v0 - true_vinf) * np.exp(-t_ms / true_tau) + 0.4 * rng.normal(size=t_ms.size)


def trace_loss(tau, v_inf):  # plain numbers: tau in ms, v_inf in mV
    model = v_inf + (v0 - v_inf) * jnp.exp(-t_ms / tau)
    return jnp.mean((model - trace) ** 2)


opt = braintools.optim.ScipyOptimizer(
    trace_loss, bounds=[(1.0 * u.ms, 0.1 * u.second), (-80.0 * u.mV, -40.0 * u.mV)], method="L-BFGS-B")
res = opt.minimize(n_iter=3)
tau_fit, vinf_fit = (float(v) for v in res.x)
print(f"tau = {tau_fit:.2f} ms, V_inf = {vinf_fit:.2f} mV (true 18 ms, -55 mV)")
print("note: the upper bound 0.1 s was converted to the lower bound's unit (ms)")

(c_tau, c_vinf), _ = sopt.curve_fit(lambda t, tau, vi: vi + (v0 - vi) * np.exp(-t / tau), t_ms, trace, p0=[10, -60])
print(f"scipy.optimize.curve_fit: tau = {c_tau:.2f} ms, V_inf = {c_vinf:.2f} mV")
tau = 17.48 ms, V_inf = -55.09 mV (true 18 ms, -55 mV)
note: the upper bound 0.1 s was converted to the lower bound's unit (ms)
scipy.optimize.curve_fit: tau = 17.48 ms, V_inf = -55.09 mV

4. Random Restarts for Multimodal Losses#

Local methods find the minimum of the basin they start in. n_iter restarts from random points and keeps the best result, which helps when the loss has several minima. The function below has 7 local minima in \([-10, 10]\); the global one is near \(x = -1.3\).

def wavy(x):
    return 0.1 * x ** 2 + jnp.sin(2 * x) * 3


xs = np.linspace(-10, 10, 1000)
x_global = xs[np.argmin(np.asarray(wavy(jnp.asarray(xs))))]
opt = braintools.optim.ScipyOptimizer(wavy, bounds=[(-10.0, 10.0)], method="L-BFGS-B")
for n in [1, 3, 10]:
    hits = 0
    for trial in range(50):
        res = opt.minimize(n_iter=n)
        hits += abs(float(res.x[0]) - x_global) < 0.05
    print(f"n_iter = {n:2d}: global minimum found in {hits}/50 runs")
n_iter =  1: global minimum found in 12/50 runs
n_iter =  3: global minimum found in 26/50 runs
n_iter = 10: global minimum found in 49/50 runs

5. Constraints#

Methods such as 'SLSQP' accept SciPy constraints. Constraint functions receive SciPy’s flat float64 parameter vector, not the structured parameters: list bounds are concatenated in order, and dict bounds in alphabetical key order (JAX’s flattening order). Here, the closest point to \((2, 1)\) is found on the line \(x + y = 1\) with \(x \ge 0.6\):

opt = braintools.optim.ScipyOptimizer(
    lambda **p: (p['x'] - 2) ** 2 + (p['y'] - 1) ** 2,  # bounds below list 'y' before 'x'
    bounds={"y": (-5.0, 5.0), "x": (-5.0, 5.0)},
    method="SLSQP",
    constraints=[
        {"type": "eq", "fun": lambda v: v[0] + v[1] - 1.0},  # v = [x, y]: alphabetical
        {"type": "ineq", "fun": lambda v: v[0] - 0.6},  # x >= 0.6
    ],
)
res = opt.minimize(n_iter=3)
print(f"x = {float(res.x['x']):.4f}, y = {float(res.x['y']):.4f} (analytic: x = 1, y = 0); success {res.success}")
x = 1.0000, y = 0.0000 (analytic: x = 1, y = 0); success True

The unconstrained minimum \((2, 1)\) violates \(x + y = 1\); projecting it onto the line gives \((1, 0)\), which also satisfies \(x \ge 0.6\).

6. Choosing a Method, and Numerical Precision#

Gradient-based methods need far fewer evaluations on smooth problems; gradient-free ones ('Powell') work without a gradient but scale poorly with dimension. Here is the 50-weight linear model of Section 2 again. Iteration limits are method-specific SciPy options (TNC calls its limit maxfun).

b = [(-5 * jnp.ones(50), 5 * jnp.ones(50))]
limits = {"L-BFGS-B": {}, "TNC": {"maxfun": 5000}, "SLSQP": {}, "Powell": {"maxiter": 20000}}


def compare(X, y, w_ref):
    loss = lambda w: jnp.mean((X @ w - y) ** 2)
    for method, options in limits.items():
        np.random.seed(0)
        res = braintools.optim.ScipyOptimizer(loss, bounds=b, method=method, options=options).minimize()
        err = np.max(np.abs(np.asarray(res.x[0]) - w_ref))
        print(f"{method:9s} {res.nfev:6d} evaluations, max weight error {err:.1e}, {res.message}")


compare(X, y, w_lstsq)
L-BFGS-B      33 evaluations, max weight error 3.1e-05, CONVERGENCE: RELATIVE REDUCTION OF F <= FACTR*EPSMCH
TNC           68 evaluations, max weight error 3.1e+00, Linear search failed
SLSQP         27 evaluations, max weight error 7.9e-05, Optimization terminated successfully
Powell      7029 evaluations, max weight error 2.7e-04, Optimization terminated successfully.

TNC stops with “Linear search failed”. JAX computes the loss and gradient in float32 by default, while SciPy’s line searches assume double precision: near the optimum, float32 rounding hides the small decreases the line search looks for. Enabling 64-bit JAX fixes it. Set the flag after importing braintools (importing brainstate resets it), and recreate the arrays so they are float64:

import jax

jax.config.update("jax_enable_x64", True)
X64, y64 = jnp.asarray(np.asarray(X), dtype=jnp.float64), jnp.asarray(np.asarray(y), dtype=jnp.float64)
compare(X64, y64, w_lstsq)
jax.config.update("jax_enable_x64", False)
L-BFGS-B      22 evaluations, max weight error 6.9e-06, CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL
TNC           41 evaluations, max weight error 2.3e-05, Converged (|f_n-f_(n-1)| ~= 0)
SLSQP         27 evaluations, max weight error 7.9e-05, Optimization terminated successfully
Powell      4515 evaluations, max weight error 2.8e-04, Optimization terminated successfully.

Summary#

  • The gradient comes from JAX automatically; pass any scipy.optimize.minimize method.

  • bounds sets the structure of the parameters and of result.x; list bounds are positional, dict bounds keyword arguments.

  • With quantity bounds the loss sees plain numbers in the lower bound’s unit, and results come back without units.

  • n_iter gives random restarts (seed NumPy); use them for multimodal losses.

  • Constraint functions see the flat parameter vector (dict bounds in alphabetical order).

  • SciPy’s line searches assume double precision; enable jax_enable_x64 for tight fits.

  • For non-smooth or black-box objectives, use NevergradOptimizer instead.