Bounded Optimization with ScipyOptimizer#
ScipyOptimizer runs scipy.optimize.minimize on a JAX loss function and supplies the gradient automatically with JAX. It suits smooth, low- to medium-dimensional problems such as fitting model parameters to data, where quasi-Newton methods ('L-BFGS-B', 'TNC', 'SLSQP') converge in a handful of iterations.
Argument |
Meaning |
|---|---|
|
returns a scalar; called as |
|
|
|
any |
|
passed through to SciPy |
minimize(n_iter) runs n_iter independent minimizations from random starting points drawn uniformly within the bounds and returns the best OptimizeResult; result.x has the structure of bounds.
Two points to keep in mind:
units are stripped: with quantity bounds,
loss_funreceives plain numbers expressed in the unit of each lower bound (as withNevergradOptimizer), andresult.xis returned without units;starting points come from NumPy’s global generator, so call
np.random.seed(...)for reproducible restarts.
1. Setup#
import brainunit as u
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np
from scipy import optimize as sopt
import braintools
np.random.seed(0)
2. A First Example#
The Rosenbrock function has a curved valley that slows down plain gradient descent. L-BFGS-B, with exact JAX gradients, reaches the minimum at \((1, 1)\) in a few dozen evaluations. With dict bounds the loss is called by keyword.
def rosenbrock(x, y):
return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2
opt = braintools.optim.ScipyOptimizer(rosenbrock, bounds={"x": (-2.0, 2.0), "y": (-1.0, 3.0)}, method="L-BFGS-B")
res = opt.minimize(n_iter=1)
print(f"x = {float(res.x['x']):.6f}, y = {float(res.x['y']):.6f}, loss {res.fun:.2e}, "
f"{res.nfev} function and {res.njev} gradient evaluations, success {res.success}")
x = 1.000000, y = 0.999999, loss 1.65e-12, 20 function and 20 gradient evaluations, success True
Array-valued bounds give array-valued arguments; here 50 weights of a linear model are fitted at once:
rng = np.random.default_rng(1)
X = jnp.asarray(rng.normal(size=(200, 50)))
w_true = jnp.asarray(rng.normal(size=50))
y = X @ w_true + 0.1 * jnp.asarray(rng.normal(size=200))
opt = braintools.optim.ScipyOptimizer(lambda w: jnp.mean((X @ w - y) ** 2),
bounds=[(-5 * jnp.ones(50), 5 * jnp.ones(50))], method="L-BFGS-B")
res = opt.minimize()
w_fit = res.x[0]
w_lstsq = np.linalg.lstsq(np.asarray(X), np.asarray(y), rcond=None)[0]
print(f"{res.nit} iterations; max |w - least-squares solution| = {np.max(np.abs(np.asarray(w_fit) - w_lstsq)):.1e}")
17 iterations; max |w - least-squares solution| = 5.7e-05
3. Physical Units: Fitting a Membrane Time Constant#
A neuron’s voltage relaxes after a current step as \(V(t) = V_\infty + (V_0 - V_\infty)e^{-t/\tau}\). Below, \(\tau\) and \(V_\infty\) are fitted to a noisy trace. The bounds are quantities, but the loss receives plain numbers in the bounds’ units (ms for \(\tau\), mV for \(V_\infty\)), so the model must be written in those units too.
t_ms = np.arange(0, 100, 0.5)
true_tau, true_vinf, v0 = 18.0, -55.0, -70.0
trace = true_vinf + (v0 - true_vinf) * np.exp(-t_ms / true_tau) + 0.4 * rng.normal(size=t_ms.size)
def trace_loss(tau, v_inf): # plain numbers: tau in ms, v_inf in mV
model = v_inf + (v0 - v_inf) * jnp.exp(-t_ms / tau)
return jnp.mean((model - trace) ** 2)
opt = braintools.optim.ScipyOptimizer(
trace_loss, bounds=[(1.0 * u.ms, 0.1 * u.second), (-80.0 * u.mV, -40.0 * u.mV)], method="L-BFGS-B")
res = opt.minimize(n_iter=3)
tau_fit, vinf_fit = (float(v) for v in res.x)
print(f"tau = {tau_fit:.2f} ms, V_inf = {vinf_fit:.2f} mV (true 18 ms, -55 mV)")
print("note: the upper bound 0.1 s was converted to the lower bound's unit (ms)")
(c_tau, c_vinf), _ = sopt.curve_fit(lambda t, tau, vi: vi + (v0 - vi) * np.exp(-t / tau), t_ms, trace, p0=[10, -60])
print(f"scipy.optimize.curve_fit: tau = {c_tau:.2f} ms, V_inf = {c_vinf:.2f} mV")
tau = 17.48 ms, V_inf = -55.09 mV (true 18 ms, -55 mV)
note: the upper bound 0.1 s was converted to the lower bound's unit (ms)
scipy.optimize.curve_fit: tau = 17.48 ms, V_inf = -55.09 mV
4. Random Restarts for Multimodal Losses#
Local methods find the minimum of the basin they start in. n_iter restarts from random points and keeps the best result, which helps when the loss has several minima. The function below has 7 local minima in \([-10, 10]\); the global one is near \(x = -1.3\).
def wavy(x):
return 0.1 * x ** 2 + jnp.sin(2 * x) * 3
xs = np.linspace(-10, 10, 1000)
x_global = xs[np.argmin(np.asarray(wavy(jnp.asarray(xs))))]
opt = braintools.optim.ScipyOptimizer(wavy, bounds=[(-10.0, 10.0)], method="L-BFGS-B")
for n in [1, 3, 10]:
hits = 0
for trial in range(50):
res = opt.minimize(n_iter=n)
hits += abs(float(res.x[0]) - x_global) < 0.05
print(f"n_iter = {n:2d}: global minimum found in {hits}/50 runs")
n_iter = 1: global minimum found in 12/50 runs
n_iter = 3: global minimum found in 26/50 runs
n_iter = 10: global minimum found in 49/50 runs
5. Constraints#
Methods such as 'SLSQP' accept SciPy constraints. Constraint functions receive SciPy’s flat float64 parameter vector, not the structured parameters: list bounds are concatenated in order, and dict bounds in alphabetical key order (JAX’s flattening order). Here, the closest point to \((2, 1)\) is found on the line \(x + y = 1\) with \(x \ge 0.6\):
opt = braintools.optim.ScipyOptimizer(
lambda **p: (p['x'] - 2) ** 2 + (p['y'] - 1) ** 2, # bounds below list 'y' before 'x'
bounds={"y": (-5.0, 5.0), "x": (-5.0, 5.0)},
method="SLSQP",
constraints=[
{"type": "eq", "fun": lambda v: v[0] + v[1] - 1.0}, # v = [x, y]: alphabetical
{"type": "ineq", "fun": lambda v: v[0] - 0.6}, # x >= 0.6
],
)
res = opt.minimize(n_iter=3)
print(f"x = {float(res.x['x']):.4f}, y = {float(res.x['y']):.4f} (analytic: x = 1, y = 0); success {res.success}")
x = 1.0000, y = 0.0000 (analytic: x = 1, y = 0); success True
The unconstrained minimum \((2, 1)\) violates \(x + y = 1\); projecting it onto the line gives \((1, 0)\), which also satisfies \(x \ge 0.6\).
6. Choosing a Method, and Numerical Precision#
Gradient-based methods need far fewer evaluations on smooth problems; gradient-free ones ('Powell') work without a gradient but scale poorly with dimension. Here is the 50-weight linear model of Section 2 again. Iteration limits are method-specific SciPy options (TNC calls its limit maxfun).
b = [(-5 * jnp.ones(50), 5 * jnp.ones(50))]
limits = {"L-BFGS-B": {}, "TNC": {"maxfun": 5000}, "SLSQP": {}, "Powell": {"maxiter": 20000}}
def compare(X, y, w_ref):
loss = lambda w: jnp.mean((X @ w - y) ** 2)
for method, options in limits.items():
np.random.seed(0)
res = braintools.optim.ScipyOptimizer(loss, bounds=b, method=method, options=options).minimize()
err = np.max(np.abs(np.asarray(res.x[0]) - w_ref))
print(f"{method:9s} {res.nfev:6d} evaluations, max weight error {err:.1e}, {res.message}")
compare(X, y, w_lstsq)
L-BFGS-B 33 evaluations, max weight error 3.1e-05, CONVERGENCE: RELATIVE REDUCTION OF F <= FACTR*EPSMCH
TNC 68 evaluations, max weight error 3.1e+00, Linear search failed
SLSQP 27 evaluations, max weight error 7.9e-05, Optimization terminated successfully
Powell 7029 evaluations, max weight error 2.7e-04, Optimization terminated successfully.
TNC stops with “Linear search failed”. JAX computes the loss and gradient in float32 by default, while SciPy’s line searches assume double precision: near the optimum, float32 rounding hides the small decreases the line search looks for. Enabling 64-bit JAX fixes it. Set the flag after importing braintools (importing brainstate resets it), and recreate the arrays so they are float64:
import jax
jax.config.update("jax_enable_x64", True)
X64, y64 = jnp.asarray(np.asarray(X), dtype=jnp.float64), jnp.asarray(np.asarray(y), dtype=jnp.float64)
compare(X64, y64, w_lstsq)
jax.config.update("jax_enable_x64", False)
L-BFGS-B 22 evaluations, max weight error 6.9e-06, CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL
TNC 41 evaluations, max weight error 2.3e-05, Converged (|f_n-f_(n-1)| ~= 0)
SLSQP 27 evaluations, max weight error 7.9e-05, Optimization terminated successfully
Powell 4515 evaluations, max weight error 2.8e-04, Optimization terminated successfully.
Summary#
The gradient comes from JAX automatically; pass any
scipy.optimize.minimizemethod.boundssets the structure of the parameters and ofresult.x; list bounds are positional, dict bounds keyword arguments.With quantity bounds the loss sees plain numbers in the lower bound’s unit, and results come back without units.
n_itergives random restarts (seed NumPy); use them for multimodal losses.Constraint functions see the flat parameter vector (dict bounds in alphabetical order).
SciPy’s line searches assume double precision; enable
jax_enable_x64for tight fits.For non-smooth or black-box objectives, use
NevergradOptimizerinstead.