Tuning Surrogate Hyperparameters#

Which surrogate should you use, and how steep? This notebook answers the question empirically on a small spiking classifier, and shows how to optimize a surrogate parameter by gradient descent correctly. It covers:

  1. why jax.grad of the training loss with respect to a surrogate parameter is zero;

  2. the shape of the surrogate: comparing eight surrogates;

  3. the steepness: when training fails, why, and the fix;

  4. hypergradients: differentiating the result of training with respect to the steepness.

1. Setup: a Spike-Pattern Classification Task#

import time

import jax
import jax.numpy as jnp
import matplotlib.pyplot as plt
import numpy as np
import optax

import braintools.surrogate as surrogate

Six classes, each a frozen random spike pattern over 40 input channels and 80 time steps. Samples are noisy copies: the whole pattern is shifted by up to ±3 steps, 20 % of its spikes are deleted and random noise spikes are added.

The network has depth layers of 64 LIF neurons (\(v \leftarrow 0.9\,v + W s_{in}\), spike at \(v \ge 1\), subtractive reset) and a leaky, non-spiking readout; the class score is the maximum of the readout over time. detach_reset=True stops the gradient through the reset. Training uses Adam (from optax) on mini-batches of 64.

n_in, n_hidden, n_classes, n_time = 40, 64, 6, 80
rng = np.random.default_rng(0)
templates = rng.random((n_classes, n_time, n_in)) < 0.04

def make_data(n, seed):
    r = np.random.default_rng(seed)
    y = r.integers(0, n_classes, n)
    x = np.stack([np.roll(templates[c], s, axis=0) for c, s in zip(y, r.integers(-3, 4, n))])
    x &= r.random(x.shape) > 0.2            # delete 20 % of the spikes
    x |= r.random(x.shape) < 0.02           # add noise spikes
    return jnp.asarray(x, jnp.float32), jnp.asarray(y)

x_train, y_train = make_data(600, 1)
x_val, y_val = make_data(400, 2)

def init_params(seed, depth):
    keys = jax.random.split(jax.random.PRNGKey(seed), depth + 1)
    sizes = [n_in] + [n_hidden] * depth
    params = {f"w{i}": 3.0 * jax.random.normal(keys[i], (sizes[i], sizes[i + 1])) / np.sqrt(sizes[i]) for i in range(depth)}
    params["out"] = jax.random.normal(keys[-1], (n_hidden, n_classes)) / np.sqrt(n_hidden)
    return params

def forward(params, x, spike_fn, depth, detach_reset=False, beta=0.9):
    batch = x.shape[0]

    def step(carry, x_t):
        vs, readout = carry
        inp, new_vs, rates = x_t, [], []
        for i in range(depth):
            v = beta * vs[i] + inp @ params[f"w{i}"]
            s = spike_fn(v - 1.0)
            v = v - (jax.lax.stop_gradient(s) if detach_reset else s)
            new_vs.append(v); rates.append(s.mean()); inp = s
        readout = beta * readout + inp @ params["out"]
        return (new_vs, readout), (readout, jnp.stack(rates))

    init = ([jnp.zeros((batch, n_hidden))] * depth, jnp.zeros((batch, n_classes)))
    _, (readouts, rates) = jax.lax.scan(step, init, jnp.swapaxes(x, 0, 1))
    return readouts.max(0), rates.mean(0)

def cross_entropy(params, x, y, spike_fn, depth, detach_reset=False):
    logits, _ = forward(params, x, spike_fn, depth, detach_reset)
    return optax.softmax_cross_entropy_with_integer_labels(logits, y).mean()

def train(spike_fn, seed, depth=1, detach_reset=False, steps=150, lr=1e-2):
    params = init_params(seed, depth)
    opt = optax.adam(lr)
    state = opt.init(params)

    @jax.jit
    def update(params, state, i):
        idx = jax.random.choice(jax.random.PRNGKey(i), x_train.shape[0], (64,), replace=False)
        grads = jax.grad(cross_entropy)(params, x_train[idx], y_train[idx], spike_fn, depth, detach_reset)
        updates, state = opt.update(grads, state, params)
        return optax.apply_updates(params, updates), state, optax.global_norm(grads)

    first_grad_norm = None
    for i in range(steps):
        params, state, gnorm = update(params, state, i)
        first_grad_norm = float(gnorm) if first_grad_norm is None else first_grad_norm
    logits, rates = forward(params, x_val, spike_fn, depth, detach_reset)
    return float((logits.argmax(1) == y_val).mean()), first_grad_norm, np.asarray(rates)

2. The Training Loss Does Not Depend on the Surrogate Parameters#

The forward pass uses the exact step function, so the network output, and with it the training loss, are the same for every surrogate and every steepness. Their derivative with respect to a surrogate parameter is therefore zero:

params = init_params(0, 1)
for alpha in [1.0, 4.0, 16.0]:
    loss = cross_entropy(params, x_train[:64], y_train[:64], lambda v: surrogate.sigmoid(v, alpha), 1)
    print(f"alpha = {alpha:4}: training loss {float(loss):.6f}")
d_alpha = jax.grad(lambda a: cross_entropy(params, x_train[:64], y_train[:64], lambda v: surrogate.sigmoid(v, a), 1))(4.0)
print("d loss / d alpha =", float(d_alpha))
alpha =  1.0: training loss 2.581966
alpha =  4.0: training loss 2.581966
alpha = 16.0: training loss 2.581966
d loss / d alpha = 0.0

A surrogate parameter matters only through the weight updates it produces. To tune it, train with several values and compare validation results (sections 3 and 4), or differentiate the result of training with respect to the parameter (section 5).

3. The Shape of the Surrogate#

Eight surrogates, one hidden layer, three seeds each:

shapes = {"Sigmoid(4)": surrogate.Sigmoid(4.0), "ERF(1)": surrogate.ERF(1.0), "Arctan(1)": surrogate.Arctan(1.0),
          "PiecewiseQuadratic(1)": surrogate.PiecewiseQuadratic(1.0), "ReluGrad": surrogate.ReluGrad(),
          "GaussianGrad": surrogate.GaussianGrad(), "InvSquareGrad": surrogate.InvSquareGrad(),
          "LogTailedRelu": surrogate.LogTailedRelu()}
t0 = time.time()
for name, sg in shapes.items():
    acc = [train(sg, seed)[0] for seed in range(3)]
    print(f"{name:22s} validation accuracy {np.round(acc, 3)}   mean {np.mean(acc):.3f}")
print(f"({time.time() - t0:.0f} s)")
Sigmoid(4)             validation accuracy [0.99  0.99  0.995]   mean 0.992
ERF(1)                 validation accuracy [0.99  0.995 0.995]   mean 0.993
Arctan(1)              validation accuracy [0.987 0.99  0.995]   mean 0.991
PiecewiseQuadratic(1)  validation accuracy [0.99  0.99  0.995]   mean 0.992
ReluGrad               validation accuracy [0.99  0.993 0.995]   mean 0.992
GaussianGrad           validation accuracy [0.993 0.99  0.995]   mean 0.992
InvSquareGrad          validation accuracy [0.993 0.99  0.998]   mean 0.993
LogTailedRelu          validation accuracy [0.993 0.995 0.998]   mean 0.995
(28 s)

All eight reach 99 % with differences smaller than the variation between seeds, although their gradient mass differs by a factor of 50 or more (see the first notebook) and their tails range from finite support to non-localized. Part of the reason is Adam: it normalizes each parameter’s update by the running gradient magnitude, which removes the overall scale of the surrogate. With plain SGD, the mass acts like a factor on the learning rate and has to be compensated.

4. The Steepness#

Now Sigmoid with steepness from 0.25 to 256, for one and three hidden layers:

alphas = [0.25, 1.0, 4.0, 16.0, 64.0, 256.0]
results = {}
for depth in [1, 3]:
    for detach in [False, True]:
        for alpha in alphas:
            runs = [train(surrogate.Sigmoid(alpha), seed, depth, detach) for seed in range(2)]
            results[(depth, detach, alpha)] = (np.mean([r[0] for r in runs]), runs[0][1])

for detach in [False, True]:
    print(f"reset {'detached' if detach else 'attached'}:")
    for depth in [1, 3]:
        line = "   ".join(f"{a:g}: {results[(depth, detach, a)][0]:.2f}" for a in alphas)
        print(f"  depth {depth}  accuracy by alpha   {line}")
    for depth in [1, 3]:
        line = "   ".join(f"{a:g}: {results[(depth, detach, a)][1]:.0e}" for a in alphas)
        print(f"  depth {depth}  first gradient norm {line}")
reset attached:
  depth 1  accuracy by alpha   0.25: 0.99   1: 0.99   4: 0.99   16: 0.99   64: 0.98   256: 0.98
  depth 3  accuracy by alpha   0.25: 0.96   1: 0.97   4: 0.98   16: 0.91   64: 0.38   256: 0.44
  depth 1  first gradient norm 0.25: 6e+00   1: 6e+00   4: 6e+00   16: 6e+00   64: 5e+02   256: 5e+01
  depth 3  first gradient norm 0.25: 3e+01   1: 3e+01   4: 3e+01   16: 3e+01   64: 6e+06   256: 2e+07
reset detached:
  depth 1  accuracy by alpha   0.25: 0.99   1: 0.99   4: 0.99   16: 0.99   64: 0.99   256: 0.99
  depth 3  accuracy by alpha   0.25: 0.83   1: 0.96   4: 0.99   16: 0.98   64: 0.89   256: 0.89
  depth 1  first gradient norm 0.25: 6e+00   1: 6e+00   4: 7e+00   16: 7e+00   64: 7e+00   256: 8e+00
  depth 3  first gradient norm 0.25: 3e+01   1: 4e+01   4: 8e+01   16: 8e+01   64: 2e+02   256: 6e+02
fig, ax = plt.subplots(1, 2, figsize=(12, 4))
for detach, ls in [(False, "-"), (True, "--")]:
    for depth, color in [(1, "C0"), (3, "C1")]:
        label = f"depth {depth}, reset {'detached' if detach else 'attached'}"
        ax[0].semilogx(alphas, [results[(depth, detach, a)][0] for a in alphas], ls, color=color, marker="o", label=label)
        ax[1].loglog(alphas, [results[(depth, detach, a)][1] for a in alphas], ls, color=color, marker="o", label=label)
ax[0].set_xlabel("Sigmoid steepness alpha"); ax[0].set_ylabel("validation accuracy"); ax[0].legend(fontsize=8)
ax[1].set_xlabel("Sigmoid steepness alpha"); ax[1].set_ylabel("gradient norm at the first step")
plt.tight_layout(); plt.show()
../_images/be89880258bfbb4c100090dc1c5f32737a2dbed38248deec4969c87b3f4e59d7.png

With the gradient flowing through the reset (attached, solid lines), one hidden layer tolerates any steepness, but three layers degrade at \(\alpha = 16\) and fail at \(\alpha \ge 64\). At the same point the gradient norm at the first step jumps from about 30 to \(10^6\)–\(10^7\): the backward pass explodes.

The cause is the reset. Within one time step, \(v' = \beta v + W s_{in}\), \(s = H(v' - 1)\) and \(v_{new} = v' - s\), so

\[\frac{\partial v_{new}}{\partial v} = \beta\,\big(1 - \sigma'(v' - 1)\big).\]

Whenever the surrogate’s peak exceeds \(1 + 1/\beta \approx 2.1\), this factor exceeds 1 in magnitude for potentials near threshold, and the gradient grows from one time step to the next. For Sigmoid (peak \(\alpha/4\)) that is \(\alpha > 8.4\); the explosion shows up once potentials stay near threshold over consecutive steps, here between \(\alpha = 16\) and 64, and more readily in deeper networks.

With the reset detached (dashed lines), the factor is just \(\beta\) and nothing explodes: one layer trains at every steepness, three layers improve from 0.38–0.44 to 0.89 at \(\alpha \ge 64\). Detaching does not make the steepness irrelevant, though: the three-layer network is best around \(\alpha = 4\) and degrades for very flat surrogates too (\(\alpha = 0.25\): 0.83), where the gradient through each layer is small.

5. Hypergradients Through Training#

Although the loss does not depend on \(\alpha\), the result of training does: the weights after \(K\) gradient steps are a differentiable function of \(\alpha\), because \(\alpha\) shapes every weight update. Unrolling \(K\) steps of gradient descent and differentiating the validation loss of the final weights gives a hypergradient \(\partial L_{val}(w_K(\alpha)) / \partial \alpha\). This requires the derivative of surrogate_grad with respect to \(\alpha\) inside the unrolled updates, which the surrogate primitive provides.

We unroll 30 full-batch gradient-descent steps on 128 training samples from a fixed initialization and parametrize \(\alpha = e^{\theta}\) to keep it positive:

x_b, y_b = x_train[:128], y_train[:128]
x_v, y_v = x_val[:200], y_val[:200]
params0 = init_params(0, 1)
K, inner_lr = 30, 1.0

def meta_loss(log_alpha):
    spike_fn = lambda v: surrogate.sigmoid(v, jnp.exp(log_alpha))
    params = params0
    for _ in range(K):
        grads = jax.grad(cross_entropy)(params, x_b, y_b, spike_fn, 1)
        params = jax.tree.map(lambda w, g: w - inner_lr * g, params, grads)
    return cross_entropy(params, x_v, y_v, spike_fn, 1)

meta_value_and_grad = jax.jit(jax.value_and_grad(meta_loss))
grid = [0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0]
meta = [meta_value_and_grad(jnp.log(a)) for a in grid]
for a, (v, g) in zip(grid, meta):
    print(f"alpha = {a:5}: validation loss after {K} steps {float(v):.3f}   d/d log(alpha) = {float(g):+.3e}")
alpha =  0.25: validation loss after 30 steps 0.756   d/d log(alpha) = -1.454e-01
alpha =   0.5: validation loss after 30 steps 0.514   d/d log(alpha) = -2.244e-01
alpha =   1.0: validation loss after 30 steps 0.258   d/d log(alpha) = -9.324e-02
alpha =   2.0: validation loss after 30 steps 0.259   d/d log(alpha) = -1.774e-01
alpha =   4.0: validation loss after 30 steps 0.250   d/d log(alpha) = -1.562e-01
alpha =   8.0: validation loss after 30 steps 0.295   d/d log(alpha) = -1.811e+00
alpha =  16.0: validation loss after 30 steps 0.293   d/d log(alpha) = +1.204e+10
# gradient descent on log(alpha); each step is clipped to +-0.2 because the hypergradient explodes for steep surrogates
log_alpha, path = jnp.log(0.25), []
for it in range(20):
    v, g = meta_value_and_grad(log_alpha)
    path.append((float(jnp.exp(log_alpha)), float(v)))
    log_alpha = log_alpha - jnp.clip(2.0 * g, -0.2, 0.2)
print("hypergradient descent from alpha = 0.25:")
for it in [0, 4, 8, 12, 16, 19]:
    print(f"  step {it:2d}: alpha = {path[it][0]:.3f}, validation loss {path[it][1]:.3f}")

plt.figure(figsize=(6, 3.8))
plt.semilogx(grid, [float(v) for v, _ in meta], "o-", label="grid")
plt.semilogx(*zip(*path), ".-", label="hypergradient descent")
plt.xlabel("alpha"); plt.ylabel(f"validation loss after {K} steps"); plt.legend(); plt.tight_layout(); plt.show()
hypergradient descent from alpha = 0.25:
  step  0: alpha = 0.250, validation loss 0.756
  step  4: alpha = 0.556, validation loss 0.368
  step  8: alpha = 0.639, validation loss 0.312
  step 12: alpha = 0.839, validation loss 0.251
  step 16: alpha = 0.852, validation loss 0.277
  step 19: alpha = 0.773, validation loss 0.289
../_images/f5f5b02d986409445f278eff442bdbfef7fe4c8e8a677e6556f60c24b57fdc47.png

The validation loss after training falls from 0.76 at \(\alpha = 0.25\) to about 0.25 for \(\alpha\) between 1 and 4 and rises slightly beyond. In this range the hypergradient is negative, pointing towards steeper surrogates, consistent with the grid. At \(\alpha = 16\) it reaches \(10^{10}\) although the loss is fine: the exploding backward pass of section 4, compounded over 30 unrolled steps. Hypergradient descent from \(\alpha = 0.25\) reaches \(\alpha \approx 0.8\) and a validation loss of 0.25 within about 12 steps, then hovers: on small scales the loss after training is not smooth in \(\alpha\) (spike counts change in jumps), so the hypergradient is noisy near the optimum.

For a single scalar like \(\alpha\), a coarse grid on a logarithmic scale is cheaper and more robust. Hypergradients become worthwhile for many parameters at once, such as one steepness per layer, where a grid is infeasible; start them in the trainable regime and clip the steps.

Summary#

  • The forward pass uses the exact step, so the loss does not depend on surrogate parameters and jax.grad with respect to them is 0. Tune them by validation, or by differentiating through unrolled training.

  • On this task the shape of the surrogate barely matters: eight different surrogates reach the same accuracy.

  • The steepness matters. With the gradient flowing through a subtractive reset, a peak surrogate gradient above about 2 lets the backward pass through time explode; for Sigmoid this means \(\alpha \gtrsim 8\). Keep the peak moderate, or detach the reset, which makes training robust over a wide range of steepness.

  • Hypergradients through unrolled training are well defined and useful in the trainable regime; in the exploding regime they explode too.