Save and Load Model States with msgpack Checkpointing#

braintools.file provides a compact checkpoint format based on msgpack. It stores nested Python containers whose leaves are NumPy/JAX arrays, Python scalars, strings, brainunit quantities, and brainstate states.

Function / class

Purpose

msgpack_save(filename, target, overwrite=True, async_manager=None)

Write a checkpoint (atomically, via a temporary file and a rename)

msgpack_load(filename, target=None, mismatch='error')

Read a checkpoint, optionally restoring it into a template target

AsyncManager

Run the file write in a background thread

msgpack_register_serialization(cls, to_dict, from_dict)

Teach the format about your own classes

msgpack_to_state_dict / msgpack_from_state_dict

The in-memory conversion both functions use

InvalidCheckpointPath

Raised for corrupt files and for overwrite=False on an existing file

The single most important idea is the template: a checkpoint stores values, not Python types, so you load it into an object with the expected structure.

1. Setup#

All files in this tutorial go to a temporary directory that is removed at the end.

import shutil
import tempfile
import warnings
from pathlib import Path

import brainstate
import brainunit as u
import jax.numpy as jnp
import numpy as np

import braintools
from braintools.file import AsyncManager, InvalidCheckpointPath, msgpack_load, msgpack_save

workdir = Path(tempfile.mkdtemp(prefix="braintools_ckpt_"))
print("checkpoints go to a temporary directory")
checkpoints go to a temporary directory

2. Saving and Loading Plain Data#

msgpack_save accepts any nesting of dicts, lists, tuples, and named tuples whose leaves are arrays, numbers, strings, booleans, or None. It creates missing parent directories and returns the file name.

state = {
    "weights": jnp.array([[1.0, 2.0], [3.0, 4.0]]),
    "bias": np.array([0.1, 0.2], dtype=np.float32),
    "layer_sizes": (2, 2),
    "config": {"activation": "relu", "learning_rate": 1e-3, "use_bias": True},
    "step": 1000,
}

path = msgpack_save(workdir / "run" / "basic.msgpack", state)
print(f"saved {Path(path).name} ({Path(path).stat().st_size} bytes)")
saved basic.msgpack (155 bytes)

Loading without a template returns the raw stored structure: arrays come back as NumPy arrays, every dict key is a string, and tuples and lists become dicts keyed '0', '1', … because the format has no tuple type.

raw = msgpack_load(workdir / "run" / "basic.msgpack")

for key, value in raw.items():
    print(f"{key:12s} -> {type(value).__name__:8s} {value if not hasattr(value, 'shape') else value.shape}")
weights      -> ndarray  (2, 2)
bias         -> ndarray  (2,)
layer_sizes  -> dict     {'0': 2, '1': 2}
config       -> dict     {'activation': 'relu', 'learning_rate': 0.001, 'use_bias': True}
step         -> int      1000

Loading with a template restores the original structure and types. The template only needs the right structure; its leaf values are replaced by the stored ones. Arrays must have the same shape (dtype is not enforced).

template = {
    "weights": jnp.zeros((2, 2)),
    "bias": np.zeros(2, dtype=np.float32),
    "layer_sizes": (0, 0),
    "config": {"activation": "", "learning_rate": 0.0, "use_bias": False},
    "step": 0,
}

restored = msgpack_load(workdir / "run" / "basic.msgpack", target=template)
print(f"layer_sizes: {restored['layer_sizes']} ({type(restored['layer_sizes']).__name__})")
print(f"config:      {restored['config']}")
print(f"weights equal: {bool(jnp.array_equal(restored['weights'], state['weights']))}")
layer_sizes: (2, 2) (tuple)
config:      {'activation': 'relu', 'learning_rate': 0.001, 'use_bias': True}
weights equal: True

In practice the template is usually the freshly constructed object you want to continue with, e.g. a newly built model, so the template costs nothing extra.

3. Restoring brainstate Models#

model.states() collects every State of a brainstate.nn.Module. Save it, then load the checkpoint into the states() of a new instance: State leaves are restored in place, so the new model is ready to use immediately.

class TinyNet(brainstate.nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = brainstate.nn.Linear(4, 8)
        self.readout = brainstate.nn.Linear(8, 2)
        self.running_mean = brainstate.ShortTermState(jnp.zeros(8))

    def __call__(self, x):
        h = jnp.tanh(self.hidden(x))
        return self.readout(h)


brainstate.random.seed(0)
net = TinyNet()
net.running_mean.value = jnp.full(8, 0.5)  # some non-parameter state

msgpack_save(workdir / "tinynet.msgpack", {"model": net.states(), "epoch": 12})

brainstate.random.seed(1)
fresh = TinyNet()  # different random initialization
x = jnp.ones((3, 4))
print(f"before loading, outputs match: {bool(jnp.allclose(net(x), fresh(x)))}")

checkpoint = msgpack_load(workdir / "tinynet.msgpack", target={"model": fresh.states(), "epoch": 0})
print(f"after loading,  outputs match: {bool(jnp.allclose(net(x), fresh(x)))}")
print(f"running_mean restored: {fresh.running_mean.value[:3]}")
print(f"epoch: {checkpoint['epoch']}")
before loading, outputs match: False
after loading,  outputs match: True
running_mean restored: [0.5 0.5 0.5]
epoch: 12

Use states(brainstate.ParamState) instead of states() if you only want to store trainable parameters.

4. Physical Units#

brainunit quantities are stored as their numeric value plus the unit’s dimension and scale. Loading with a template restores the quantity with its unit, and a template with an incompatible unit is reported as a mismatch.

neuron_params = {
    "tau_m": 10.0 * u.ms,
    "R_m": 100.0 * u.Mohm,
    "C_m": 200.0 * u.pF,
    "V_th": jnp.array([-50.0, -52.0, -48.0]) * u.mV,
}
msgpack_save(workdir / "neuron.msgpack", neuron_params)

loaded_params = msgpack_load(workdir / "neuron.msgpack", target=neuron_params)
for name, value in loaded_params.items():
    print(f"{name:5s} = {value}")

try:
    msgpack_load(workdir / "neuron.msgpack", target={**neuron_params, "tau_m": 10.0 * u.second})
except ValueError as exc:
    print(f"\nunit mismatch -> ValueError: {exc}")
tau_m = 10. ms
R_m   = 100. Mohm
C_m   = 200. pF
V_th  = [-50. -52. -48.] mV

unit mismatch -> ValueError: Unit mismatch: expected s, got 10.0^-3 * s at path ./tau_m

Without a template a quantity comes back as its raw stored form, a dict with 'mantissa', 'scale', 'base', 'dim', and 'factor', so always load unit-carrying checkpoints into a template.

raw_tau = msgpack_load(workdir / "neuron.msgpack")["tau_m"]
print(f"tau_m without a template: {type(raw_tau).__name__} with keys {list(raw_tau)}")
tau_m without a template: dict with keys ['mantissa', 'scale', 'base', 'dim', 'factor']

5. Handling Structural Mismatches#

Models evolve: layers are added, removed, or resized. mismatch controls what happens when the template and the checkpoint disagree:

mismatch=

Effect

'error' (default)

raise ValueError

'warn'

emit a UserWarning and continue

'ignore'

continue silently

When loading continues, the rules are:

  • a key in the template but not in the checkpoint keeps the template’s value (newly added components keep their fresh initialization);

  • a key in the checkpoint but not in the template is dropped without a warning or error, even in 'error' mode;

  • an array with a different shape is replaced by the stored array, so the result has the checkpoint’s shape;

  • a list or tuple of a different length is restored element-wise for the common prefix.

old_model = {
    "layer1": {"w": jnp.ones((5, 3))},
    "layer2": {"w": jnp.ones((3, 2))},
    "dropped_layer": {"w": jnp.ones(4)},
}
msgpack_save(workdir / "old_model.msgpack", old_model)

new_model = {
    "layer1": {"w": jnp.zeros((5, 3))},
    "layer2": {"w": jnp.zeros((3, 2))},
    "layer3": {"w": jnp.full((2, 1), 7.0)},  # new layer, not in the checkpoint
}

try:
    msgpack_load(workdir / "old_model.msgpack", target=new_model)
except ValueError as exc:
    print(f"'error':  ValueError: {exc}")

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    merged = msgpack_load(workdir / "old_model.msgpack", target=new_model, mismatch="warn")
print(f"'warn':   {len(caught)} warning(s)")
print(f"  layer1 loaded from checkpoint: {bool(jnp.all(merged['layer1']['w'] == 1))}")
print(f"  layer3 kept from template:     {bool(jnp.all(merged['layer3']['w'] == 7))}")
print(f"  dropped_layer in result:       {'dropped_layer' in merged}")

merged = msgpack_load(workdir / "old_model.msgpack", target=new_model, mismatch="ignore")
print(f"'ignore': loaded silently, keys {sorted(merged)}")
'error':  ValueError: The target dict keys and state dict keys do not match, target dict contains keys {'layer3'} which are not present in state dict at path .
'warn':   1 warning(s)
  layer1 loaded from checkpoint: True
  layer3 kept from template:     True
  dropped_layer in result:       False
'ignore': loaded silently, keys ['layer1', 'layer2', 'layer3']
# A shape change: the stored array wins when loading continues.
resized = {"layer1": {"w": jnp.zeros((5, 4))}, "layer2": {"w": jnp.zeros((3, 2))}}
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    loaded = msgpack_load(workdir / "old_model.msgpack", target=resized, mismatch="warn")
print(f"warning: {caught[0].message}")
print(f"layer1 shape after loading: {loaded['layer1']['w'].shape}  (template asked for (5, 4))")
warning: Array shape mismatch: expected (5, 4), got (5, 3) at path ./layer1/w
layer1 shape after loading: (5, 3)  (template asked for (5, 4))

Because a shape mismatch under 'warn'/'ignore' hands you an array of the old shape, check shapes yourself before using a partially compatible checkpoint.

6. Custom Classes#

To store your own class, register two functions:

  • to_dict(obj) returns a dict of serializable values;

  • from_dict(obj, state_dict, mismatch='error') receives the template object and the stored dict and returns the restored object.

Inside them, msgpack_to_state_dict and msgpack_from_state_dict convert nested values (arrays, quantities, other registered classes), so units and structure are handled for you.

from braintools.file import (
    msgpack_from_state_dict,
    msgpack_register_serialization,
    msgpack_to_state_dict,
)


class SynapseGroup:
    """A plain Python class holding synaptic parameters."""

    def __init__(self, n_pre, n_post, tau=5.0 * u.ms):
        self.n_pre = n_pre
        self.n_post = n_post
        self.tau = tau
        self.weights = jnp.zeros((n_pre, n_post))


def synapse_to_dict(group):
    return {
        "shape": msgpack_to_state_dict((group.n_pre, group.n_post)),
        "tau": msgpack_to_state_dict(group.tau),
        "weights": group.weights,
    }


def synapse_from_dict(group, state_dict, mismatch="error"):
    n_pre, n_post = msgpack_from_state_dict((group.n_pre, group.n_post), state_dict["shape"], mismatch=mismatch)
    restored = SynapseGroup(n_pre, n_post)
    restored.tau = msgpack_from_state_dict(group.tau, state_dict["tau"], mismatch=mismatch)
    restored.weights = msgpack_from_state_dict(group.weights, state_dict["weights"], mismatch=mismatch)
    return restored


msgpack_register_serialization(SynapseGroup, synapse_to_dict, synapse_from_dict)

syn = SynapseGroup(3, 4, tau=8.0 * u.ms)
syn.weights = jnp.arange(12.0).reshape(3, 4)
msgpack_save(workdir / "synapses.msgpack", {"exc": syn})

loaded = msgpack_load(workdir / "synapses.msgpack", target={"exc": SynapseGroup(3, 4)})["exc"]
print(f"type: {type(loaded).__name__}, shape: ({loaded.n_pre}, {loaded.n_post}), tau: {loaded.tau}")
print(f"weights restored: {bool(jnp.array_equal(loaded.weights, syn.weights))}")
type: SynapseGroup, shape: (3, 4), tau: 8. ms
weights restored: True

7. Asynchronous Saving#

With an AsyncManager, msgpack_save serializes the data immediately and writes the file in a background thread, so training can continue during the disk write. The saved file is a snapshot taken when msgpack_save was called; later changes to the model do not leak into it.

A later save waits for the previous write to finish, and leaving the with block (or calling wait_previous_save()) waits for the last one.

params = {"w": jnp.zeros((256, 256))}

with AsyncManager(max_workers=1) as manager:
    for epoch in range(3):
        msgpack_save(workdir / f"async_epoch_{epoch}.msgpack", {"params": params, "epoch": epoch},
                     async_manager=manager)
        # Training continues while the file is written...
        params = {"w": params["w"] + 1.0}

for epoch in range(3):
    stored = msgpack_load(workdir / f"async_epoch_{epoch}.msgpack")
    print(f"epoch {epoch}: stored w = {float(stored['params']['w'][0, 0])}")
epoch 0: stored w = 0.0
epoch 1: stored w = 1.0
epoch 2: stored w = 2.0

8. Errors and Safe Overwrites#

  • A missing file raises ValueError("Checkpoint not found: ...").

  • A corrupt or truncated file raises InvalidCheckpointPath.

  • overwrite=False refuses to replace an existing file (InvalidCheckpointPath).

  • Writes are atomic: data goes to a temporary file that is renamed into place, so an interrupted save never leaves a half-written checkpoint under the final name.

try:
    msgpack_load(workdir / "does_not_exist.msgpack")
except ValueError:
    print("missing file   -> ValueError")

(workdir / "corrupt.msgpack").write_bytes(b"definitely not msgpack")
try:
    msgpack_load(workdir / "corrupt.msgpack")
except InvalidCheckpointPath:
    print("corrupt file   -> InvalidCheckpointPath")

try:
    msgpack_save(workdir / "neuron.msgpack", neuron_params, overwrite=False)
except InvalidCheckpointPath:
    print("overwrite=False on an existing file -> InvalidCheckpointPath")
missing file   -> ValueError
corrupt file   -> InvalidCheckpointPath
overwrite=False on an existing file -> InvalidCheckpointPath

9. Pattern: Periodic Checkpoints and Resuming#

A typical training script saves a checkpoint every few epochs and, on start-up, resumes from the newest one if it exists. Everything needed to continue goes into one dict: model states, the epoch counter, and the random-number state (brainstate.random.DEFAULT is itself a State).

def save_checkpoint(directory, model, epoch):
    payload = {"model": model.states(), "epoch": epoch, "rng": brainstate.random.DEFAULT}
    return msgpack_save(Path(directory) / f"epoch_{epoch:04d}.msgpack", payload)


def resume(directory, model):
    """Load the newest checkpoint into ``model``; return the epoch to start from."""
    checkpoints = sorted(Path(directory).glob("epoch_*.msgpack"))
    if not checkpoints:
        return 0
    template = {"model": model.states(), "epoch": 0, "rng": brainstate.random.DEFAULT}
    restored = msgpack_load(checkpoints[-1], target=template)
    return restored["epoch"] + 1


run_dir = workdir / "training_run"
brainstate.random.seed(42)
model = TinyNet()
for epoch in range(resume(run_dir, model), 6):
    # ... one epoch of training would update model here ...
    if epoch % 2 == 1:
        save_checkpoint(run_dir, model, epoch)
print(f"checkpoints: {[p.name for p in sorted(run_dir.glob('*.msgpack'))]}")

# A new process starts from scratch and resumes:
restarted = TinyNet()
start_epoch = resume(run_dir, restarted)
print(f"resuming at epoch {start_epoch}; parameters match: {bool(jnp.allclose(model(x), restarted(x)))}")
checkpoints: ['epoch_0001.msgpack', 'epoch_0003.msgpack', 'epoch_0005.msgpack']
resuming at epoch 6; parameters match: True

10. Clean Up#

shutil.rmtree(workdir)
print("temporary checkpoint directory removed")
temporary checkpoint directory removed

Summary#

  • msgpack_save writes nested containers of arrays, scalars, strings, quantities, and States atomically; AsyncManager moves the write to a background thread while keeping a consistent snapshot.

  • Load into a template (target=) to recover tuples, units, custom classes, and to restore brainstate models in place. Without a template you get the raw stored dicts.

  • mismatch='error' | 'warn' | 'ignore' controls template/checkpoint disagreements. Template-only keys keep their values, checkpoint-only keys are dropped silently, and arrays with a changed shape take the stored shape.

  • Register to_dict / from_dict with msgpack_register_serialization for your own classes.

  • Missing files raise ValueError; corrupt files and overwrite=False collisions raise InvalidCheckpointPath.