braintools.file module#
File I/O and Checkpointing Utilities.
This module provides utilities for file input/output operations and model checkpointing, specifically designed for neuroscience and machine learning workflows.
Key Features:
MATLAB File I/O: Load and parse .mat files with automatic type conversion
Model Checkpointing: Save and restore model states using efficient msgpack serialization
Async Saving: Non-blocking checkpoint saves for better performance
Type Safety: Proper handling of BrainUnit quantities and BrainState objects
Flexible Loading: Support for mismatch handling between saved and current model structures
Neuro data formats (optional extras): HDF5 with unit metadata (
pip install braintools[hdf5]), NWB units tables and time series (braintools[nwb]), and NEO spike-train interoperability (braintools[neo])
Quick Start:
from braintools.file import load_matfile, msgpack_save, msgpack_load
# Load MATLAB data
data = load_matfile('experiment_data.mat')
# Save model checkpoint
msgpack_save('model_checkpoint.msgpack', model_state)
# Load checkpoint back
restored_state = msgpack_load('model_checkpoint.msgpack', target=model_state)
MATLAB File Loading:
from braintools.file import load_matfile, save_matfile
# Load with default settings (excludes MATLAB headers)
data = load_matfile('data.mat')
# Include MATLAB metadata
data = load_matfile('data.mat', include_header=True)
# Access nested structures (automatically converted to Python dicts/lists)
spike_times = data['trial_data']['spike_times']
# Save data back to a MATLAB .mat file
save_matfile('out.mat', {'spike_times': spike_times})
Model Checkpointing:
import brainstate
from braintools.file import msgpack_save, msgpack_load, AsyncManager
# Simple synchronous save
msgpack_save('checkpoint.msgpack', model.states())
# Load checkpoint with mismatch handling
state = msgpack_load('checkpoint.msgpack', target=model.states(), mismatch='warn') # restores in place
# Async saving for large models (non-blocking)
with AsyncManager() as manager:
msgpack_save('checkpoint.msgpack', model.states(), async_manager=manager)
# Continue training while save happens in background
# Custom serialization for user-defined types
from braintools.file import msgpack_register_serialization
def my_type_to_dict(obj):
return {'data': obj.data}
def my_type_from_dict(obj, state_dict, mismatch='error'):
obj.data = state_dict['data']
return obj
msgpack_register_serialization(MyType, my_type_to_dict, my_type_from_dict)
Neuro data formats:
import brainunit as u
import jax.numpy as jnp
from braintools.spike import SpikeTrains
from braintools.file import save_hdf5, load_hdf5, save_nwb_units, load_nwb_units, to_neo
trains = SpikeTrains.from_list([jnp.array([1.0, 5.0]) * u.ms, jnp.array([2.0]) * u.ms])
# Nested pytrees of arrays, Quantities and SpikeTrains (needs h5py)
save_hdf5('run.h5', {'v': jnp.zeros(3) * u.mV, 'spikes': trains})
restored = load_hdf5('run.h5')
# NWB units table (needs pynwb); times are stored in seconds
save_nwb_units('session.nwb', trains, session_description='demo', identifier='run-1')
trains_s = load_nwb_units('session.nwb')
# NEO spike trains (needs neo and quantities)
neo_trains = to_neo(trains)
Utilities for loading and saving experiment artifacts, including MATLAB archives and high-performance MsgPack checkpoints.
MATLAB I/O#
load_matfile() reads a .mat file (via scipy.io.loadmat()),
recursively converting MATLAB structs to dicts and cell arrays to lists. Pass
include_header=True to keep the __header__ / __version__ /
__globals__ metadata. MATLAB v7.3 (HDF5) files are not supported and raise
NotImplementedError. save_matfile() writes a dict of variables back to
a .mat file.
Checkpointing#
msgpack_save() / msgpack_load() serialize PyTrees (including
brainunit.Quantity and brainstate.State leaves) to and from
msgpack. Notes:
Mismatch handling. When a
targetis given,mismatchcontrols what happens on a structural difference (dict keys, list/tuple length, namedtuple fields, unit, array shape):'error'(default) raises,'warn'warns and keeps the target’s value,'ignore'keeps it silently.In-place State restore.
Stateleaves are restored in place: the template’s.valueis mutated. Pass a throwaway template to preserve the original.Large arrays. Arrays above ~1 GiB are transparently chunked to bypass the msgpack 2 GiB leaf limit.
msgpack_loadaccepts an optionalmax_sizeguard (None= unlimited).Async saves.
AsyncManagerruns the file write in the background; serialization happens synchronously so the on-disk snapshot is consistent.Custom types. Register handlers with
msgpack_register_serialization().
A simple function to load a .mat file using scipy from Python. |
|
Save a dictionary of variables to a MATLAB |
|
Restores the state of the given target using a state dict. |
|
Returns a dictionary with the state of the given target. |
|
Register a type for serialization. |
|
Save a checkpoint of the model. |
|
Load the checkpoint from the given path using the |
|
A simple object to track async checkpointing. |
|
Raised when a rename would overwrite an existing file. |
|
Raised for an invalid checkpoint target or corrupt checkpoint data. |
Neuro Data Formats#
Optional backends for standard neuroscience formats; each imports its library
lazily and raises an ImportError with the install hint when it is missing.
HDF5 (
pip install braintools[hdf5]):save_hdf5()/load_hdf5()round-trip nested dicts, lists and tuples of arrays and scalars,brainunit.Quantityleaves (with their exact unit) andbraintools.spike.SpikeTrains.NWB (
pip install braintools[nwb]):save_nwb_units()/load_nwb_units()map a units table to and fromSpikeTrains(times stored in seconds);save_nwb_timeseries()/load_nwb_timeseries()store a regularly sampled signal.NEO (
pip install braintools[neo]):to_neo()/from_neo()convert betweenSpikeTrainsand a list ofneo.SpikeTrain.
Save a nested tree of arrays, Quantities and spike trains to an HDF5 file. |
|
Load a tree saved with |
|
Write a |
|
Read the Units table of an NWB file into a |
|
Write a regularly sampled signal as an NWB |
|
Read a regularly sampled NWB |
|
Convert a |
|
Build a |