Pipeline spec

A pipeline: operations (nodes, by id) and the edges between them.

Download the schema for your editor or validator: pipeline.schema.json.

The spec

Field	Type	Required	Description
dspk_pipeline	integer	yes	Spec version; must equal SPEC_VERSION.
name	string		Free-form name.
nodes	object	yes	Every operation, by node id.
edges	list of list	yes	[from, to] node ids.
execution	Execution		Default: {"mode": "auto", "chunk_samples": 65536, "upload": "auto"}.

Nodes

A node’s op says what it is. read and write are the ends; the others process.

read

The input recording(s): path (one file, or a glob such as sessions/*.bin, which runs the pipeline once per match) or inputs (a list of paths / globs and merges, run one after the other). A merge reads several recordings as one, channels side by side. Relative paths are relative to the spec file. With several inputs the write path needs {stem} or {index}.

Field	Type	Required	Description
path	string
inputs	list of Input
source	string		Source id within the file (dspk_io::sources; with path only); default: the file’s default source.
channels	list of integer		Channels to read, in this order; default: all.
start	integer		First sample to process (samples, not seconds, so a range is exact); default: 0.
stop	integer		Sample after the last one to process; default: the end of the recording. Halos of window mode may read real data outside start..stop.

write

The output file, float32 whatever the input’s stored type (every recording processed by a pipeline comes out in the same form). Relative paths are relative to the spec file.

Field	Type	Required	Description
path	string	yes
format	"raw" | "zarr"		Default: "raw".
compression	Compression		Zarr only: chunk compression; default zstd level 3 (dspk_io::DEFAULT_ZARR_COMPRESSION). Chunks are compressed and stored in parallel.
chunk_samples	integer		Zarr only: samples per chunk along time (every chunk spans all channels); default dspk_io::generic::zarr_traces::DEFAULT_CHUNK_SAMPLES (5000).
parallel	boolean		Write in parallel (Zarr: chunks compressed and stored on several threads; raw: frames built on several threads); default true. false writes on one thread (e.g. a machine busy with other work, or to compare).

scale

y = alpha * x + beta.

Field	Type	Required	Description
alpha	number	yes
beta	number		Default: 0.0.

subtract_baseline

y = x - baseline.

Field	Type	Required	Description
baseline	number	yes

clamp

y = clip(x, min, max).

Field	Type	Required	Description
min	number	yes
max	number	yes

rectify

y = \|x\| (full-wave rectification; then a low-pass gives an EMG envelope).

square

y = x² (instantaneous power; then a moving average and sqrt give a moving RMS).

sqrt

y = √x, negative values taken as 0.

log10

y = log₁₀ \|x\|, magnitudes below 1e-12 taken as 1e-12.

laplacian

Surface Laplacian of an electrode grid of [rows, cols] (channels in row-major order: channel r · cols + c): each channel minus the mean of its orthogonal neighbours, which removes what volume conduction spreads over the grid (HD-EMG, ECoG). The chain’s channels must number rows · cols.

Field	Type	Required	Description
grid	list of integer	yes

whiten

Spatial whitening (global ZCA): decorrelates the channels and gives each unit variance, as sorters do before detection. Fitted before the run on the signal the nodes before it produce: fit_s seconds in chunks chunks spread over the recording; epsilon regularises the covariance (in the signal’s unit squared).

Field	Type	Required	Description
epsilon	number		Default: 9.999999974752427e-07.
fit_s	number		Default: 10.0.
chunks	integer		Default: 5.

iir

IIR filter designed for the recording’s sample rate.

Field	Type	Required	Description
band	"lowpass" | "highpass" | "bandpass" | "bandstop"	yes
cutoff	list of number	yes	Hz: one value (lowpass, highpass) or [low, high] (bandpass, bandstop).
order	integer		Default: 5.
design	"butterworth" | "chebyshev1" | "bessel"		Default: "butterworth".
ripple_db	number		Pass-band ripple, chebyshev1 only.
mode	"forward_backward" | "forward"		Default: "forward_backward".
start	"rest" | "steady_state"		Default: "rest".

notch

Second-order notch (scipy.signal.iirnotch).

Field	Type	Required	Description
freq	number	yes
q	number	yes
mode	"forward_backward" | "forward"		Default: "forward_backward".
start	"rest" | "steady_state"		Default: "rest".

sos

Explicit second-order sections [b0, b1, b2, a0, a1, a2] (scipy layout), already designed for the recording’s sample rate.

Field	Type	Required	Description
sections	list of list of number	yes
mode	"forward_backward" | "forward"		Default: "forward_backward".
start	"rest" | "steady_state"		Default: "rest".

fir

FIR filter with explicit taps (design them with dspk.ops.filter.firwin or scipy): causal, or centered (odd number of taps around each sample).

Field	Type	Required	Description
taps	list of number	yes
centered	boolean		Default: false.
edge	"zeros" | "odd" | "reflect" | "nearest"		Default: "zeros".

resample

Resampling by up / down (scipy.signal.resample_poly: polyphase FIR, Kaiser beta window); must be the last node before write. The output’s sample rate is rate · up / down; with a read range, its start must be a multiple of down.

Field	Type	Required	Description
up	integer	yes
down	integer	yes
beta	number		Default: 5.0.
edge	"zeros" | "odd" | "reflect" | "nearest"		Default: "zeros".

remap

Channels reordered, dropped and renamed by pairs, like a hardware channel mapper: pairs written here (base 0, or 1 for 1-based numbers), or path to a map file (.toml / .json with the pairs under remap; relative to the spec file). Output channel to reads input from; inputs left out are dropped. Right after read (or another remap): it folds into the read, at no cost.

Field	Type	Required	Description
base	integer		Default: 0.
pairs	list of ChannelPair
path	string

car

Common average reference.

cmr

Common median reference.

gaussian_smooth

Zero-phase Gaussian smoothing.

Field	Type	Required	Description
sigma_samples	number	yes
edge	"zeros" | "odd" | "reflect" | "nearest"		Default: "reflect".

median

Running median over an odd width.

Field	Type	Required	Description
width	integer	yes
edge	"zeros" | "odd" | "reflect" | "nearest"		Default: "zeros".

teager_kaiser

Teager–Kaiser energy.

Field	Type	Required	Description
edge	"zeros" | "odd" | "reflect" | "nearest"		Default: "reflect".

Input

One input of a read node: a path (or glob), or several recordings merged side by side.

• string

• object

Field	Type	Required	Description
merge	list of MergePart	yes	The recordings, in channel order; same sample rate and length.

MergePart

One recording of a merge.

Field	Type	Required	Description
path	string	yes
source	string		Source id within the file; default: the file’s default source.
channels	list of integer		Channels taken from this recording, in order; default: all.

Compression

Compression of Zarr chunks: {"codec": "zstd", "level": 3}, {"codec": "blosc_lz4", "level": 5} (LZ4 after a byte shuffle; faster, usually a little larger) or {"codec": "none"}.

• object

Field	Type	Required	Description
codec	"none"	yes

• object

Field	Type	Required	Description
codec	"zstd"	yes
level	integer		Default: 3.

• object

Field	Type	Required	Description
codec	"blosc_lz4"	yes
level	integer		Default: 5.

ChannelPair

One output channel: input channel from becomes output channel to, optionally named.

Field	Type	Required	Description
from	integer	yes
to	integer	yes
name	string

Execution

How the executor runs the chain.

Field	Type	Required	Description
mode	"auto" | "stream" | "windows"		Default: "auto".
chunk_samples	integer		Samples per channel processed per window. Default: 65536.
runtime	string		wgpu, cuda, hip, cpu; default: DSPK_RUNTIME, else the first compiled in.
upload	"auto" | "stored" | "scaled"		Default: "auto".