braindecode · NeuralBench · SDSC Expanse

Do the iEEG ports reproduce their public results?

Mostly. Six intracranial models were ported to braindecode, and five have a public number to check against. Rerun in NeuralBench on the Brain Treebank, 27 of 31 split and task groups land within 0.05 AUROC of that number. PopT and one BaRISTA split do not, yet.

The dataset · Brain Treebank on NEMAR (nm000253)

One dataset, one protocol

To test whether the braindecode ports reproduce their public results, we ran each of them in NeuralBench on the Brain Treebank, with the splits, windows and readouts of the original evaluation. The public target is the Neuroprobe leaderboard for BrainBERT, PopT, MAPA and DIVER-1, and the paper tables for BaRISTA. A split and task pass when our mean test AUROC lies within 0.05 of the public mean over the same test cases. Overall, these results show that 27 of 31 split and task groups pass (Fig. 1), and that the four that fail come from PopT and from the chronological split of BaRISTA.

In the text

“intracranial recordings from 10 epilepsy patients collected over a total of 26 sessions as they watched Hollywood films […] Neural data is provided at a sampling rate of 2048 Hz.”
BaRISTA paper, describing the Brain Treebank (Wang et al., 2024)
“For all linear regression, we used the sklearn package […] In all cases, the features were first normalized using the sklearn StandardScaler.”
Neuroprobe paper, appendix

In NeuralBench

  • Recordings from NEMAR nm000253 v1.0.0, read as BIDS.
  • Neuroprobe 0.1.8 splits for 15 tasks: across sessions, within session (2 folds) and across subjects. Lite uses 12 subject and session pairs; across subjects, the probe is trained on subject 2 and tested on the other 10 pairs.
  • Three NEMAR gaps filled from the original release: subject 1's channel names, the coordinates of subject 2's “#” contacts, and “*” renamed to “x” in 62 channels of subjects 2, 7, 9 and 10.
  • Readout for the frozen models: StandardScaler, then LogisticRegression (random state 42, 10,000 iterations, tolerance 0.001), fitted on the training split only.
Fig. 1. Most ports reproduce their public results within 0.05. Each row shows one model and one split. Each small dot is one test cell (a test case, fold or seed), placed at the difference between our test AUROC and the public AUROC. The large mark is the mean across cells, and its bar the SEM across cells. The shaded band is the tolerance of 0.05 on each side of exact agreement (vertical line at 0). Cells beyond 0.2 on either side are drawn at the edge. Right: tasks that pass, out of the tasks run.

How to read the panels below (Fig. 2): our AUROC against the public AUROC, for each test cell. Each dot is a test cell, and its colour and shape give the split. The diagonal is exact agreement and the shaded band the tolerance of 0.05. Dashed lines indicate chance (0.5). Hover or focus a dot for its case and task.

BrainBERT · #1104 · Wang et al., 2023

BrainBERT reproduces the leaderboard in all three splits

To test the BrainBERT port, we followed the public runner of the leaderboard entry: the same rereferenced contacts, the same spectrogram, every output token, and the same logistic regression. Our AUROC differs from the leaderboard by +0.007 across sessions (17 cells), +0.004 across subjects (18 cells) and +0.021 within sessions (34 cells; Fig. 2A), so that all 10 groups pass. The largest task differences are within sessions: sentence onset runs higher than the leaderboard (+0.030), and frame brightness lower (-0.037). To test whether this difference comes from our pipeline, we ran the official runner, unchanged, on our rereferenced copy: for sentence onset in subject 1, session 1, it gives 0.854 and 0.824 on the two folds, against 0.854 and 0.816 with NeuralBench and 0.735 and 0.784 on the leaderboard. The difference within sessions thus lies between the public code and the leaderboard file, whose rereferenced data are not public.

In the text

“the BrainTreebank data was Laplacian rereferenced […], with line noise removed, and then passed into the BrainBERT model as provided by [Wang et al.]. The output features were concatenated and used as input to the linear regression. For the electrodes which could not be Laplacian rereferenced, non-rereferenced data was inputted into BrainBERT.”
Neuroprobe paper, appendix
“This model was taken off-the-shelf and had per-window STFT z-scoring.”
Leaderboard entry, A. Zahorodnii, 2 May 2026

In NeuralBench

  • Lite contacts, each referenced to its ±1 neighbours among the non-corrupted contacts; contacts without a neighbour keep their raw signal.
  • Line-noise notches at 60 to 300 Hz (Q 12, zero phase).
  • STFT as the public runner azaho/BrainBERT@neuroprobe-eval: window 400, overlap 350, no edge padding, first 40 bins, z-scored per window.
  • Every output token kept (keepall); across subjects, features averaged per Desikan-Killiany region shared with subject 2.
recipe brainbert_frozen_perwindow_stft · braindecode/brainbert-pretrained @ d5abbde6
Fig. 2A. BrainBERT.
MAPA · #1178 · B. Tang, 2026

MAPA reproduces once its session front end is kept

The released MAPA evaluation normalizes each band of each contact by statistics of the whole session, and only then cuts the windows. This step matters: with a per-window front end, the cross-subject AUROC of subject 3, session 1 drops to 0.918, against 0.970 with the session front end and 0.961 on the leaderboard. With the session front end, all 7 groups pass (-0.013 across sessions, +0.009 across subjects, -0.006 within sessions; Fig. 2B).

In the text

“A masked autoencoder for intracranial EEG, evaluated with a frozen encoder and a linear probe.”
Leaderboard entry, B. Tang, 9 Sep 2026. The protocol itself is the released code, evals/neuroprobe/encode.py and readout.py.

In NeuralBench

  • On the whole session: mean reference per shaft, notches at 60 to 300 Hz, 0.5 Hz high-pass.
  • Three band spectrograms (hop 64, 32 frames per second), each bin normalized by the session median and 1.4826 × MAD; 1 s windows cut from this series.
  • Raw output of encoder block 12 per contact, kept in fp16; across subjects, averaged per shared DKT region.
  • MAPA's ridge, its penalty chosen on the validation half.
recipe mapa_neuroprobe · bentang18/MAPA @ 988efbf3
Fig. 2B. MAPA.
DIVER-1 · #1170 · Y. Gwon, 2026

DIVER-1 matches in linear probing and in fine-tuning

DIVER-1 scales attention by one over the head dimension, rather than by its square root. Without this, the port differed from the official model by up to 0.34; with it, the outputs are identical. The frozen features then reach 0.724 against 0.723 on the leaderboard, and full fine-tuning 0.962 against 0.954 (Fig. 2C). These fine-tuning runs used seed 33, whereas the DIVER-1 script uses seed 41; the seed 41 runs are queued.

In the text

“linear probing on frozen DIVER-1(0.1spatch,tiny,pretrained on iEEG) model”
“full finetuing [sic] on DIVER-1(0.1spatch,tiny) pretrained model”
Leaderboard entries, Y. Gwon, 9 Jan 2026 (within session only)

In NeuralBench

  • As scripts/finetune_neuroprobe.sh: AdamW (lr 2e-3, weight decay 0.01), up to 40 epochs, early stopping on validation AUROC (patience 10), label smoothing 0.1.
  • Laplacian reference, resampling to 500 Hz, flatten and linear head.
  • Frozen recipe: only the head is trained.
  • Fix added to the PR: mup_attention=True.
recipes diver1_tiny_frozen, diver1_tiny_finetuned · official iEEG checkpoint, converted
Fig. 2C. DIVER-1, frozen and fine-tuned.
BaRISTA · #1173 · Oganesian et al.

BaRISTA exceeds its paper on the random split

BaRISTA was evaluated with its own split rules, 3 s windows and 5 fine-tuning seeds. On the random split, it reaches 0.936 against 0.912 for the per-subject values of the paper (38 runs; Fig. 2D). This difference is small but has the same sign for both tasks (+0.025 for sentence onset, +0.023 for speech). This consistency suggests a shared cause, such as the balancing seed of the paper, which is not public. The chronological split does not reproduce: our first runs exceed the paper by +0.092 (24 runs). However, these runs come from an older build and only use seed 0, so the chronological split is being run again before we interpret this gap.

In the text

“we generated non-overlapping 3-second-long neural data segments and randomly assigned them to 80/10/10 train/valid/test splits”
“we allowed for overlapping neural segments and generated the 80/10/10 train/valid/test splits chronologically in time to avoid any overlap between these splits.”
BaRISTA paper, section on the dataset

In NeuralBench

  • BaRISTA's clean-laplacian contact list, ventricle contacts removed.
  • Causal notch filters (Q 30), each contact referenced to its two neighbours.
  • 3 s windows with the paper's labels, split and balancing rules; seed 0 for balancing.
  • Tokenizer frozen, learned pooling head; one learning rate instead of upstream's two groups.
recipe barista_parcels_paper · braindecode/BaRISTA-parcels @ 56c42de5
Fig. 2D. BaRISTA. For BaRISTA, the public value is the per-subject value of the paper (random split) or its mean over subjects (chronological split).
PopT · #1105 · Chau et al., 2025

PopT remains below its per-window entry

The runner of the PopT entry is not public. We therefore tried 4 variants of the pipeline on two cells, and kept the one closest to the leaderboard: BrainBERT tokens averaged over all frames, and the PopT CLS token. PopT nevertheless remains below the leaderboard (-0.046 across sessions, -0.032 within sessions), and sentence onset fails in both splits (-0.074 and -0.067; Fig. 2E). Since the same BrainBERT features pass on their own, these results suggest that the gap lies in the unknown runner rather than in the port.

In the text

“PopulationTransformer model (Chau et al. 2024). This model was taken off-the-shelf and had per-window STFT z-scoring.”
Leaderboard entry, A. Zahorodnii, 2 May 2026 (across and within sessions only)
“We use the publicly available weights which were pretrained on data from 10 iEEG subjects, using 5s BrainBERT temporal embeddings from individual channels.”
Neuroprobe paper, appendix

In NeuralBench

  • The same rereferenced Lite contacts and per-window STFT as BrainBERT.
  • Per contact, BrainBERT tokens averaged over all frames; the PopT CLS token into the Neuroprobe logistic regression.
  • Variants on sentence onset, subject 3 session 1 and subject 2 session 0 (leaderboard 0.891 and 0.844): this recipe 0.857 and 0.801; centre 10 frames 0.820; clean contacts only 0.858 and 0.791; every PopT token 0.840 and 0.754.
recipe popt_perwindow_stft · braindecode/popt-pretrained @ 50b02d6f
Fig. 2E. PopT.
New · BrainBERT and PopT, global z-scoring

Session-wide z-scoring of the spectrogram

Two further entries, by the PopT authors, replace the per-window z-scoring of the spectrogram by a global one, but their code is not public. We implemented the most direct reading: each contact and frequency bin is z-scored with the mean and standard deviation of its whole session, and the windows are then cut from this series, as for MAPA. The first cells are running; their targets, for sentence onset across sessions, are 0.934 and 0.931 for BrainBERT, and 0.977 and 0.937 for PopT.

In the text

“This model had global z-scoring as opposed to per-window z-scoring of STFT inputs.”
Leaderboard entries “BrainBERT (frozen, global z-scoring)” and “Population Transformer (global z-scoring)”, G. Chau, 2026

In NeuralBench

  • neuroprobe_session_stft: the same rereferenced copy and STFT as the per-window recipes, over the whole session.
  • Each contact and bin z-scored with its session mean and standard deviation; no labels are used.
  • The extractor reads this series at 40.96 frames per second; each 1 s window keeps 34 frames, as a per-window STFT.
  • The fitting scope (session, subject or training split) is our choice, not a recovered procedure.
recipes brainbert_frozen_global_z, popt_laplacian_stft_global_z
BrainBERT · sub 3, session 10.934per-window entry 0.937 · running
BrainBERT · sub 2, session 00.931per-window entry 0.770 · running
PopT · sub 3, session 10.977per-window entry 0.891 · running
PopT · sub 2, session 00.937per-window entry 0.844 · running
Brant · #1100 · Zhang et al., 2023

Brant has no public benchmark

Brant reports downstream results only on private clinical data, so no public score can be recomputed. We therefore checked the weights and the forward pass instead: all 210 of 210 shared tensors are identical to the official release, and pooled features match the official model to 6.5e-5.

CheckResult
Weights against Daoze/Brant @ 7e6f9156210 of 210 shared tensors identical
Band-power front end against upstreammax difference 4e-16 (float64), 2e-7 (float32)
Full forward pass, same inputpooled features within 6.5e-5; token cosine 1.000000
Sampling rate given as 250 instead of 256 Hzfeatures differ by up to 0.19; the PR documents it
What remains

Open items before the cards are final

Every group

The numbers behind Fig. 1

Public values: Neuroprobe leaderboard JSON files (per test case, fold mean across sessions and subjects, per fold within session); BaRISTA paper Table 1, appendix per-subject table, Appendix K. Our values: the newest finished NeuralBench run per split, task, test case, fold and seed, final recipes only.

Code: NeuralBench branch feat/ieeg-sentence-onset; cards and extraction in pr-evidence-20260924/ieeg_card.py, report_data.py. Prose checked with the Jean-Rémi King author pack gate.