BrainEncoding26 Challenge 1 – Models and Pipeline Explained

A full walkthrough of every notebook in experiments/challenge1 of meg-encoding-course-tbd: what each model is, why it was chosen, and exactly what each pipeline step does. Figures referenced below live in BrainEncoding26 – All Figures.

Part 1 — The Shared Pipeline

Every notebook (except comparison_subject60.ipynb, which reimplements it — see below) is built from the same five building blocks in tbdencoder. Understanding these once explains 90% of every notebook’s code.

1. Fixation cropping — `extract_dva_crop` / `Cropper`

The raw input to every model is not the full scene image but a small crop centered on where Subject 60 (or subjects 1–5) was actually fixating at that moment. extract_dva_crop converts a fixation point (gx, gy, in screen pixels) into a crop window sized in degrees of visual angle (DVA) — accounting for screen size, viewing distance, and how much of the screen was actually used during recording (SCREEN_USAGE). A ±3° DVA window is used throughout (x_off_dva=3, y_off_dva=3), converted to pixels via tan(deg2rad(offset)) × viewing_distance. If the fixation point itself is off-screen, the crop is entirely gray and flagged valid=False; if only part of the crop window falls off-screen, the valid portion is kept and the rest padded gray. Cropper is a thin wrapper that loads the scene image and calls this function per row.

Why this matters biologically: this is what makes the whole setup a fixation-locked encoding model, not a whole-image one — the model only ever sees what was foveated, matching what actually drove the retina at that moment.

2. `DataProcessor` — the dataset wrapper

A torch.utils.data.Dataset subclass that reads a metadata CSV (columns like sceneID, mean_gx, mean_gy, subject), and for each row calls the Cropper to produce (crop, valid), then applies whatever torchvision.transforms pipeline was configured (set_transform). filter_by_mask / reset_metadata_index let you drop invalid rows after a first pass and keep the metadata table in sync with the feature arrays.

3. `extract_features` — running the vision model, with a pooling choice

Runs a batched forward pass through whatever feature-extractor module is handed to it (built via torchvision.models.feature_extraction.create_feature_extractor, which taps intermediate layers of a pretrained network without needing to subclass it). The key parameter across every notebook is spatial_pooling:

"mean" — global average pool over the feature map’s height/width. Used for the DVD/ResNet notebook. Produces a single “what’s in this image overall” vector — semantic, location-agnostic.
"center" (with center_size=3) — instead of averaging the whole feature map, take only the 3×3 patch at the exact center of the map and flatten it. Because the input crop is already centered on the fixation point, the center of a convolutional feature map corresponds to what the eye was looking at — i.e. this is a deliberately retinotopic, V1-like pooling choice. Used for every conv-layer feature in every other notebook.
FC (fully-connected) layer outputs are already flat 2D vectors (ndim==2, e.g. AlexNet’s classifier.2) — pooling is skipped automatically for these; they pass through unchanged as 4096-dim semantic vectors.

This single spatial_pooling switch is the mechanism behind almost every “V1 vs. semantic cortex” comparison in this report: center-pooled early-conv features → posterior/occipital topographies; unpooled FC or CLS-token features → frontal/temporal topographies.

4. MEG target preprocessing — outlier removal + robust scaling

Before any encoder sees the MEG data:

detect_meg_outliers flags any fixation whose peak absolute amplitude across all 204 sensors sits above the 99.9th percentile — a crude but cheap way to drop a small number of extreme artifact trials before fitting.
RobustScaler (scikit-learn) rescales each sensor by its median and interquartile range instead of mean/std — much less sensitive to the remaining heavy-tailed artifacts than a standard scaler. See BrainEncoding26 – All Figures and the subject-variance diagnostic for why this matters: sub-1 and sub-5’s raw RMS is dominated by exactly this kind of outlier tail.
An important, explicitly-noted design choice: MEG targets are raw, not gaze-residualized. An earlier version tried regressing out gaze position first, but a diagnostic showed this pulled out real visual signal along with eye-movement (EOG) artifact — because the crop itself is gaze-centered, image content and gaze position are correlated by construction, so residualizing on gaze also throws away signal. Training on raw MEG gave cleaner, more interpretable posterior/visual-cortex topographies.

5. `FeatureEncoder` — the actual regression model

Every notebook fits the same kind of model on top of whatever features it extracted: a scikit-learn Pipeline of PCA → RidgeCV, wrapped in FeatureEncoder.k_fold_fit():

PCA reduces the (often 1000s-dimensional) feature vector to a manageable number of components before ridge regression — both to fight overfitting and to keep the ridge solve tractable.
RidgeCV(alpha_per_target=True) fits an L2-regularized linear regression from features → all 204 MEG sensors at once, with scikit-learn’s built-in cross-validation picking a separate regularization strength per sensor (sensors have very different signal-to-noise ratios, so one shared alpha would be a poor compromise).
k_fold_fit wraps this in manual k-fold cross-validation (typically 3 folds): fits one pipeline clone per fold, scores each on its held-out fold via Pearson correlation per sensor, reports mean/worst/best-sensor r, and finally refits on the full dataset for actual prediction. predict() then averages the predictions of all fold-models (a cheap ensembling side-effect).
Memory note repeated across notebooks: an early attempt at n_jobs=-1 (parallel folds) with alpha_per_target=True on high-dimensional FC features crashed the laptop (each forked worker holds a full copy of the ~180k-row feature matrix). Every subsequent notebook explicitly caps PCA components (≤256) and runs folds sequentially (n_jobs=1) with capped BLAS threads instead.

6. Evaluation & export

Pearson r is computed per sensor between predicted and true MEG (pearson() — mean-centered dot product over norm), reported as mean r across 204 sensors and best single sensor r (the challenge’s primary leaderboard metric). Final predictions are packaged as a (7750, 204) (or (7896, 204) for the eval split) float32 array, saved as predictions.npy, zipped to predictions.zip. Invalid rows (fixation off-screen, ~137–283 of ~7750–7896) are padded with the per-sensor mean of the valid predictions rather than dropped, since the export format requires every row present.

Part 2 — The Models, Notebook by Notebook

AlexNet (`alexnet_subject60.ipynb`)

Model: AlexNet, ImageNet-pretrained, via torchvision.models.alexnet. Five tappable layers: features.2/5/7 (Conv1/2/3 + ReLU + MaxPool, in ascending depth) and classifier.2/5 (FC6/FC7 + ReLU, both 4096-dim).

Pipeline: single forward pass extracts all five layers at once (create_feature_extractor). Conv layers get center-3×3 pooling (V1-like); FC layers bypass pooling (semantic). A layer sweep fits a separate PCA+Ridge encoder per single layer, then a combo condition concatenates features.2 + features.5 + classifier.2 + classifier.5.

Extra: a Reduced-Rank Ridge Regression (RRR) variant is also tried on the combo features — instead of letting each of the 204 sensors have an independent coefficient vector (plain ridge), RRR constrains the coefficient matrix to low rank, forcing sensors to share a small number of latent components. The custom ReducedRankRidgeCV class sweeps (alpha, rank) pairs via internal 2-fold CV (fitting ridge once per alpha, then testing multiple rank-truncations of the resulting solution almost for free via SVD).

Result: best single layer classifier.2 (FC6), mean r ≈ 0.094; combo ≈ 0.094 as well; RRR combo ≈ 0.093 — essentially no gain from either the multi-layer combo or the rank constraint over FC6 alone. See BrainEncoding26 – All Figures → AlexNet — layer sweep / AlexNet — topomaps.

CLIP ViT-B/32 (`clip_subject60.ipynb`)

Model: OpenAI’s CLIP, ViT-B/32 image encoder, via open_clip. Trained contrastively on 400M image-text pairs — known from brain-encoding literature to align unusually well with human visual cortex.

The interesting engineering bit — getting spatial features out of a ViT. CLIP’s normal encode_image() only returns the CLS token: one global, semantic summary vector per image, produced by attention-pooling over all patches. To get something spatially localized (for a V1-style comparison), CLIPExtractor registers a forward hook on the last transformer block, capturing its full patch-token sequence. ViT-B/32 divides the 224×224 input into a 7×7 grid of 32-pixel patches; the hook reshapes the token sequence back into that 7×7 grid and slices out just the center 1×1 patch (center_n=1) — the single patch covering the fixation point, 768-dim.

Two conditions compared: cls (global/semantic) vs. center_patches (local/spatial).

Result: CLS token wins clearly (mean r ≈ 0.085, best sensor 0.261 — the single best best-sensor score of any condition in the whole project) over center patches (mean r ≈ 0.063). Confirms the semantic-features-win pattern seen with AlexNet FC6, and CLIP CLS narrowly edges out AlexNet FC6 on best-sensor r while AlexNet FC6 wins on mean r. See BrainEncoding26 – All Figures → CLIP bar / CLIP topomap.

Gabor + AlexNet (`gabor_alexnet_subject60.ipynb`)

Models compared, in five conditions: (1) a hand-built Gabor filter bank — the classical, neuroscience-textbook model of V1 simple cells (orientation- and frequency-tuned edge detectors) — 4 frequencies × 4 orientations = 16 filters; (2) AlexNet conv1+2 (data-driven V1/V2 analogue); (3) Gabor + AlexNet conv combined; (4) AlexNet FC6+7 (semantic); (5) the full combo of all of the above.

Engineering detail — FastGaborExtractor. The original tbdencoder.models.gabor_filter.GaborFilterBank runs each of the 16 filters as its own separate nn.Conv2d in a Python loop — correct but slow. FastGaborExtractor stacks all 16 real-part and 16 imaginary-part kernels (padded to a common size) into two single Conv2d layers with 16 output channels each, so one forward call computes all 16 filter responses at once (real² + imag² → magnitude) — about 10× faster, since it’s one cuDNN/MPS kernel launch instead of 16.

Caching: since Gabor+AlexNet feature extraction over 180k training rows takes ~110 minutes, both the raw extracted features and the per-condition fitted r-scores are cached to disk (cache/gabor_alexnet_results/), so re-running the notebook after the first pass is near-instant.

A concrete bug hit and fixed in this notebook: exporting predictions initially failed with expected shape (7896, 204), got (7750, 204) — the evaluation pipeline had filtered out invalid rows before predicting, but the export format needs all rows (invalid ones included, padded later). The fix: reload the unfiltered cached dev features (all 7896 rows) specifically for the export step, keeping the filtered version only for the r-score evaluation.

Result: Gabor only mean r ≈ 0.017 (weakest condition project-wide), AlexNet conv1+2 ≈ 0.040, Gabor+conv1+2 ≈ 0.040 (no synergy — the two V1-style feature sets are apparently redundant, not complementary), AlexNet FC6+7 ≈ 0.094 (best), full combo ≈ 0.082 (worse than FC6+7 alone — adding low-level features under the same PCA/ridge budget diluted the semantic signal rather than adding to it). See BrainEncoding26 – All Figures → Gabor + AlexNet bar / topomaps.

Anchor + Attractor Ensemble (`anchor_attractor_subject60.ipynb`)

Theoretical framing: inspired by attractor-network theory — a stimulus first drives a fast, spatially-specific “anchor” response (feedforward, V1/V2-like), before the system settles into a stable, category-level “attractor” state (semantic, IT-like). At 110 ms post-fixation, both stages plausibly coexist in the MEG signal.

Feature mapping: anchor = AlexNet features.2+5 (center-pooled, 2304-dim); attractor = AlexNet classifier.2/FC6 (flat, 4096-dim, unpooled).

Four conditions, testing this framing directly: (1) anchor only, (2) attractor only, (3) naively concatenated features → one encoder, (4) additive ensemble — train two separate encoders (one per stage) and simply sum their predictions at the sensor level. The logic for (4): letting each stage’s encoder specialize without the other’s features interfering during PCA/ridge fitting, then combining only at the final prediction — a much looser coupling than concatenation.

Result: anchor only mean r ≈ 0.040, attractor only ≈ 0.095 (again, the semantic FC6 feature wins outright), concatenated ≈ 0.084, additive ensemble ≈ 0.088 — the ensemble modestly beats naive concatenation but still doesn’t beat attractor-alone. The theoretically-motivated combination doesn’t pay off here; a single strong semantic feature set is hard to beat by combining with a much weaker one, however cleverly. See BrainEncoding26 – All Figures → Anchor+Attractor bar / topomaps.

DVD — Developmental Visual Diet (`dvd_subject60.ipynb`)

Idea: test whether simulating immature (infantile) vision before feature extraction changes prediction quality, using the external DVD library, which progressively degrades acuity, contrast sensitivity, and color perception as a function of a months parameter (0 = newborn-like, 240 = adult/no change).

Model: ResNet-18 (ImageNet-pretrained), tapping layer1.0.conv2 (early) and layer4.0.conv2 (late), global-average-pooled (spatial_pooling default "mean" — unlike every other notebook, this one uses semantic/whole-image pooling throughout, not center-pooling).

Design: sweep DVD_MONTHS = [0, 60, 120, 180, 240], rebuilding the transform pipeline and re-extracting features fresh for each age (no caching in this notebook), fitting a fresh encoder each time.

Status: the notebook is fully written (including a preview cell that visualizes what each developmental age actually looks like) but has no recorded outputs — it has not actually been run yet. Its result (months=240, i.e. adult vision applied as a fixed baseline, not swept) does appear in the master comparison notebook: mean r ≈ 0.085, right in the middle of the pack — but the actual DVD-age sweep (does infant vision predict Subject 60 any better or worse?) remains an open, unrun experiment.

Model Comparison (`comparison_subject60.ipynb`)

This is the master notebook that puts all eight conditions on one plot, using the same Subject 60 ground truth throughout. As covered above, it’s self-contained — it doesn’t import tbdencoder at all, because the installed package version had drifted out of sync with what every other notebook assumes (FeatureEncoder.k_fold_fit() and extract_features(..., spatial_pooling=..., center_size=...) didn’t exist in that version). Instead it reimplements Cropper, extract_dva_crop, DataProcessor, detect_meg_outliers, extract_features, FeatureEncoder, and GaborFilterBank from scratch in its first cell, using only external libraries (torch, sklearn, joblib, skimage, PIL) — functionally identical to the package versions, just decoupled from them.

Beyond the headline bar chart and topomap grid (already the two flagship figures in BrainEncoding26 – All Figures), this notebook runs four additional analyses worth knowing about:

Sorted-sensor curves — each model’s 204 per-sensor r-values sorted best-to-worst and plotted as a curve. Distinguishes “wins on a few sensors, flat elsewhere” from “broad, even coverage” — two very different ways to reach the same mean r.
Sensor coverage above threshold — simple count of how many of the 204 sensors exceed r > 0.05 per model, a practical “how much of the array does this model actually explain” metric.
Cross-model similarity matrix — pairwise correlation between each pair of models’ 204-dim r-patterns. Low correlation between two otherwise-strong models is exactly the signal that motivated the Anchor+Attractor ensemble (two models winning on genuinely different sensors are worth combining; two models winning on the same sensors are redundant).
ROI (anterior-posterior) breakdown — sensors are split into four y-coordinate quartiles (posterior/occipital → anterior/frontal), and mean r per model is computed per quartile, testing directly whether early/Gabor features really do win posterior sensors and semantic features (FC6, CLIP CLS) really do win anterior ones, rather than just asserting it from the topomap plots by eye.
Statistical comparison against the top model — paired Wilcoxon signed-rank test (per-sensor r, paired between the best model and every other model), Benjamini-Hochberg FDR-corrected across the 7 comparisons. Explicitly caveated in the notebook: since neighboring MEG sensors are spatially correlated (not independent samples), this is a useful relative signal, not a formally valid p-value — treated as suggestive, not confirmatory.
Difference topomaps — direct sensor-by-sensor subtraction between (a) the #1 and #2 ranked models, and (b) the best single model vs. the Anchor+Attractor ensemble, to see where on the head one model’s edge over another actually comes from.

Subject Variance Diagnostic (`subject_variance.ipynb`)

Covered in full in a separate note — see BrainEncoding26 – All Figures for the figures and the earlier project report for the write-up. Short version: sub-1 and sub-5 look alarming on raw RMS (50–114× sensor range) but are actually the most representative subjects once compared via robust statistics (median MAD within 1.3× across all five) — the issue is a long tail of artifact-contaminated fixations, not a different underlying neural signal. Feeds directly into the detect_meg_outliers + RobustScaler step used by every encoding notebook above.

Part 3 — Leaderboard Recap

Condition	Model	Pooling	Best sensor r	Mean r
CLIP CLS token	ViT-B/32, global	—	0.261	0.085
AlexNet FC6	classifier.2	center 3×3	0.257	0.095
Anchor + Attractor ensemble	conv1+2 + FC6, summed	center 3×3	0.240	0.088
DVD ResNet-18 (months=240)	layer1/4 conv	global avg	0.233	0.085
CLIP center patch	ViT-B/32, local	—	0.194	0.062
AlexNet conv1+2	features.2+5	center 3×3	0.122	0.040
Gabor + AlexNet conv	16-filter bank + conv1+2	center 3×3	0.122	0.040
Gabor only	16-filter bank	center 3×3	0.081	0.017

The pattern holding across every single notebook: semantic, late-layer features (FC6, CLIP CLS) consistently beat low-level, retinotopic features (Gabor, early conv) by roughly 2–5×, even at a timepoint (110 ms) early enough that classical visual-cortex physiology would predict a strong V1/V2 signature.

Brain Online

Explorer

BrainEncoding26 Challenge 1 – Models and Pipeline Explained