ArXiv: 2305.18274

🎯 Pitch

MindEye can identify the exact image someone saw among billions of candidates with over 90% accuracy using only brain scans. This is achieved through a surprising discovery that retrieval and reconstruction demand fundamentally different brain-embedding geometries, forcing a split architecture where a contrastive pathway pinpoints the original image while a separate diffusion prior generates a high-fidelity replica.

1. Executive Summary

This paper introduces MindEye, a novel fMRI-to-image approach that maps brain activity to visual representations using two parallel, specialized submodules — a contrastive-learning-based retrieval pathway and a diffusion-prior-based reconstruction pathway — that together achieve state-of-the-art performance on the Natural Scenes Dataset (NSD). The retrieval submodule produces "disjointed CLIP fMRI" embeddings via a bidirectional mixup contrastive loss (BiMixCo), enabling top-1 identification of the exact original image among 982 test candidates with 93.2% accuracy (chance = 0.1%), while the diffusion prior aligns these disjointed embeddings for input to pretrained generative models like Versatile Diffusion. A separate low-level pipeline maps voxels directly to Stable Diffusion's VAE latent space, preserving fine-grained perceptual features (color, texture, spatial position) that the semantic pipeline misses, with the two streams combined through img2img to yield state-of-the-art reconstructions across both low- and high-level image metrics — achieving 94.7% two-way identification on AlexNet(2) and 94.1% on CLIP, outperforming the previous best by substantial margins. The design reveals a fundamental tradeoff between retrieval and reconstruction objectives that cannot be satisfied by a single embedding space, establishing that specialized submodules with decoupled losses — an MLP projector for contrastive retrieval and a diffusion prior for MSE-based reconstruction — are necessary to achieve top performance on both tasks simultaneously.

2. Context and Motivation

The Core Problem: Extracting Rich Visual Content from Non-Invasive Brain Recordings

The fundamental challenge this paper tackles is decoding the specific visual content of a person's subjective experience from non-invasive measurements of brain activity. Given a snapshot of fMRI data collected while someone views a natural scene, can we reconstruct an image that captures both the semantic content (what objects and entities are present) and the perceptual details (colors, textures, spatial layout) of what that person saw? This is not merely a classification problem — identifying that a person looked at "a zebra" — but a fine-grained generation problem that requires recovering exemplar-specific information: which particular zebra image, from which angle, with which background, in which color palette.

This capability sits at the intersection of two fields with distinct priorities:

For neuroscience, the ability to decode detailed perceptual content from brain activity tests our fundamental understanding of how visual information is represented in the human cortex. If we can accurately reconstruct what someone sees from their fMRI signals, it demonstrates that the distributed patterns of brain activity measured by fMRI contain not just coarse category information (faces vs. houses vs. scenes) but rich, high-dimensional representations that preserve fine-grained visual details. This provides empirical constraints on computational theories of visual processing and opens the door to using reconstruction quality as an objective function for evaluating models of visual cortex.

For brain-computer interfaces and clinical applications, non-invasive visual decoding could enable communication with patients who cannot speak or move (e.g., locked-in syndrome, disorders of consciousness) by allowing them to convey visual information through mental imagery. As the authors note in Section 5, "reconstructions are expected to be systematically distorted due to mental state, neurological conditions, etc. This could potentially enable novel clinical diagnosis and assessment approaches." For example, patients with major depressive disorder might produce reconstructions where emotionally negative aspects of images are more salient, providing a quantitative biomarker for attentional bias.

The specific instantiation of this problem that MindEye addresses is mapping blood-oxygen-level-dependent (BOLD) fMRI signals — measured across roughly 13,000–16,000 cortical voxels — to the embedding spaces of pretrained vision models, from which images can be generated using existing generative models. The key technical difficulty is that fMRI data is noisy, high-dimensional (each voxel measures a mixture of neural activity from a 1.8mm³ cube of tissue), surprisingly low-sample (each participant provides only ~25,000 training examples after ~40 hours in the scanner), and the mapping from voxel activation patterns to specific image pixels is highly nonlinear and participant-specific due to individual differences in cortical anatomy and functional organization.

Why Prior Approaches Fall Short

Prior work on fMRI-to-image decoding has progressed through several generations of methods, each with characteristic limitations that MindEye aims to overcome. Understanding these limitations requires some historical context about how the field has evolved.

Era 1: Linear Decoding of Coarse Categories (2000s–early 2010s)

The earliest fMRI decoding studies (cited in Section 4) demonstrated that linear classifiers could extract surprisingly specific information from voxel activation patterns: spatial position (retinotopy), edge orientation, and coarse image categories such as faces versus houses versus objects. These results were foundational — they established that fMRI signals contain decodable visual information — but were limited in two critical ways. First, they operated at the level of discrete category labels rather than continuous visual features, meaning they could tell you that someone was looking at a face, but not which face. Second, they used linear decoders (e.g., support vector machines) that are fundamentally limited in the complexity of the mapping they can learn, essentially assuming that visual information is encoded in a linearly separable manner across voxels.

Era 2: Generative Models for Reconstruction (mid-2010s–early 202s)

The introduction of generative adversarial networks (GANs) and variational autoencoders (VAEs) enabled a qualitative leap: rather than decoding to discrete labels, researchers could now decode to the continuous latent spaces of generative models and then produce actual images. Seminal work by Shen et al. (2019) [52] reconstructed natural images by iteratively optimizing pixel values to match both a decoded feature representation and a natural image prior. Ozcelik et al. (2022) [5] mapped fMRI to the latent space of Instance-Conditioned GANs, achieving higher-quality reconstructions for natural scenes.

However, these GAN-based approaches had fundamental constraints. GAN latent spaces are typically low-dimensional and semantically entangled — interpolating in the latent space doesn't always produce semantically meaningful image variations, and the diversity of images a GAN can generate is limited by its training data. The resulting reconstructions often captured coarse gist (e.g., "there was a large animal in a grassy field") but lacked fine-grained fidelity to the specific viewed image.

Era 3: CLIP-Based Mapping with Simple Regressors (2022–early 2023)

The release of CLIP [7] — a model trained on 400 million image-text pairs that produces semantically rich, high-dimensional embeddings — and the Natural Scenes Dataset (NSD) [26] — the first large-scale 7-Tesla fMRI dataset with 8–10 unique images per participant — catalyzed the current generation of methods. The key insight, which MindEye builds on, is that CLIP image embeddings serve as an excellent intermediate target: they capture both semantic and fine-grained visual information (CLIP can distinguish between different zebra images, not just zebra vs. non-zebra), and they can be fed as conditioning inputs to powerful generative models like Stable Diffusion.

Two landmark papers established the baseline that MindEye directly improves upon:

Takagi and Nishimoto (2022) [3] used ridge regression — a linear method with L2 regularization — to map flattened fMRI voxels to both Stable Diffusion's latent space (for low-level visual features) and CLIP text embeddings (for high-level semantic features). Different voxel selections were used for different model components (e.g., early visual cortex for low-level features, higher visual areas for semantic features). This approach demonstrated that CLIP-space decoding was feasible, but ridge regression is fundamentally a linear mapping technique. It cannot capture nonlinear relationships between voxel activation patterns and visual features — for example, the same object category might elicit different voxel patterns depending on its position in the visual field, a nonlinear interaction that a linear model would average over.

Lin et al. (2022) [11] used a small convolutional network to map voxels to CLIP space, then fine-tuned the Lafite GAN to generate images conditioned on these CLIP embeddings. Their approach achieved 11% image retrieval accuracy (top-1 out of 300 candidates, where chance is ~0.3%). This was an improvement over pure linear methods, but several limitations are apparent: the convolutional architecture was relatively small, they used the InfoNCE contrastive loss (which, as MindEye's ablations in Table 3 show, is one-sided and poorly optimizes image retrieval), and they applied both contrastive loss and MSE loss to the same embedding space without separation — a design choice that MindEye demonstrates creates a destructive tradeoff between retrieval and reconstruction objectives (Table 4, "Only MLP Backbone").

Ozcelik and VanRullen (2023) [4], the immediate predecessor and primary baseline for MindEye, used the same two-pipeline approach (low-level + high-level) that MindEye adopts. Their high-level pipeline mapped voxels to CLIP space via ridge regression (257 separate ridge regression models, each with approximately 12 million parameters, totaling ~3 billion parameters for the mapping). Their low-level pipeline mapped voxels to the latent space of a Very Deep VAE (VDVAE). Outputs from both pipelines were fed through Versatile Diffusion for final reconstruction.

Brain-Diffuser represented the state of the art at the time of MindEye's publication, achieving 87.2% two-way identification on InceptionV3 and 91.5% on CLIP, with 21.1% image retrieval accuracy. Where does it fall short? Four specific weaknesses motivate MindEye's design:

  1. Ridge regression as the mapping function. While ridge regression has the advantage of being convex (guaranteed to find a global optimum) and computationally efficient to fit, it cannot capture the nonlinear interactions between voxels that are necessary for fine-grained decoding. Visual cortex exhibits complex nonlinear computations — e.g., gain control, surround suppression, and contextual modulation — that linear models fundamentally cannot represent. The result is a ceiling on the fidelity of the decoded representation, manifesting as reconstructions that capture broad scene gist but blur or hallucinate specific details.

  2. No contrastive learning. Brain-Diffuser did not use any contrastive objective. The mapping to CLIP space was trained purely with MSE loss. This means the model optimizes for the magnitude and direction of the embedding vector simultaneously. As research on multimodal contrastive learning has shown (Liang et al., 2022 [24]), encoding different modalities into a shared space via pure MSE creates a "modality gap" — the embeddings from different modalities occupy disjoint regions of the embedding space. This gap fundamentally limits retrieval performance because nearest-neighbor search assumes that matched pairs are closer in embedding space than unmatched pairs. Brain-Diffuser's 21.1% image retrieval accuracy reflects this: the model's brain embeddings and image embeddings are not well-aligned.

  3. No separation between retrieval and reconstruction objectives. Brain-Diffuser uses a single set of CLIP-space mappings for both retrieval (if one wanted to do it) and reconstruction. As MindEye's central architectural insight demonstrates, there is a fundamental tradeoff: contrastive learning optimizes for angular similarity (directional alignment between brain and image embeddings), producing "disjointed" embeddings that are directionally correct but of different magnitude than true CLIP image embeddings, while generative models expect embeddings with the same magnitude statistics as the CLIP image embeddings they were trained on. A single embedding space cannot satisfy both objectives well. Brain-Diffuser sidesteps this tradeoff by not doing contrastive learning at all — achieving mediocre reconstruction and poor retrieval — rather than solving it.

  4. VDVAE for low-level features. Brain-Diffuser's low-level pipeline used a VDVAE rather than Stable Diffusion's VAE. While both serve a similar purpose (encoding images to a compressed latent space that preserves perceptual details), the choice of autoencoder affects compatibility with the diffusion model used for final generation. MindEye's use of Stable Diffusion's VAE for the low-level pipeline enables the img2img approach, where the noised VAE encoding from the low-level pipeline serves as the initialization for diffusion.

Why the Problem Remained Unsolved

Beyond specific methodological limitations, a deeper reason the problem remained unsolved is the tension between high-level semantic fidelity and low-level perceptual fidelity. CLIP embeddings inherently prioritize semantic content because CLIP was trained to match images with text captions — captions describe what is in the image (objects, actions, relationships) but not how it looks (exact colors, textures, spatial positions). Mapping fMRI to CLIP space thus produces reconstructions that get the gist right (e.g., a zebra in a field) but may swap colors, alter textures, or shift object positions. Meanwhile, mapping to VAE latent space preserves low-level features but loses semantics because the VAE is not trained on a semantic objective — a VAE encoding of two different zebra images will be almost as different as a VAE encoding of a zebra and a horse. This tension cannot be resolved by simply using one latent space or the other; a hybrid approach is necessary, but naive hybrids (e.g., simply concatenating CLIP and VAE latents) risk the semantics being corrupted by the VAE pathway or the textures being overwritten by the CLIP pathway.

How MindEye Positions Itself

MindEye's positioning is best understood through the specific design choices that directly address each of Brain-Diffuser's limitations, plus one additional innovation that emerges from the neuroscience context:

Against linear regression: Deep nonlinear mapping. MindEye replaces ridge regression with a residual MLP backbone containing 940 million parameters — roughly 7× the number of trainable parameters in Brain-Diffuser's high-level pipeline when counting the 257 separate ridge regressors. This is a deliberate bet on the hypothesis that (a) the fMRI-to-CLIP mapping is highly nonlinear, (b) large overparameterized models can learn these nonlinearities without overfitting despite the limited sample size (~25K training samples), and (c) the model benefits from jointly learning across all 257 CLIP embedding positions rather than treating them independently. Table 2 provides the empirical evidence: going from zero residual blocks (873M parameters) to 4 residual blocks (940M parameters) improves image retrieval from 88.0% to 89.6%, and a version that maps only to the CLIP CLS token (135M parameters) drops to 61.1%. The fact that a 940M-parameter model does not overfit on 25K samples is non-obvious — it suggests that the fMRI voxels contain a high-dimensional, structured signal that provides effective degrees of freedom far beyond the naive sample count.

For retrieval: Contrastive learning with specialized projector. MindEye introduces bidirectional contrastive training (CLIP loss rather than InfoNCE, making it bidirectional and improving both image and brain retrieval as shown in Table 3) with two critical innovations. First, the BiMixCo augmentation (Equation 2) extends MixCo — originally developed by Kim et al. [20] for semi-supervised classification with InfoNCE — to the bidirectional CLIP loss setting, and the authors show it's particularly effective in low-data regimes where augmentation is most needed. Second, the training schedule switches from hard BiMixCo to a soft contrastive loss (inspired by knowledge distillation, Equation 3) one-third of the way through training, following recent findings that mixup augmentation degrades performance if continued to convergence [21, 22]. This schedule is designed specifically to balance retrieval and reconstruction performance in a single model — Table 4 shows that pure BiMixCo gives the best retrieval (97.4% image retrieval) but slightly worse reconstruction, while the hybrid schedule gives 97.2% retrieval with better reconstruction metrics.

For reconstruction: Diffusion prior as alignment mechanism. The diffusion prior (borrowed conceptually from DALL-E 2 [8] but trained from scratch on this mapping) serves a specific, well-motivated role: it learns the distribution of true CLIP image embeddings conditioned on the disjointed CLIP fMRI embeddings output by the contrastive projector. This converts directionally-correct-but-magnitude-incorrect embeddings into embeddings that fall within the distribution expected by pretrained generative models. The UMAP visualization in Figure 9 empirically demonstrates this: the disjointed embeddings from the MLP projector occupy a different region of CLIP space than real CLIP image embeddings, while the prior outputs are much better aligned (lower Euclidean distance). Critically, the prior is trained end-to-end with the backbone and projector, meaning the backbone learns to produce representations that are useful both for contrastive alignment and as conditioning inputs to the diffusion prior. The authors explicitly test the alternative (Table 4: "Only MLP Backbone") — applying both contrastive and MSE losses to the same embedding space — and find it dramatically fails at retrieval (13.3% vs. 96%+) while underperforming at reconstruction, confirming that the decoupling via specialized submodules is necessary, not just convenient.

For perceptual preservation: Direct VAE-space mapping with img2img. MindEye's low-level pipeline maps directly to the VAE latent space used by Stable Diffusion, producing blurry reconstructions that score well on pixel-level and low-level feature metrics but poorly on semantics (Table 1, "MindEye (Low-Level)": PixCorr = 0.360, SSIM = 0.479, but CLIP two-way identification = 59.2%). The img2img approach combines the two pipelines by starting the diffusion denoising process from the noised VAE encoding rather than pure noise, with the strength parameter (0.15 in the final model) controlling how much influence the low-level initialization has. Table 5 shows the ablation: img2img strength of 1.0 (pure low-level) gives the best PixCorr/SSIM but poor high-level metrics; 0.0 (pure high-level) gives the inverse; 0.15 hits a sweet spot that improves low-level metrics substantially over pure high-level while causing only minor degradation in high-level metrics. This is a principled solution to the semantic-perceptual tension: the semantic pipeline determines what is in the image, while the perceptual pipeline provides a structural scaffold that biases how it looks without overriding the semantic content.

The deeper conceptual contribution: Retrieval as validation of fine-grained representation. Beyond the technical architecture, MindEye makes a conceptual argument that retrieval performance is not merely a secondary metric but a direct test of whether the decoded brain representations contain exemplar-level visual information. The claim is that if you can pick the exact zebra image someone viewed out of a dozen zebra images based on their brain activity, then your brain embeddings contain fine-grained, image-specific information — not just category-level gist. This is validated dramatically in Figure 3, where MindEye achieves 93.2% top-1 accuracy in a pool of 982 images containing many confusable examples. This level of retrieval performance is unprecedented — the previous state-of-the-art was 21.1% (Ozcelik and VanRullen [4]) — and it changes the interpretation of what fMRI signals contain. It suggests that the distributed voxel activation patterns in visual cortex, when decoded through a sufficiently powerful nonlinear model, preserve information at the grain of individual natural images, not just semantic categories.

3. Technical Approach

3.1 Reader Orientation

MindEye is a neural decoding system that takes flattened fMRI voxel activation patterns — roughly 13,000 to 16,000 numbers representing blood oxygenation levels across the posterior cortex — and maps them through a deep multilayer perceptron (MLP) into the embedding space of a pretrained CLIP vision model, from which images can be reconstructed using off-the-shelf generative diffusion models or retrieved from large-scale databases. The system solves a fundamentally underconstrained mapping problem: how to go from a noisy, subject-specific, relatively low-sample brain measurement to the rich high-dimensional representation of a specific viewed photograph, overcoming the tension between (a) the directional alignment needed for accurate image retrieval via cosine similarity, and (b) the magnitude and distributional correctness needed for feeding into pretrained generative models that expect embeddings drawn from the same distribution as CLIP image embeddings.

3.2 Big-Picture Architecture (Diagram in Words)

MindEye consists of two parallel processing streams that share no parameters but both start from the same flattened fMRI voxel vector:

  1. High-Level (Semantic) Pipeline: A large residual MLP backbone (940M parameters) maps voxels to an intermediate space of size 257 × 768, matching the last hidden layer of CLIP ViT-L/14. This backbone feeds two specialized submodules in parallel:

    • MLP Projector: Produces "disjointed CLIP fMRI embeddings" — directionally aligned with true CLIP image embeddings but with different magnitude — optimized via bidirectional contrastive loss for retrieval tasks.
    • Diffusion Prior: A transformer-based diffusion model trained from scratch to take the backbone outputs and produce "aligned CLIP fMRI embeddings" that match the distribution of true CLIP image embeddings, suitable as conditioning inputs to pretrained image generation models like Versatile Diffusion.

  2. Low-Level (Perceptual) Pipeline: A separate residual MLP backbone maps the same voxels to a (16, 16, 64)-dimensional latent space, which is then upsampled by a CNN decoder to (64, 64, 4) — the exact latent space of Stable Diffusion's variational autoencoder (VAE). Feeding these latents through the VAE decoder produces blurry reconstructions that lack semantic content but preserve pixel-level features. These serve as the initialization (via img2img) for the final diffusion-based reconstruction, biasing the output toward correct low-level appearance without overriding semantic content.

The two pipelines are combined at generation time: the aligned CLIP embeddings from the high-level pipeline condition the generative model, while the noised VAE encodings from the low-level pipeline replace the initial pure noise. The entire system is trained end-to-end with a composite loss that switches from hard bidirectional mixup contrastive loss (BiMixCo) to soft contrastive loss one-third of the way through training.

3.3 Roadmap for the Deep Dive

  • First, the formal structure of the MLP backbone and projector (Algorithm 1), since everything else depends on the quality of the voxel-to-CLIP mapping — this is where the 940M parameters live and where the critical architectural choices about depth, residual connections, and output dimensionality are made.
  • Second, the contrastive learning framework, including the standard CLIP loss, the novel BiMixCo extension that mixes voxel samples during training, the soft contrastive loss inspired by knowledge distillation, and the training schedule that switches between them — because this is the mechanism that enables state-of-the-art retrieval.
  • Third, the diffusion prior — why it's needed, how it's trained, what modifications were made from the DALL-E 2 implementation, and how it resolves the "modality gap" between contrastively-learned fMRI embeddings and the CLIP image embedding distribution expected by generative models.
  • Fourth, the low-level perceptual pipeline, including the separate MLP backbone, the CNN upsampler architecture, the choice of MAE loss over MSE, and the auxiliary contrastive loss using VICRegL features instead of CLIP.
  • Fifth, the img2img combination strategy, which blends the semantic pipeline (determining what is in the image) with the perceptual pipeline (biasing how it looks) by controlling the starting point of the diffusion denoising process.
  • Sixth, the training procedure end-to-end, including the composite loss function, the hyperparameters, the computational budget, and the dataset preprocessing — because the specific choices around multi-repetition averaging, layer normalization, dropout, and optimization schedule are critical to the system's success.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architectural design paper whose core idea is that retrieval and reconstruction objectives impose conflicting requirements on the embedding space learned from fMRI data, and that this conflict can be resolved by decoupling the two tasks through specialized submodules — a contrastive MLP projector for retrieval and a diffusion prior for reconstruction — trained jointly from a shared MLP backbone, supplemented by a separate low-level pipeline that injects perceptual fidelity via img2img initialization.

The MLP Backbone and Projector

The high-level pipeline begins with a residual multilayer perceptron that maps the subject-specific flattened fMRI voxel vector (dimensionality between 13,000 and 16,000, depending on the participant's "nsdgeneral" brain region mask) to an intermediate representation of size 257 × 768. These dimensions are chosen to match the last hidden layer output of CLIP ViT-L/14, which produces one 768-dimensional CLS (classification) token plus 256 patch tokens, each also 768-dimensional — 257 embeddings total. The choice to predict all 257 embeddings (rather than just the CLS token or the final pooled output) is deliberate and validated in Table 2: a model that predicts only the CLS token ("4 ResBlocks + Only CLS") has only 135M parameters and achieves image retrieval accuracy of 61.1% versus 89.6% for the full 257-token model. This indicates that the fMRI voxels contain fine-grained spatial information about image content that can be decoded into the distributed patch-level representation of CLIP, not just the global image representation captured by the CLS token.

The backbone architecture (Algorithm 1 in the paper) proceeds through the following sequence of operations on an input tensor x of shape (batch_size, in_dim) where in_dim is the number of voxels for the specific subject:

  1. Initial linear projection and normalization: A linear layer without bias maps from in_dim to a hidden dimension h = 4096, followed by LayerNorm, GELU activation, and aggressive dropout (0.5). This initial dropout rate of 0.5 is substantially higher than the 0.15 used in subsequent layers, suggesting that the raw voxel space contains significant noise or redundancy that the model benefits from aggressively regularizing at the input stage. The absence of bias in this layer is a deliberate choice — with bias, the LayerNorm would immediately subtract the learned bias term from the normalized activations, making the bias largely redundant with LayerNorm's learned shift parameter.

  2. Four residual blocks: Each block contains Linear(4096 → 4096, no bias) → LayerNorm → GELU → Dropout(0.15). The residual connection adds the block's input back to its output: x = block(x) + residual, then updates residual = x for the next block. This is a standard pre-activation residual design where the skip connection wraps the entire sub-block. The use of four blocks was determined empirically (Table 2): going from 2 blocks (907M parameters) to 4 blocks (940M parameters) improves image retrieval from 88.6% to 89.6%. Critically, the skip connections become more important at greater depth: the 4-block model without skip connections drops to 83.6% retrieval, whereas the 2-block model without skip connections barely degrades (88.1% vs. 88.6%), indicating that skip connections are essential for training deeper MLPs on this task.

  3. Final linear projection: A linear layer with bias maps from h = 4096 to out_dim = 257 × 768 = 197,376. The output is reshaped to (batch_size, 257, 768). This representation, which we'll call z_backbone, serves as the shared input to both the MLP projector and the diffusion prior. The use of bias here (unlike in the residual blocks) makes sense because this is the final output layer that needs to position the embeddings correctly in CLIP space, and the bias provides a learned offset that centers the distribution.

  4. MLP Projector: A separate network takes the backbone output z_backbone and produces z_proj — the "disjointed CLIP fMRI embeddings" used for retrieval. The projector architecture (applied independently to each of the 257 token positions) is: LayerNorm(768) → GELU → Linear(768 → 2048, no bias) → LayerNorm(2048) → GELU → Linear(2048 → 2048, no bias) → LayerNorm(2048) → GELU → Linear(2048 → 768, bias). This is essentially a small bottleneck MLP that projects each 768-dimensional token through a 2048-dimensional hidden layer twice before returning to 768 dimensions. The symmetric expansion-contraction design (768 → 2048 → 2048 → 768) is characteristic of projection heads in self-supervised learning (e.g., SimCLR, BYOL), where the expanded hidden representation provides capacity to learn useful transformations before collapsing back to the embedding dimension where the contrastive loss is applied. The absence of bias in the intermediate linear layers and presence of bias only in the final projection layer follows the same principle as the backbone: LayerNorm already provides shift parameters, making intermediate biases redundant.

What the backbone + projector combination computes: Given a batch of fMRI voxel vectors, the backbone produces a 257-token intermediate representation, and the projector maps each token to a 768-dimensional embedding that is optimized to have high cosine similarity with the corresponding CLIP image embedding for matching image-brain pairs and low cosine similarity for non-matching pairs. The projection head is not used for reconstruction — its outputs go only to the contrastive loss — which is the key architectural innovation that decouples the two objectives.

Why this architecture: The alternatives would be (a) using a single output head for both contrastive and MSE losses (Table 4, "Only MLP Backbone"), which forces the same embedding space to satisfy both directional similarity (for retrieval) and distributional matching (for reconstruction) — this fails dramatically at retrieval (13.3% vs. 96%+) — or (b) using no backbone at all and mapping directly with linear methods like ridge regression (the Brain-Diffuser approach), which cannot capture nonlinear voxel interactions needed for fine-grained decoding.

Contrastive Learning with BiMixCo and SoftCLIP

The contrastive learning framework trains the backbone + projector to produce embeddings that maximize cosine similarity between matched brain-image pairs while minimizing it for unmatched pairs. The paper uses three distinct loss formulations in a scheduled sequence, each addressing a specific limitation.

Standard CLIP Loss (Bidirectional InfoNCE)

The standard CLIP loss operates on a batch of N paired examples (x_i, y_i) where x_i is the fMRI sample and y_i is the corresponding viewed image. Let p_i = f(x_i) be the L2-normalized projector output for the i-th brain sample, and let t_i = CLIP_Image(y_i) be the L2-normalized CLIP image embedding for the i-th image. The bidirectional CLIP loss is:

LCLIP=−12N∑i=1Nlog⁡exp⁡(pi⋅ti/τ)∑j=1Nexp⁡(pi⋅tj/τ)−12N∑j=1Nlog⁡exp⁡(pj⋅tj/τ)∑i=1Nexp⁡(pi⋅tj/τ)\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^N \log \frac{\exp(p_i \cdot t_i / \tau)}{\sum_{j=1}^N \exp(p_i \cdot t_j / \tau)} - \frac{1}{2N} \sum_{j=1}^N \log \frac{\exp(p_j \cdot t_j / \tau)}{\sum_{i=1}^N \exp(p_i \cdot t_j / \tau)}

where N is the batch size and τ (tau) is a learned or fixed temperature parameter that controls the sharpness of the softmax distribution.

  • First term (image → brain): For each brain embedding p_i, compute its cosine similarity with all N image embeddings in the batch, apply softmax with temperature τ, and maximize the log-probability assigned to the correct pairing (p_i, t_i). This term alone is equivalent to the InfoNCE loss and optimizes for brain retrieval (given an image, finding the right brain sample).
  • Second term (brain → image): For each image embedding t_j, compute its cosine similarity with all N brain embeddings, apply softmax, and maximize the log-probability of the correct pairing (p_j, t_j). This term optimizes for image retrieval (given a brain sample, finding the right image).

What this loss computes: The average negative log-likelihood, under a softmax model, of correctly identifying the true paired sample among all N samples in the batch, in both directions. The dot product in the exponent is the cosine similarity because p_i and t_j are both L2-normalized to unit length. The temperature τ scales the logits: smaller τ makes the softmax sharper (higher confidence required), larger τ makes it softer (more tolerant of nearby-but-not-exact matches).

Why bidirectional: Table 3 shows that InfoNCE (the one-sided version) achieves only 23.7% image retrieval while CLIP loss (bidirectional) achieves 83.7%. The unidirectional loss only optimizes the brain→image mapping implicitly through the shared embedding space, whereas the bidirectional loss explicitly forces both retrieval directions to work. In the fMRI-to-image context, image retrieval is fundamentally harder than brain retrieval because there are typically more images than brain samples and images have higher intrinsic dimensionality — the bidirectional loss addresses this asymmetry by symmetrizing the objective.

BiMixCo: Bidirectional Mixup Contrastive Loss

BiMixCo extends the CLIP loss by applying mixup data augmentation to the fMRI voxels before computing the contrastive objective. For each fMRI sample x_i in the batch, a mixing partner x_{k_i} is randomly selected (where k_i is any index in [1, N]), and a mixing coefficient λ_i (lambda) is sampled from a Beta distribution:

λi∼Beta(α,α) with α=0.15λ_i \sim \text{Beta}(α, α) \text{ with } α = 0.15

The mixed voxel input is a convex combination:

xmix,i=λi⋅xi+(1−λi)⋅xkix_{\text{mix}, i} = λ_i \cdot x_i + (1 - λ_i) \cdot x_{k_i}

The projector output for the mixed input is p_i^* = f(x_{\text{mix}, i}), where f is the backbone + projector and p_i^* is L2-normalized. The BiMixCo loss for the image→brain direction (first part of the bidirectional loss) is:

LBiMixCo, img→brain=−∑i=1N[λi⋅log⁡exp⁡(pi∗⋅ti/τ)∑m=1Nexp⁡(pi∗⋅tm/τ)+(1−λi)⋅log⁡exp⁡(pi∗⋅tki/τ)∑m=1Nexp⁡(pi∗⋅tm/τ)]\mathcal{L}_{\text{BiMixCo, img→brain}} = -\sum_{i=1}^N \left[ λ_i \cdot \log \frac{\exp(p_i^* \cdot t_i / τ)}{\sum_{m=1}^N \exp(p_i^* \cdot t_m / τ)} + (1 - λ_i) \cdot \log \frac{\exp(p_i^* \cdot t_{k_i} / τ)}{\sum_{m=1}^N \exp(p_i^* \cdot t_m / τ)} \right]

where t_i is the CLIP image embedding for the i-th image and t_{k_i} is the CLIP image embedding for the mixing partner's image.

For the brain→image direction, the loss must account for the fact that a given image t_j may be the target for multiple mixed brain samples (any sample where j was either the primary index or the mixing index):

LBiMixCo, brain→img=−∑j=1N[λj⋅log⁡exp⁡(pj∗⋅tj/τ)∑m=1Nexp⁡(pm∗⋅tj/τ)+∑{l∣kl=j}(1−λl)⋅log⁡exp⁡(pl∗⋅tj/τ)∑m=1Nexp⁡(pm∗⋅tj/τ)]\mathcal{L}_{\text{BiMixCo, brain→img}} = -\sum_{j=1}^N \left[ λ_j \cdot \log \frac{\exp(p_j^* \cdot t_j / τ)}{\sum_{m=1}^N \exp(p_m^* \cdot t_j / τ)} + \sum_{\{l | k_l = j\}} (1 - λ_l) \cdot \log \frac{\exp(p_l^* \cdot t_j / τ)}{\sum_{m=1}^N \exp(p_m^* \cdot t_j / τ)} \right]

The total BiMixCo loss is the sum of these two directional losses.

What this loss computes: For each mixed brain embedding p_i^*, the loss treats it as λ_i-fraction belonging to image i and (1 - λ_i)-fraction belonging to image k_i, and computes a weighted combination of the two corresponding cross-entropy terms. In the brain→image direction, the log-probability for each image t_j receives contributions from all mixed samples where j appears as either the primary or mixing index, weighted by the corresponding λ value.

Why this form and why the Beta(0.15, 0.15) parameterization: The Beta(0.15, 0.15) distribution is strongly U-shaped — it concentrates probability mass near 0 and 1, with very little near 0.5. This means most mixed samples are dominated by one of the two source samples (e.g., λ ≈ 0.9 or λ ≈ 0.1), with only occasional near-equal mixing. This is important because near-equal mixing (λ ≈ 0.5) would create a brain embedding that is equidistant from two different image embeddings — a confusing training signal that would degrade retrieval sharpness. The concentrated distribution ensures that most training examples have a clear primary assignment while still providing a regularization benefit from the small mixture component. Kim et al. [20] observed that MixCo helps smaller models most, but the authors find it also helps large models in low-data regimes (NSD has only ~25K training samples per subject), likely because it acts as a strong regularizer that prevents the overparameterized 940M-parameter model from memorizing individual training examples.

Table 3 empirically validates BiMixCo: CLIP Loss + MixCo achieves 88.4% image retrieval vs. 83.7% for pure CLIP loss, while InfoNCE + MixCo achieves 30.3% vs. 23.7% for pure InfoNCE. The benefit is consistent across both unidirectional and bidirectional losses.

SoftCLIP: Soft Contrastive Loss via Knowledge Distillation

The soft contrastive loss replaces the hard 1-hot targets of the standard CLIP loss with a soft target distribution derived from the CLIP image embeddings' pairwise similarities. For a batch of N samples, the soft target matrix S is computed as:

Sij=exp⁡(ti⋅tj/τ)∑m=1Nexp⁡(ti⋅tm/τ)S_{ij} = \frac{\exp(t_i \cdot t_j / τ)}{\sum_{m=1}^N \exp(t_i \cdot t_m / τ)}

where t_i and t_j are L2-normalized CLIP image embeddings. This is simply the CLIP image space's own softmax-normalized similarity matrix — it captures which images are semantically similar to each other according to CLIP.

The SoftCLIP loss (shown for the image→brain direction; the brain→image direction is analogous) then trains the brain embeddings to predict this same soft target distribution:

LSoftCLIP, img→brain=−∑i=1N∑j=1N[Sij⋅log⁡exp⁡(pi⋅tj/τ)∑m=1Nexp⁡(pi⋅tm/τ)]\mathcal{L}_{\text{SoftCLIP, img→brain}} = -\sum_{i=1}^N \sum_{j=1}^N \left[ S_{ij} \cdot \log \frac{\exp(p_i \cdot t_j / τ)}{\sum_{m=1}^N \exp(p_i \cdot t_m / τ)} \right]

What this loss computes: For each brain embedding p_i, instead of trying to assign probability 1.0 to image i and 0.0 to all others (hard CLIP loss), the model tries to match the distribution that CLIP itself assigns to image i relative to all images in the batch. If image j is semantically similar to image i (high CLIP cosine similarity), then the brain embedding p_i is encouraged to assign some probability mass to image j as well, proportional to their similarity.

Why this form: The motivation comes from knowledge distillation: the soft target distribution encodes richer information than hard 1-hot labels because it captures the similarity structure of the image space. For a batch containing two different zebra images, the hard CLIP loss treats them as equally wrong for each other (both get zero target probability), whereas SoftCLIP assigns a small but non-zero probability because they are semantically related. This provides a more informative training signal, particularly when the brain embedding for one zebra image might reasonably be somewhat similar to both zebra images. Table 3 shows that SoftCLIP improves brain retrieval over hard CLIP (81.6% vs. 79.1%) but slightly reduces image retrieval (83.7% for both), suggesting that the softer targets help the model learn a more nuanced representation space that better captures semantic similarities.

Training Schedule: BiMixCo → SoftCLIP

The final MindEye training schedule uses BiMixCo for the first one-third of training epochs and SoftCLIP for the remaining two-thirds. This schedule is motivated by recent findings [21, 22] that mixup augmentation, while beneficial early in training for regularization and preventing overfitting, can hurt final performance if continued to convergence because the mixed inputs create a distribution shift between training (mixed samples) and inference (clean samples). Stopping mixup after a burn-in period allows the model to benefit from the strong regularization early — when the risk of memorization is highest — while fine-tuning on clean soft targets later for optimal retrieval quality.

Table 4 quantifies the effect: BiMixCo-only training achieves 97.4% image retrieval (the highest), while the BiMixCo→SoftCLIP schedule achieves 97.2% image retrieval but better reconstruction metrics (CLIP two-way identification: 95.0% vs. 94.4%). The small retrieval sacrifice is worth the reconstruction improvement because the backbone must serve both objectives.

The Diffusion Prior

The diffusion prior is the component that translates "disjointed" CLIP fMRI embeddings from the backbone into "aligned" embeddings suitable for image generation. To understand why this is necessary, we need to examine the geometric phenomenon it addresses.

The Modality Gap Problem

When multiple modalities (text, images, brain activity) are encoded into a shared embedding space using contrastive learning, the resulting embeddings from different modalities tend to occupy distinct, non-overlapping regions of the space — a phenomenon termed the "modality gap" by Liang et al. [24]. This occurs because contrastive learning only incentivizes the direction of embeddings to match across modalities (maximizing cosine similarity for positive pairs), not their magnitude or the shape of their distribution. The result is that, even though a given brain embedding p_i and its corresponding image embedding t_i are directionally aligned (high cosine similarity), they may be separated by a large Euclidean distance and may sit in different submanifolds of the embedding space.

This is problematic for image generation because pretrained models like Versatile Diffusion or Stable Diffusion (Image Variations) were trained to accept CLIP image embeddings as conditioning inputs — that is, embeddings drawn from the distribution P(t) of true CLIP image embeddings. When fed a disjointed fMRI embedding from outside this distribution, the generative model receives an out-of-distribution conditioning signal and produces degraded or nonsensical outputs. The UMAP visualization in Figure 9 empirically demonstrates this: the MLP backbone outputs (orange) and projector outputs (green) form clusters separated from the CLIP image embeddings (blue), while the diffusion prior outputs (red) overlap substantially with the CLIP image distribution.

Prior Architecture and Training

The diffusion prior is a transformer that performs conditional denoising: given a noised CLIP image embedding and the backbone's fMRI-derived conditioning signal, it predicts the noise that was added, allowing recovery of a clean CLIP image embedding. Following the DALL-E 2 formulation [8], the prior models the distribution P(t | z_backbone) — the conditional distribution of true CLIP image embeddings t given the backbone output z_backbone.

The training procedure works as follows. For each training example, a ground-truth CLIP image embedding t_0 (the target, shape 257 × 768) and the backbone output z_backbone (the condition, same shape) are prepared. A timestep s is sampled uniformly from {1, ..., T} where T = 100 (not the 1000 timesteps used in DALL-E 2 — the authors found 100 sufficient and it speeds up inference). Gaussian noise ε ∼ N(0, I) is added to t_0 according to the standard forward diffusion process:

ts=αˉs⋅t0+1−αˉs⋅εt_s = \sqrt{\bar{α}_s} \cdot t_0 + \sqrt{1 - \bar{α}_s} \cdot ε

where \bar{α}_s is the cumulative product of the noise schedule parameters and controls how much signal vs. noise is present at timestep s.

The prior model, a transformer with bidirectional attention, takes three inputs concatenated along the sequence dimension: (1) the 257 token positions of z_backbone (the condition), (2) a learned timestep embedding encoding s, and (3) the 257 token positions of t_s (the noised target). It outputs a prediction ε̂ of the noise ε that was added to t_0 to produce t_s. The training loss is the standard diffusion MSE:

Lprior=Et0,z,s,ε[∥ε−ε^(ts,s,zbackbone)∥22]\mathcal{L}_{\text{prior}} = \mathbb{E}_{t_0, z, s, ε} \left[ \| ε - ε̂(t_s, s, z_{\text{backbone}}) \|^2_2 \right]

where ε̂ is the model's noise prediction, ε is the true noise, and \|\cdot\|^2_2 is the squared Euclidean norm summed over all 257 × 768 = 197,376 dimensions.

What this loss computes: At each training step, the model sees a partially-noised version of the true CLIP embedding and must predict which component of the current embedding is noise versus signal, conditioned on the fMRI-derived backbone representation z_backbone. The loss is the mean squared error between the predicted and true noise vectors.

Why this form: The diffusion objective is a denoising score-matching objective that learns the gradient of the log-density ∇_{t_s} \log P(t_s | z_backbone). By learning to denoise at all noise levels (s from 1 to T), the model learns the full conditional distribution, not just a point estimate of the most likely t_0. At inference time, this allows sampling diverse but plausible CLIP image embeddings from the learned distribution by starting from pure noise and iteratively denoising. The MSE objective is the standard choice because under the Gaussian noise model, the optimal denoiser is the conditional expectation E[ε | t_s, z_backbone], which is what MSE regression recovers.

Modifications from DALL-E 2

The authors made several important modifications to the open-source DALL-E 2 prior implementation:

  1. No learnable queries: DALL-E 2's prior uses a fixed set of learned query tokens that attend to the conditioning information. MindEye removes these and directly predicts denoised CLIP embeddings from the noised embeddings. This significantly reduces memory consumption — critical because the backbone and prior are trained end-to-end on a single A100 GPU with 40–80 GB of memory, and the 257 × 768 = 197K-dimensional representation is already large.

  2. Absolute positional embeddings: To compensate for removing the learned queries, absolute positional embeddings are added to the noised CLIP embeddings. These provide the transformer with explicit position information for each of the 257 token positions, helping it distinguish the CLS token (position 0) from the 256 patch tokens (positions 1–256). Without these, the self-attention mechanism can only infer position from the embedding content itself, which is suboptimal.

  3. 100 timesteps instead of 1000: The authors found that 100 diffusion timesteps work as well as 1000 for this task, making inference 10× faster. This is possible because the CLIP embedding space is already highly structured (close to the image manifold), so the diffusion process doesn't need as many steps to traverse it compared to pixel space.

  4. Bidirectional attention rather than causal: The authors experimented with both and found no significant difference in reconstruction performance. Bidirectional attention is used for simplicity, meaning each token position can attend to all other token positions (both condition and noised target tokens) at every layer.

End-to-End Training with Composite Loss

The diffusion prior is trained jointly with the backbone and projector using a composite loss:

L=LBiMixCo|SoftCLIP+α⋅Lprior\mathcal{L} = \mathcal{L}_{\text{BiMixCo|SoftCLIP}} + α \cdot \mathcal{L}_{\text{prior}}

where α = 0.3 weights the prior loss relative to the contrastive loss. The contrastive loss portion switches from BiMixCo to SoftCLIP after one-third of training (epoch 80 of 240).

Why this composite: The backbone must serve two masters — it must produce representations that are useful for contrastive alignment (via the projector) and for conditioning the diffusion prior. The α = 0.3 coefficient balances these objectives. If α were too large, the prior loss would dominate and the backbone would optimize for reconstruction at the expense of retrieval (the same tradeoff observed in Table 4 for the "Only MLP Backbone" model). If α were too small, the prior would receive poor conditioning signals and produce low-quality aligned embeddings. The value 0.3 was presumably determined through hyperparameter tuning.

The gradient flows back from the prior loss through the backbone but not through the projector — the projector is trained only by the contrastive loss. Similarly, the prior is trained only by its MSE denoising objective. This gradient isolation is what enables the decoupling: each submodule specializes for its task without interference from the other's loss.

The Low-Level (Perceptual) Pipeline

The high-level semantic pipeline, by virtue of operating in CLIP space, necessarily prioritizes semantic content over low-level visual features. CLIP was trained on image-text pairs where captions describe what is in the image but rarely its exact color palette, texture details, or precise spatial layout. Consequently, a reconstruction conditioned only on CLIP embeddings might correctly render "a zebra standing in a grassy field" but could swap the zebra's orientation, alter the grass color, or shift the zebra's position relative to the original image. The low-level pipeline addresses this by providing a separate pathway that directly captures these perceptual features.

Architecture

The low-level pipeline uses an entirely separate MLP backbone (no parameter sharing with the high-level pipeline) with the same architecture: a linear layer (voxels → 4096) followed by 4 residual blocks and a final linear layer, but now mapping to a flattened output of size 16 × 16 × 64 = 16,384. This spatial latent grid is then fed through a CNN upsampler that performs 4× upsampling (16 → 64) to produce latents of shape (64, 64, 4), matching the exact latent space of Stable Diffusion's VAE encoder. The CNN upsampler uses a similar architecture to the VAE decoder of Stable Diffusion, which does 8× upsampling from (8, 8, 4) to (64, 64, 4).

The targets for training are generated by taking the original NSD images (which are square-cropped but of varying resolutions), resizing them to 512 × 512 via bilinear interpolation, and encoding them with the frozen pretrained Stable Diffusion VAE encoder. This produces (64, 64, 4)-dimensional latent tensors. The VAE encoding is a compressed representation that preserves pixel-level information — when decoded back through the VAE decoder, the result is a 512 × 512 image that looks nearly identical to the resized original but with slight blurring (VAEs are lossy compressors).

The low-level pipeline also includes an auxiliary contrastive MLP projector that maps the (16, 16, 64) backbone output to (16, 16, 512) — a higher-dimensional space where a soft contrastive loss is applied. However, unlike the high-level pipeline, this projector is not optimized for retrieval (no bidirectional CLIP loss or BiMixCo) but instead uses SoftCLIP loss with VICRegL features as targets, chosen because VICRegL is specialized for low-level visual features.

Loss Function: Why MAE Over MSE

The reconstruction loss for the low-level pipeline is mean absolute error (MAE, also called L1 loss) between the predicted and target VAE latents:

Llow-level recon=164×64×4∑h=164∑w=164∑c=14∣z^hwc−zhwc∣\mathcal{L}_{\text{low-level recon}} = \frac{1}{64 \times 64 \times 4} \sum_{h=1}^{64} \sum_{w=1}^{64} \sum_{c=1}^4 | \hat{z}_{hwc} - z_{hwc} |

where \hat{z} is the predicted latent tensor and z is the ground-truth VAE latent.

Why MAE instead of MSE: The authors cite literature from the low-level vision community (super-resolution, denoising, deblurring) showing that MAE produces better pixel-level metrics like PSNR and SSIM than MSE. The intuition is that MSE penalizes large errors quadratically, making the optimizer focus disproportionately on outlier pixels at the expense of overall quality, while MAE penalizes errors linearly, leading to more uniform gradient signals across all spatial positions and channels. The VAE latent space is known to behave similarly to pixel space in terms of low-level image properties — it's a linear compression — so the same principles apply.

The authors also experimented with decoding the predicted VAE latents all the way through the VAE decoder to pixel space and computing the loss there (a "full reconstruction" loss), but found this performs worse and requires significantly more GPU memory (the VAE decoder is nontrivial to backpropagate through). Computing the loss in latent space is both more efficient and more effective, confirming that the VAE latent space preserves the relevant low-level structure for direct regression.

Auxiliary Contrastive Loss with VICRegL

The low-level pipeline's auxiliary contrastive loss is not designed for retrieval but rather as an additional training signal to improve the reconstruction submodule. The key design choice is to use VICRegL ConvNext-XXL features as the target embedding space rather than CLIP.

VICRegL [79] is a self-supervised learning method that combines variance, invariance, and covariance regularization with a local feature matching objective (α = 0.75 version). Unlike CLIP, which was trained to align with text captions and thus prioritizes semantic content (what objects are present), VICRegL is trained purely on images with a focus on local feature preservation — it achieves state-of-the-art on linear segmentation and depth estimation, tasks that require fine-grained spatial and textural information rather than semantic categorization.

Why VICRegL for the low-level pipeline: The contrastive loss acts as a regularizer that encourages the low-level backbone to produce representations that are discriminative at the level of local image features rather than semantic categories. By distilling from VICRegL, the low-level backbone learns to capture textures, edges, color distributions, and spatial relationships that are preserved in the VAE latent space but would be lost if the contrastive target were CLIP (which might collapse different zebra images to similar embeddings despite their different poses and backgrounds).

The SoftCLIP loss formulation is used (not hard CLIP and not BiMixCo) because there is no retrieval objective for the low-level pipeline — the soft targets from VICRegL's own similarity matrix provide a richer training signal than hard 1-hot targets by encoding the similarity structure of the VICRegL feature space.

Img2Img: Combining Semantic and Perceptual Pipelines

The final reconstruction combines the two pipelines using img2img (also known as SDEdit), a technique where the diffusion denoising process starts not from pure random noise but from a noised version of an initial image. This allows the high-level semantic pipeline to determine what appears in the reconstruction while the low-level perceptual pipeline biases how it looks.

The procedure works as follows:

  1. The low-level pipeline produces a (64, 64, 4) VAE latent z_low.
  2. Forward diffusion is applied for a fraction strength of the total timesteps: Gaussian noise is added to z_low to produce z_noised, where the amount of noise is controlled by strength. A strength of 0.0 means no noise is added (the reconstruction would exactly match the low-level pipeline output), while a strength of 1.0 means full noise (the low-level pipeline has no influence at all — equivalent to pure high-level generation).
  3. The diffusion prior produces 16 aligned CLIP fMRI embeddings (by sampling the denoising process 16 times with different random seeds, generating diverse samples from the learned conditional distribution P(t | z_backbone)).
  4. These 16 CLIP embeddings are fed to Versatile Diffusion's image variations pipeline, which performs the reverse diffusion process starting from z_noised rather than pure noise, conditioned on the CLIP embedding.
  5. This produces 16 candidate reconstructions. The final image is selected by computing the last hidden layer CLIP embedding of each candidate and picking the one with the highest cosine similarity to the disjointed CLIP fMRI embedding z_proj from the projector — a "second-order selection" mechanism.

Why 16 samples with selection: The diffusion prior generates diverse samples from the learned distribution. Some will be better than others due to the stochastic nature of both the prior sampling and the Versatile Diffusion generation. The second-order selection uses the projector embedding — which was trained for retrieval, i.e., for discriminative similarity to the correct image — to pick the best candidate. This is analogous to DALL-E 2's approach of generating multiple images and using CLIP similarity to select the best one.

Strength parameter selection: Table 5 shows the tradeoff across different img2img strengths for Subject 1. At strength 1.0 (pure low-level), the reconstruction achieves the best pixel-level metrics (PixCorr = 0.456, SSIM = 0.493) but poor high-level metrics (CLIP two-way identification = 62.4%). At strength 0.0 (pure high-level), the opposite occurs (PixCorr = 0.209, SSIM = 0.318, but CLIP = 94.8%). The sweet spot of 0.15 was chosen because it substantially improves low-level metrics over the pure high-level baseline (PixCorr: 0.209 → 0.390, SSIM: 0.318 → 0.337) while causing only minimal degradation in high-level metrics (CLIP: 94.8% → 94.6%). This small strength means the low-level pipeline provides a subtle structural scaffold — enough to lock in color, texture, and rough spatial layout — without constraining the diffusion process so much that it cannot express the semantic content from the CLIP conditioning.

The Versatile Diffusion model used for final generation operates with 20 denoising timesteps using UniPCMultistep noise scheduling, which is a fast sampling method that achieves comparable quality to DDIM or PNDM with fewer steps.

Training Procedure and Hyperparameters

All MindEye models (including both high-level and low-level pipelines) are trained on a single A100 GPU for 240 epochs with a batch size of 32. Despite the large parameter count (~1.2B parameters total for both pipelines combined), training completes in less than 18 hours per subject — the efficiency comes from the fact that the bulk of parameters are in MLPs, which have lower computational cost per parameter than transformers or CNNs because they involve only matrix multiplications without attention patterns or spatial convolutions.

The dataset is the Natural Scenes Dataset (NSD) [26], with subject-specific models trained for each of four participants (Subjects 1, 2, 5, and 7) who completed all scanning sessions. The training set consists of 24,980 fMRI samples (not averaged across repetitions), while the test set consists of 982 samples (averaged across the three same-image repetitions). Averaging across repetitions for the test set improves signal-to-noise ratio because fMRI measurements are noisy, and the three repetitions of each image provide independent noise samples that partially cancel when averaged. The training set is not averaged, keeping all 24,980 samples to maximize training data volume (averaging would collapse the three repetitions into ~8,300 samples, substantially reducing the effective dataset size).

The fMRI data consists of session-wise z-scored single-trial beta values output from GLMSingle [75], a tool for estimating single-trial fMRI responses. The voxels are from the "nsdgeneral" region of interest, defined as the subset of posterior cortex most responsive to the visual stimuli, with between 13,000 and 16,000 voxels per participant depending on individual anatomy. These flattened voxel vectors form the input to MindEye.

For the high-level pipeline, the optimizer, learning rate, and other optimization details are not explicitly stated in the main paper but can be inferred from the training descriptions: AdamW optimizer with a batch size of 32, 240 epochs of training. The mixup parameter for BiMixCo is α = β = 0.15 for the Beta distribution. The composite loss weight α for the prior loss is 0.3. The contrastive loss switches from BiMixCo to SoftCLIP after 80 epochs (one-third of 240).

For the low-level pipeline, the MAE loss is used with an auxiliary soft contrastive loss using VICRegL targets. The img2img strength is set to 0.15 for the final model, and the inference uses 20 UniPCMultistep denoising steps with Versatile Diffusion.

One important implementation detail: the diffusion prior does not use the full 1000-timestep DALL-E 2 noise schedule. Reducing to 100 timesteps makes inference faster because each denoising step requires a forward pass through the prior model. At generation time, the prior must be run for each of the 16 candidate CLIP embeddings, so reducing timesteps from 1000 to 100 provides a 10× speedup in this sampling step. The authors note that the 100-timestep schedule works equally well, likely because the CLIP embedding manifold is already highly structured — the "path" from noise to a valid CLIP embedding is shorter in diffusion terms than the path from noise to a pixel-space image.

4. Key Insights and Innovations

Innovation 1: The Retrieval-Reconstruction Tradeoff Is Fundamental, Not an Implementation Detail — and Specialized Submodules Are the Only Way to Resolve It

The most conceptually significant finding in this paper is not any single architectural choice but rather the empirical demonstration that retrieval (contrastive alignment) and reconstruction (distributional matching for generative models) impose inherently conflicting requirements on the learned embedding space, and that this conflict cannot be papered over with better hyperparameters or more data — it requires a structural solution of decoupled, task-specific output heads.

Prior to MindEye, the dominant approach in fMRI-to-image decoding was to train a single mapping from voxels to CLIP space and use the resulting embeddings for whatever downstream task was needed. Ozcelik and VanRullen [4] trained their CLIP-space mapping purely with MSE loss, achieving passable reconstruction but poor retrieval (21.1%). Lin et al. [11] used a single set of CLIP-space embeddings trained with both contrastive and MSE losses, achieving 11.0% image retrieval. Neither work recognized the retrieval-reconstruction relationship as a tradeoff requiring architectural intervention — they treated poor retrieval as a limitation of their specific training procedure rather than a symptom of a deeper conflict.

MindEye's diagnostic experiment in Table 4 ("Only MLP Backbone") makes the tradeoff explicit and quantifiable: training a single embedding space with both contrastive and MSE losses crashes image retrieval to 13.3% while reconstruction metrics (Incep: 81.5%, CLIP: 82.6%) remain well below what either specialized submodule achieves. The "Backbone + Prior" model (no projector — contrastive loss on backbone, MSE on prior) reaches 93.4% retrieval and 95.1% CLIP two-way identification, while the full MindEye with both projector and prior reaches 97.2% retrieval and 95.0% CLIP — near-optimal on both axes.

Why this is a conceptual innovation rather than an architectural tweak: The field's implicit assumption was that good CLIP-space mappings should be useful for everything — if you can accurately decode what someone saw into CLIP space, your embeddings should work for both retrieval (cosine similarity search) and generation (conditioning a diffusion model). MindEye shows this assumption is false at a fundamental level. Contrastive learning optimizes for directional alignment (cosine similarity), producing embeddings that point in the right direction but have the wrong magnitude — a direct consequence of the "modality gap" phenomenon identified by Liang et al. [24]. Generative models, having been trained on real CLIP image embeddings, expect inputs drawn from that specific distribution with that specific magnitude profile. These two objectives pull the embedding space in genuinely incompatible directions, not just different directions that could be balanced with a loss-weighting coefficient.

The resolution — separate MLP projector for contrastive loss, separate diffusion prior for MSE loss, both fed by a shared backbone — is elegant because it recognizes that the backbone can learn representations that are useful for both tasks without being directly usable for either. The backbone's job becomes producing a rich intermediate representation from which both a directionally-aligned retrieval embedding and a distributionally-correct reconstruction embedding can be derived. This decoupling principle echoes similar insights from self-supervised learning (BYOL, SimCLR) where projection heads separate the representation used for the contrastive objective from the representation used for downstream tasks, but MindEye extends it to the qualitatively different case where the two downstream tasks have conflicting geometric requirements.

The practical implication is that future fMRI-to-image systems should not expect a single embedding space to serve all purposes. This is not a limitation that can be engineered away with better training recipes — it is a structural property of the contrastive-vs-generative tension that will persist regardless of model scale or data volume.

Innovation 2: Retrieval as a First-Class Diagnostic for Representational Fidelity, Not Just a Convenience Metric

MindEye reframes image retrieval from a secondary evaluation metric — something you report alongside reconstructions — into a direct assay of whether decoded brain representations contain exemplar-specific visual information. This conceptual move has implications beyond the specific system: it provides a quantitative, falsifiable criterion for claims about what information fMRI signals contain.

The paper's retrieval results are staggering in absolute terms: 93.2% top-1 accuracy in a pool of 982 test images for Subject 1, compared to 21.1% for the previous state-of-the-art [4] and 11.0% for Lin et al. [11]. But the raw numbers matter less than what they mean. The NSD test set contains images that are intentionally confusable — the authors highlight "over a dozen zebras" as an example. Achieving 93.2% accuracy means the model is not just decoding semantic category ("zebra") but fine-grained exemplar identity ("that specific zebra at that specific angle with that specific background"). This is a fundamentally stronger claim about the information content of fMRI signals than previous work could support.

Prior to MindEye, the dominant narrative in the fMRI decoding literature was that distributed voxel patterns contain coarse category-level information (faces vs. scenes vs. objects) that can be decoded with linear classifiers, and that nonlinear methods can extract somewhat richer semantic gist suitable for rough image reconstruction — but that exemplar-level detail was beyond the resolution of fMRI given its coarse spatial and temporal scale. MindEye's retrieval performance directly challenges this narrative. If you can pick the exact image out of nearly a thousand candidates — including many that differ only in fine spatial configuration, color palette, or object pose — then the voxel patterns must, in fact, contain information at that grain.

Why this matters beyond the metric: The retrieval result acts as a bridge between the engineering goal of high-quality reconstruction and the neuroscientific goal of understanding neural representation. A reconstruction can look plausible without being faithful to the specific viewed image — it might produce a generic zebra in a grassy field that captures the gist but misses the details. Retrieval is unforgiving: there is exactly one correct answer among many highly similar candidates, and success requires genuine exemplar-specific information. This makes retrieval a stricter test of representational fidelity than reconstruction metrics, which typically measure distribution-level similarity (e.g., two-way identification, which asks "is reconstruction A more similar to ground truth A than to ground truth B?" rather than "can you find ground truth A among all candidates?").

The scaling-up to LAION-5B retrieval (Figure 3, bottom) further demonstrates that this exemplar-level signal generalizes: even when the candidate pool is billions of images rather than hundreds, the model retrieves images that are strikingly similar to the originals. This is possible because the CLIP fMRI embeddings genuinely live in the same semantic neighborhood as the corresponding CLIP image embeddings — a claim that could not be made about previous methods where retrieval was near chance once the candidate pool exceeded a few hundred.

However, an important caveat tempers this interpretation: the LAION-5B retrieval results are qualitatively evaluated — the paper shows examples where the retrieved image looks similar to the original, but does not quantify precision or recall at scale. The retrieval might be finding semantically similar images rather than exact matches when the exact original is not in the database (which it likely isn't, since NSD images come from MS-COCO and LAION-5B has different sources). This is still useful — it means the system can "recognize" what someone saw without having the exact image in its database — but it is a different claim than the closed-set 93.2% accuracy.

Innovation 3: Mixup Contrastive Learning (BiMixCo) as a Specifically Motivated Regularizer for Low-Sample Multimodal Alignment

The introduction of BiMixCo — bidirectional mixup contrastive learning with a Beta(0.15, 0.15) mixing distribution and a scheduled transition to soft contrastive targets — is more than a data augmentation trick. It represents a principled adaptation of mixup regularization to the specific geometry of contrastive multimodal alignment in low-data regimes, with careful attention to why standard mixup formulations fail in this setting.

Standard mixup [15] trains on convex combinations of inputs: x_mix = λ·x_i + (1-λ)·x_j with targets mixed in the same proportion. In the context of classification, this is well-motivated as vicinal risk minimization — it encourages the model to behave linearly between training examples, improving generalization. But in a contrastive learning setting with a cross-modal objective, the standard formulation runs into a problem: what is the correct "target" for a mixed brain sample? The brain encoding of a mixture of two viewed images is not the same as a mixture of the brain encodings of those images viewed separately (brain activity is not linear in the stimulus), so there is no ground-truth target for the mixed sample.

BiMixCo solves this by treating the mixed brain embedding as λ-fraction belonging to image i and (1-λ)-fraction belonging to image k_i, and computing a weighted combination of the two corresponding contrastive losses rather than trying to predict a mixed target. This is a subtle but critical distinction from standard mixup: the augmentation happens in the brain embedding space (where linear mixing is at least a reasonable approximation, since linear combinations of voxel patterns can represent co-activation of neural populations), but the loss is defined in the target (CLIP image) space where mixing makes no geometric sense.

The choice of Beta(0.15, 0.15) for the mixing distribution is also specifically motivated. A U-shaped Beta distribution with small α concentrates probability mass near 0 and 1, meaning most mixed samples are dominated by one source. This avoids the near-equal mixing regime (λ ≈ 0.5) where the brain embedding would be approximately equidistant from two different image embeddings — a fundamentally confusing training signal that would encourage the model to place the mixed embedding at the midpoint between two correct targets, degrading the sharpness needed for retrieval. Kim et al. [20] used a similar MixCo formulation with InfoNCE for semi-supervised classification but with a symmetric Beta distribution (and found it helped smaller models most). MindEye's contribution is extending this to the bidirectional CLIP loss and demonstrating that it helps even very large models (940M parameters) when training data is scarce (~25K samples).

The scheduled transition from BiMixCo to SoftCLIP after one-third of training reflects a careful reading of recent literature on mixup training dynamics. Liu and Wang [21] and Yu et al. [22] showed that continuing mixup to convergence can hurt final performance because the model never sees clean training examples — the distribution it learns to process differs from the distribution at inference. By using BiMixCo for the first 80 epochs (strong regularization when overfitting risk is highest, early in training) and SoftCLIP for the remaining 160 epochs (fine-tuning on clean, but soft, targets), the schedule extracts the regularization benefit of mixup without its asymptotic cost. Table 4 shows this is not a free lunch: pure BiMixCo gives the highest retrieval (97.4%) at a small cost to reconstruction, while the scheduled version trades ~0.2 percentage points of retrieval for improved reconstruction.

Why this matters beyond MindEye: The BiMixCo formulation is directly applicable to any low-sample multimodal alignment problem where one modality is substantially noisier or harder to collect than the other — fMRI-to-image, EEG-to-text, neural-to-behavioral mapping. The key insight — that mixup should happen in the input space of the noisier modality and the loss should combine the two corresponding contrastive terms rather than using a mixed target — is general and non-obvious.

Innovation 4: The Diffusion Prior as an Alignment Mechanism for Disjointed Multimodal Embeddings, Not Just a Generative Model Component

MindEye's use of a diffusion prior — adapted from DALL-E 2 [8] but trained from scratch on fMRI-to-CLIP mapping — represents a repurposing of diffusion models from generation to distribution alignment. In DALL-E 2, the diffusion prior maps CLIP text embeddings to CLIP image embeddings, solving a within-modality-family alignment problem (both text and image CLIP embeddings live in the same space but were shown to be somewhat disjointed). MindEye generalizes this to a true cross-modal alignment problem where the source modality (fMRI voxels) is fundamentally different from the target modality (CLIP image embeddings), and the mapping must bridge not just a geometric displacement but a deep representational difference.

The novelty here is not the diffusion model itself — diffusion priors were known — but the recognition that diffusion-based alignment is the right tool for the specific geometric pathology that contrastive learning creates. Contrastive learning (CLIP loss, BiMixCo, SoftCLIP) produces embeddings that are directionally correct — high cosine similarity with the true CLIP image embedding — but that live on a different submanifold of the embedding space with different magnitude statistics. The UMAP visualization in Figure 9 makes this geometric: the MLP projector outputs (green) form a cluster that is displaced from the CLIP image embeddings (blue), while the diffusion prior outputs (red) overlap substantially with the CLIP image distribution.

The diffusion prior solves this by learning the conditional distribution P(t_CLIP_image | z_backbone) rather than a point estimate. This means it doesn't just predict the "closest" CLIP image embedding to the backbone output — it models the full distribution of plausible CLIP image embeddings, which captures both the manifold structure and the magnitude statistics of real CLIP embeddings. The key geometric insight is that you cannot get from a contrastively-trained embedding to the correct distribution via a deterministic mapping (like a simple linear projection or MSE-trained MLP) because the contrastive loss has collapsed out all information about embedding magnitude — the backbone outputs contain direction but not scale, and a deterministic function cannot recover scale from direction alone. The diffusion process, by starting from noise and iteratively denoising conditioned on the backbone output, can sample diverse embeddings from the correct distribution because it has access to both the conditioning signal (for direction) and the learned prior over CLIP embeddings (for magnitude and manifold structure).

The practical consequence — that MindEye works with any pretrained model accepting CLIP image embeddings "without fine-tuning" — is significant but almost a side effect. The deeper contribution is the architectural pattern: when mapping between modalities via contrastive learning, use a projector for retrieval tasks (directional alignment sufficient) and a generative alignment model (diffusion prior or equivalent) for tasks requiring distributional matching. This pattern likely generalizes beyond fMRI-to-image: any system that must both retrieve (compare via cosine similarity) and generate (feed into a pretrained decoder) from cross-modal embeddings could benefit from this decoupling.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use the Natural Scenes Dataset (NSD) [26], a public 7-Tesla fMRI dataset containing brain responses from human participants passively viewing natural scenes from MS-COCO [27]. The paper trains subject-specific models for the four participants who completed all scanning sessions (Subjects 1, 2, 5, and 7), using a standardized train/test split: 24,980 training samples (not averaged across repetitions) and 982 test samples (averaged across three same-image repetitions, following Takagi and Nishimoto [3]). Each participant viewed 9,000–10,000 unique images for three seconds each, repeated three times across 30–40 scanning sessions, yielding ~22,000–30,000 total fMRI trials.

  • Base model(s). MindEye does not use a pretrained base model in the conventional sense — it trains subject-specific MLPs from scratch on each participant's fMRI data. However, it heavily depends on three frozen pretrained models: (1) CLIP ViT-L/14 [7] provides the 257 × 768-dimensional image embedding targets for the high-level pipeline and serves as the evaluation space for retrieval metrics; (2) Versatile Diffusion [6] is the primary image generation model that accepts CLIP image embeddings as conditioning input; (3) Stable Diffusion's VAE [9] provides the (64, 64, 4)-dimensional latent space for the low-level pipeline. The authors also evaluate with Stable Diffusion (Image Variations) [31] and a fine-tuned Lafite GAN [32, 11] as alternative generation backends. The choice of CLIP ViT-L/14 specifically (rather than ViT-B/32 or other variants) is motivated by its richer representational capacity — the last hidden layer preserves 257 token embeddings (CLS + 256 patches) rather than a single pooled vector, providing the spatial granularity that Table 2 shows is essential for fine-grained retrieval.

  • Metrics. The paper evaluates along three axes with multiple complementary metrics per axis. Reconstruction quality (high-level): Two-way identification accuracy using features from AlexNet [37] (layers 2 and 5), InceptionV3 [38] (last pooling layer), and CLIP (final layer of ViT-L/14) — each measures the percentage of pairwise comparisons where the ground-truth image embedding is more similar to its paired reconstruction than to a randomly selected different reconstruction (chance = 50%). EfficientNet-B1 [35] and SwAV-ResNet50 [36] are reported as average correlation distance (lower is better), capturing distribution-level similarity. Reconstruction quality (low-level): Pixelwise correlation (PixCorr) and structural similarity index (SSIM) [34] between ground-truth and reconstructed images — these measure pixel-level and perceptual fidelity respectively. Retrieval: Image retrieval (forward retrieval) measures the percentage of test samples where the correct image is the top-1 match out of 300 randomly selected candidates when queried with a brain embedding (chance ≈ 0.33%), while brain retrieval (backward retrieval) measures the converse — the percentage of test samples where the correct brain embedding is top-1 out of 300 given an image embedding. Both retrieval metrics are averaged over 30 random batch samplings to account for variability in candidate selection. The LAION-5B retrieval results are qualitative only (no precision/recall metrics reported at scale).

  • Baselines. Four prior fMRI-to-image methods serve as quantitative baselines, all evaluated on the same NSD test splits. Lin et al. (2022) [11]: Used a small convolutional model with InfoNCE loss to map voxels to CLIP space, then fine-tuned a Lafite GAN — reported only for Subject 1; achieved 11.0% image retrieval and 49.0% brain retrieval, with 78.2% Inception two-way identification. Takagi and Nishimoto (2022) [3]: Used ridge regression to map voxels to both Stable Diffusion latents and CLIP text latents with separate voxel selections for different model components; reported 83.0% AlexNet(2), 83.0% AlexNet(5), 76.0% Inception, and 77.0% CLIP two-way identification (no retrieval metrics). Gu et al. (2023) [28]: Used surface-based convolutional networks (not flattening voxels) to map to SwAV features, combined with IC-GAN for reconstruction; reported PixCorr = 0.150, SSIM = 0.325, EfficientNet-B1 distance = 0.862, SwAV distance = 0.465. Ozcelik and VanRullen (2023) [4] (Brain-Diffuser): The immediate predecessor and primary baseline — used ridge regression (257 separate models, ~3B parameters total) to map voxels to CLIP space for the high-level pipeline and a VDVAE for the low-level pipeline, with Versatile Diffusion for final generation; reported across the same 4 subjects with PixCorr = 0.254, SSIM = 0.356, AlexNet(2) = 94.2%, AlexNet(5) = 96.2%, Inception = 87.2%, CLIP = 91.5%, Eff = 0.775, SwAV = 0.423, image retrieval = 21.1%, brain retrieval = 30.3%. Additionally, for low-level-only comparisons on Subject 1, Ozcelik and VanRullen's low-level pipeline achieves PixCorr = 0.358, SSIM = 0.437, AlexNet(2) = 97.7%, AlexNet(5) = 97.6%.

  • Generation budget / compute accounting. The paper does not use a standardized "generation budget" concept as the primary axis of comparison (unlike best-of-N scaling papers). Instead, compute is accounted for implicitly through model parameter counts (Table 10) and training time (all models trained on a single A100 GPU in <18 hours). For inference, the paper reports that the diffusion prior uses 100 timesteps (not 1000 as in DALL-E 2) and Versatile Diffusion uses 20 UniPCMultistep denoising steps. For fair comparison, all methods are evaluated on the same test images using the same Versatile Diffusion model where applicable (Ozcelik and VanRullen [4] and MindEye both use Versatile Diffusion; Lin et al. [11] used Lafite; Gu et al. [28] used IC-GAN; Takagi and Nishimoto [3] used Stable Diffusion). The critical fair-comparison element is that all models are trained on the same NSD data with the same train/test splits for the same four subjects.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation for hyperparameter selection across subjects — subject-specific models are trained independently for each participant using their respective fMRI data. For retrieval metrics, 30 random samplings of 300-candidate batches are averaged to account for variability in negative sample selection. For two-way identification, each test sample is compared against all other 981 test samples, and the average correctness across all pairwise comparisons is computed — this exhaustive comparison eliminates sampling variability. The paper developed MindEye using Subject 1's training and validation data, with the test set and other subjects' data untouched until final model training, representing a held-out evaluation protocol rather than cross-validation. The single-trial results (Appendix A.9, Table 8) use random selection of one of three repetitions per image, providing a lower-bound estimate of performance without test-set averaging.

Main Quantitative Results

The paper's experimental narrative is organized around demonstrating state-of-the-art performance across three axes (retrieval, high-level reconstruction, low-level reconstruction), then explaining why the system works through ablations of architecture, loss functions, and training strategies.

Retrieval: Exemplar-Level Decoding with >90% Accuracy

The headline retrieval result appears in Table 1 (top section) and Figure 3: MindEye achieves 93.6% image retrieval and 90.1% brain retrieval accuracy averaged across 4 subjects (chance = 0.33% for 300-candidate retrieval). This represents an improvement of 4.4× over the previous state-of-the-art on image retrieval (Ozcelik and VanRullen [4] at 21.1%) and 3.0× on brain retrieval (30.3% → 90.1%). For Subject 1 specifically, retrieval reaches 97.2% image and 94.7% brain (Table 7) — effectively solving the closed-set retrieval problem on this dataset.

What these numbers mean concretely: When given a brain sample from viewing one of 300 candidate images (containing many confusable examples — the paper highlights over a dozen zebra images in the NSD test set), MindEye identifies the exact original image 93.6% of the time. The 6.4% error rate reflects cases where the retrieved image is visually similar to the ground truth but not identical — i.e., the model correctly identifies semantic category but fails at exemplar-level discrimination in a minority of cases.

Closed-set vs. open-set retrieval: The 93.6% figure is closed-set accuracy within the 982-image test pool (or the 300-candidate subsampling). The LAION-5B retrieval results (Figure 3, bottom; Table 1: "MindEye (LAION)") represent open-set retrieval at scale, but are evaluated only with reconstruction-style metrics (not retrieval accuracy, since ground-truth images are not in LAION-5B). The LAION-retrieved images achieve PixCorr = 0.130, SSIM = 0.308, AlexNet(2) = 84.0%, AlexNet(5) = 92.6%, Incep = 86.9%, CLIP = 86.1% — substantially below the closed-set Versatile Diffusion reconstructions but comparable to or better than previous methods' reconstructions (e.g., Takagi and Nishimoto [3] at AlexNet(2) = 83.0%). This demonstrates that the CLIP fMRI embeddings contain enough signal to find semantically and visually similar images even at billion-image scale, though the retrieved images are approximate matches rather than exact reconstructions.

Nearest-neighbor baselines (implicit): The paper does not report a "CLIP image embedding nearest-neighbor" retrieval baseline — i.e., if you query the CLIP embeddings of the ground-truth images against themselves, what is retrieval accuracy? This would establish the ceiling for CLIP-space retrieval and quantify how much information is lost in the fMRI→CLIP mapping. Since CLIP image embeddings are the targets MindEye tries to predict, the 93.6% retrieval accuracy implies that the predicted CLIP fMRI embeddings preserve ~94% of the discriminative information present in the true CLIP image embeddings for exemplar-level retrieval, but this ceiling comparison is not explicitly quantified.

High-Level Reconstruction: State-of-the-Art Semantic Fidelity

The headline reconstruction result appears in Table 1 (top section) and Figure 4: MindEye achieves 94.7% AlexNet(2), 97.8% AlexNet(5), 93.8% Inception, and 94.1% CLIP two-way identification (averaged across 4 subjects). This represents consistent improvements over Brain-Diffuser [4] across all high-level metrics: AlexNet(2) from 94.2% → 94.7% (+0.5 percentage points), AlexNet(5) from 96.2% → 97.8% (+1.6 p.p.), Inception from 87.2% → 93.8% (+6.6 p.p.), CLIP from 91.5% → 94.1% (+2.6 p.p.), Eff distance from 0.775 → 0.645 (−0.130, i.e., 16.8% improvement), SwAV distance from 0.423 → 0.367 (−0.056, i.e., 13.2% improvement).

Why the InceptionV3 improvement is particularly notable: The InceptionV3 metric shows the largest relative gain (+6.6 percentage points) over Brain-Diffuser. InceptionV3, trained on ImageNet classification, encodes mid-to-high-level visual features that are more semantically structured than AlexNet's early-layer features but less abstract than CLIP's vision-language representations. The large gain suggests that MindEye's nonlinear mapping and diffusion prior are particularly effective at recovering these intermediate-level semantic features that ridge regression (Brain-Diffuser's approach) struggles to capture. This aligns with the architectural motivation: linear methods can capture coarse category-level information but miss the nonlinear interactions between visual features that InceptionV3 is sensitive to.

Distribution-level metrics (Eff and SwAV): The EfficientNet-B1 and SwAV correlation distances measure a different property than two-way identification — they capture how well the distribution of reconstructed images matches the distribution of ground-truth images in these pretrained feature spaces, rather than pairwise matching. MindEye improves both substantially (Eff: 0.775 → 0.645, SwAV: 0.423 → 0.367), indicating that the reconstructed images are not just individually better-matched to their ground truths but also collectively more representative of the natural image manifold. This matters for downstream applications where the goal is to generate image distributions that are statistically indistinguishable from the viewed images (e.g., studying population-level neural representations).

The high-level-only vs. combined pipeline comparison (Table 1): "MindEye (High-Level)" — the pure semantic pipeline without img2img — achieves essentially identical high-level metrics to the combined model (Incep: 93.6% vs. 93.8%, CLIP: 94.2% vs. 94.1%), confirming that the low-level pipeline's contribution is primarily to low-level perceptual features without degrading semantic fidelity. The high-level-only model actually achieves marginally better CLIP two-way identification (94.2% vs. 94.1%), consistent with the small tradeoff seen in the img2img strength ablation (Table 5), where increasing perceptual influence slightly reduces high-level metric performance.

Qualitative comparison (Figure 4): The side-by-side comparison shows MindEye reconstructions that capture both semantic content and spatial layout more faithfully than Brain-Diffuser, Lin et al., Takagi and Nishimoto, and Gu et al. For example, in images containing multiple objects or complex scenes, MindEye's reconstructions better preserve the relative positions, sizes, and identities of objects, whereas other methods often collapse multiple objects into a single blob or swap object categories. However, the qualitative assessment is inherently subjective — the paper relies on quantitative metrics for rigorous comparison.

Low-Level Reconstruction: Perceptual Fidelity via Separate VAE Mapping

The low-level pipeline results (Table 1, middle section: "MindEye (Low-Level)") show that mapping directly to Stable Diffusion's VAE latent space produces reconstructions with PixCorr = 0.360 and SSIM = 0.479 — substantially better than Brain-Diffuser's VDVAE-based low-level pipeline (PixCorr = 0.254, SSIM = 0.356). However, these pure low-level reconstructions achieve poor high-level metrics (AlexNet(2) = 78.1%, CLIP = 59.2%), confirming that VAE latents preserve pixel-level structure at the expense of semantic content.

Subject 1 comparison (Table 1, bottom section): For Subject 1 specifically, MindEye's low-level pipeline achieves PixCorr = 0.456 and SSIM = 0.493, compared to Ozcelik and VanRullen's low-level pipeline at PixCorr = 0.358 and SSIM = 0.437. This represents a 27.4% improvement in PixCorr and 12.8% improvement in SSIM. Critically, Ozcelik and VanRullen's VDVAE-based approach achieves substantially better high-level metrics on these same blurry reconstructions (AlexNet(2) = 97.7% vs. 87.1%), indicating that VDVAE latents preserve more semantic information than Stable Diffusion's VAE latents even in blurry reconstructions. This is a tradeoff: MindEye's low-level pipeline prioritizes pixel-level fidelity at the cost of semantic content, but this is acceptable because the high-level pipeline provides the semantics and the low-level pipeline only needs to contribute perceptual scaffold.

Combined pipeline results: With img2img strength of 0.15, the combined MindEye achieves PixCorr = 0.309 and SSIM = 0.323 (Table 1, top section). Compared to the pure high-level pipeline (PixCorr = 0.194, SSIM = 0.308 from Table 5 at strength 0.0, or Table 1: "MindEye (High-Level)" at PixCorr = 0.194, SSIM = 0.308 — wait, Table 1 shows High-Level SSIM = 0.308, but the text and Table 5 show 0.318 for strength 0.0 — this appears to be an inconsistency in the paper where Table 1 reports SSIM = 0.308 for "MindEye (High-Level)" but Table 5 reports SSIM = 0.318 for strength 0.0 on Subject 1. Since Table 1 averages across subjects and Table 5 is Subject 1 only, the difference is likely due to other subjects having lower SSIM, bringing the average down to 0.308). The combined pipeline thus improves PixCorr by ~59% (0.194 → 0.309) relative to pure high-level, while causing minimal degradation in high-level metrics (CLIP: 94.2% → 94.1%). This demonstrates effective disentanglement of the two pipelines — the semantic pathway determines content, the perceptual pathway biases appearance, and the img2img strength parameter controls the tradeoff.

Comparison with Brain-Diffuser on combined metrics: MindEye achieves better PixCorr (0.309 vs. 0.254) but slightly worse SSIM (0.323 vs. 0.356) than Brain-Diffuser overall. The SSIM reversal — Brain-Diffuser having better SSIM despite worse PixCorr — is interesting. PixCorr measures per-pixel linear correlation, which is sensitive to global luminance and contrast matching, while SSIM measures perceived structural similarity through luminance, contrast, and structure comparisons in local windows. Brain-Diffuser's higher SSIM but lower PixCorr might indicate that its reconstructions have better local texture structure but worse global intensity matching. However, MindEye's separately reported "Low-Level" pipeline achieves substantially better SSIM than Brain-Diffuser's low-level pipeline (0.479 vs. 0.356 average across subjects, or 0.493 vs. 0.437 for Subject 1), suggesting that the img2img blending process — specifically at the low strength of 0.15 — doesn't fully preserve the low-level pipeline's SSIM advantage.

Single-Trial Results: Performance Without Repetition Averaging

The standard test-set evaluation averages across three same-image fMRI repetitions, which improves signal-to-noise ratio but is unrealistic for many applications where only a single brain measurement is available. Appendix A.9 (Table 8) evaluates MindEye using single-trial brain activity (randomly selected from one of three repetitions per image). Single-trial performance drops across all metrics but remains competitive: averaged across subjects, PixCorr drops from 0.309 → 0.255 (−17.5%), SSIM from 0.323 → 0.308 (−4.6%), Incep from 93.8% → 91.3% (−2.5 p.p.), CLIP from 94.1% → 91.6% (−2.5 p.p.), image retrieval from 93.6% → 80.3% (−13.3 p.p.), and brain retrieval from 90.1% → 77.6% (−12.5 p.p.).

Why retrieval degrades more than reconstruction: The retrieval metrics drop substantially more (−13 p.p.) than the two-way identification metrics (−2.5 p.p.). This makes sense: two-way identification asks the relatively easier question "is reconstruction A closer to ground truth A or ground truth B?" while retrieval asks "can you find ground truth A among 300 candidates?" Single-trial noise adds variance to the CLIP fMRI embeddings, which spreads out the embedding space and makes it harder to precisely locate the correct match among many similar candidates — a problem that is more punishing for the 300-way classification implicit in retrieval than for the pairwise comparisons in two-way identification.

Subject variability (Table 8): Subject 1 maintains strong single-trial performance (image retrieval: 89.0%, Incep: 92.8%), while Subject 4 drops substantially (image retrieval: 68.5%, Incep: 87.6%). The rank-ordering of subjects by single-trial performance matches the rank-ordering by averaged performance (Table 7), suggesting that inter-subject variability in data quality (perhaps due to head motion, attention, or neuroanatomical differences) is the primary factor rather than differential sensitivity to averaging.

Dataset Size Sensitivity: How Much Training Data Is Needed?

Appendix A.10 (Table 9) trains MindEye's high-level pipeline with reduced training data on Subject 1. With half the training data (~12,500 samples), retrieval metrics remain state-of-the-art: image retrieval = 77.5%, brain retrieval = 60.8% — these numbers are 3.7× and 2.0× higher than Brain-Diffuser's full-data performance (21.1% and 30.3%), despite using only half the data. Reconstruction degrades more noticeably: Incep drops from 94.5% → 87.1%, CLIP from 94.8% → 90.1%. With only ~1,500 training samples (equivalent to data from roughly two scanning sessions), image retrieval falls to 17.9% — comparable to Brain-Diffuser's full-data performance on retrieval (21.1%) — and Incep drops to 79.2% — comparable to Takagi and Nishimoto's full-data performance (76.0%).

What this tells us about data efficiency: The retrieval task shows remarkable robustness to data reduction — half the training data still yields 77.5% retrieval, which is state-of-the-art relative to prior full-data methods. This suggests that the contrastive learning framework with BiMixCo is highly data-efficient for learning directionally correct embeddings, and that the exemplar-level signal in fMRI voxels is strong enough that a well-regularized nonlinear model can extract it from relatively few examples. Reconstruction degrades more rapidly with data reduction, consistent with the harder problem of learning the full conditional distribution of CLIP embeddings (requiring the diffusion prior to model the manifold) versus just learning directional alignment (requiring only that the projector separate different images).

The two-session result is particularly striking for its practical implications: achieving retrieval and reconstruction performance comparable to or exceeding prior methods' full-data results with only ~2 hours of scanning (vs. ~40 hours for the full NSD protocol) suggests that MindEye's approach could be adapted to much more practical data collection regimes, though the absolute performance in this regime (17.9% retrieval, 79.2% Incep) is still far below the full-data model.

Individual Subject Results: Consistency Across Participants

Table 7 and Figure 11 show results for all four subjects individually. Performance varies systematically across subjects: Subject 1 achieves the best results across nearly all metrics (CLIP = 94.6%, image retrieval = 97.2%), while Subject 4 achieves the lowest (CLIP = 93.0%, image retrieval = 89.4%). The rank ordering is consistent: Subject 1 > Subject 2 > Subject 3 > Subject 4 for most metrics. Despite this variation, all four subjects substantially outperform all baselines — Subject 4's image retrieval of 89.4% is still 4.2× higher than Brain-Diffuser's best retrieval (21.1%), and Subject 4's CLIP two-way identification of 93.0% exceeds Brain-Diffuser's 91.5%.

This cross-subject consistency is important for two reasons: First, it demonstrates that MindEye's architecture is not overfit to a single high-quality participant — the method transfers across individuals with different brain anatomy, signal quality, and attention levels. Second, it establishes a performance floor: the worst subject still dramatically outperforms prior methods, suggesting that future improvements in data quality (better fMRI sequences, longer scanning, motion correction) could push already-strong subjects even higher.

Ablation Studies and Robustness Checks

The paper's ablation strategy is organized around understanding three sources of performance gains: architectural depth and parameter count (Table 2), loss function and data augmentation choices (Table 3), and the decoupling of retrieval and reconstruction objectives (Table 4). Each ablation trains only the retrieval pipeline (no diffusion prior) for 120 epochs with batch size 300 on Subject 1, making the numbers directly comparable within each table but not directly comparable to the full MindEye system (which includes the prior and trains for 240 epochs).

Model depth and parameter count (Table 2): Increasing from no residual blocks (873M parameters) to 2 residual blocks (907M) to 4 residual blocks (940M) produces monotonic improvements in image retrieval (88.0% → 88.6% → 89.6%) but non-monotonic behavior in brain retrieval (82.0% → 83.7% → 82.2%). The brain retrieval peaking at 2 blocks and then declining at 4 blocks suggests that deeper architectures improve the discrimativeness of brain embeddings for finding the correct image but may hurt the inverse direction — perhaps because the additional capacity allows the model to specialize its representations in a way that makes brain embeddings more separable (good for image retrieval) but less well-aligned with the CLIP image embedding manifold (bad for brain retrieval). Skip connections become critical at depth: the 4-block model without skip connections collapses to 83.6% image retrieval and 76.7% brain retrieval, while the 2-block model without skips barely degrades (88.1% vs. 88.6% for image retrieval). This confirms the standard deep learning principle that residual connections enable training deeper networks, but the specific depth threshold where they become necessary (between 2 and 4 blocks) is task-specific.

Output dimensionality (Table 2): Mapping to only the CLS token (final layer CLIP output, 1 × 768) rather than all 257 tokens (last hidden layer, 257 × 768) reduces parameter count 7× (940M → 135M) and collapses retrieval performance: 89.6% → 61.1% image retrieval, 82.2% → 57.6% brain retrieval. This is one of the paper's most important results because it demonstrates that the additional 256 patch tokens contain decodable spatial information from fMRI voxels. The CLS token alone captures global image semantics (what objects are present), while the patch tokens capture spatial layout (where objects are positioned). The large performance gap indicates that fMRI voxels encode fine-grained spatial information — consistent with the known retinotopic organization of visual cortex, where different cortical locations respond to different regions of the visual field. This result also validates the architectural choice to use the last hidden layer (pre-pooling) rather than the final pooled output as the mapping target.

Loss function comparison (Table 3): InfoNCE (unidirectional) achieves only 23.7% image retrieval but 78.4% brain retrieval — a 3.3× asymmetry. This reflects the mathematical structure of InfoNCE, which optimizes for brain→image discrimination (given an image, identifying the correct brain sample) but provides no direct gradient for the reverse direction. CLIP loss (bidirectional) essentially equalizes the two: 83.7% image retrieval, 79.1% brain retrieval. MixCo augmentation benefits both: InfoNCE + MixCo reaches 30.3% image retrieval (+6.6 p.p.), CLIP + MixCo (BiMixCo) reaches 88.4% image retrieval (+4.7 p.p.) and 84.1% brain retrieval (+5.0 p.p.). The BiMixCo benefit is larger for brain retrieval than image retrieval, suggesting that mixing voxel samples provides a regularization effect that particularly helps the brain→image mapping learn more robust representations.

SoftCLIP vs. hard CLIP (Table 3): SoftCLIP achieves 83.7% image retrieval (same as hard CLIP loss) and 81.6% brain retrieval (better than hard CLIP's 79.1%). The brain retrieval improvement suggests that soft targets, which encode the similarity structure of the image space, help the model learn representations that better preserve semantic relationships — making it easier to find the correct brain sample for a given image, especially when similar images are in the batch. The MindEye schedule combining BiMixCo and SoftCLIP (BiMixCo for first third, SoftCLIP for remainder) achieves the best overall: 89.6% image retrieval, 82.2% brain retrieval. This schedule outperforms either loss alone, demonstrating that the benefits of mixup regularization (early training) and soft target distillation (later training) are complementary.

Decoupling retrieval and reconstruction (Table 4): The "Only MLP Backbone" model — applying both contrastive and MSE losses to the same backbone outputs without separate submodules — achieves only 13.3% image retrieval and 63.1% brain retrieval, confirming that the two losses conflict destructively. Adding a separate MLP projector ("Backbone + Projector") restores retrieval to 88.8% image / 84.9% brain while improving reconstruction modestly over the backbone-only model but still far below state-of-the-art. Replacing the projector with a diffusion prior ("Backbone + Prior") achieves strong reconstruction (Incep = 94.8%, CLIP = 95.1%) while maintaining 93.4% image retrieval — nearly as good as the full system. The full MindEye with both submodules achieves the best balance: 97.2% image retrieval (with BiMixCo schedule) or 96.0% brain retrieval (in the final 0.33 BiMixCo version) while achieving Incep = 94.5% and CLIP = 94.6% on reconstruction (Table 1, Subject 1 values, not Table 4 — note that Table 4 reports slightly different numbers because these are retrieval-pipeline-only models without the full 240-epoch training and img2img combination).

The key negative finding in Table 4: The "Backbone + Prior" model achieves reconstruction metrics comparable to or even slightly better than the full MindEye (Incep = 94.8% vs. 94.6%, CLIP = 95.1% vs. 95.0%) while achieving competitive retrieval (93.4% image, 90.1% brain). This suggests that the diffusion prior alone, without the separate MLP projector, can largely resolve the retrieval-reconstruction tradeoff — the prior's denoising objective provides enough directional signal to support reasonable retrieval even without explicit contrastive training of a separate projector. However, the projector adds ~3.8 p.p. of image retrieval (93.4% → 97.2%), which matters for the paper's central claim about exemplar-level decoding. This near-redundancy of the projector for reconstruction is an interesting result that the paper doesn't fully explore — it suggests that a simplified architecture (backbone + prior only) might achieve nearly state-of-the-art reconstruction with competitive retrieval while being simpler and faster to train.

Img2img strength (Table 5): Varying the img2img strength from 0.0 (pure high-level, no perceptual scaffold) to 1.0 (pure low-level, no semantic conditioning) reveals a smooth tradeoff. Low-level metrics (PixCorr, SSIM) monotonically increase with strength, while high-level metrics show an inverted-U pattern peaking at intermediate strengths. The sweet spot at strength = 0.15 achieves: PixCorr = 0.390 (vs. 0.209 at strength 0.0, +86.6%), SSIM = 0.337 (vs. 0.318, +6.0%), AlexNet(2) = 97.4% (vs. 92.8%, +4.6 p.p.), Incep = 94.5% (vs. 94.5%, no change), CLIP = 94.6% (vs. 94.8%, -0.2 p.p.). The fact that intermediate strengths improve some high-level metrics (AlexNet(2), AlexNet(5) peak at strength 0.3–0.5) relative to pure high-level suggests that the low-level scaffold doesn't just preserve perceptual features passively — it also provides structural constraints that help the diffusion model produce semantically more accurate outputs, perhaps by preventing the model from drifting into unrealistic image configurations during denoising.

Alternative generation models (Table 6, Appendix A.7): MindEye + Versatile Diffusion outperforms MindEye + Stable Diffusion (Image Variations) and MindEye + Lafite on most metrics for Subject 1. Versatile Diffusion achieves CLIP = 94.6% vs. SD Image Variations at 92.5% and Lafite at 87.0%. However, SD Image Variations achieves the highest SSIM (0.350 vs. 0.337 for Versatile Diffusion and 0.304 for Lafite), suggesting it better preserves structural details despite lower semantic fidelity. This variation across generation backends demonstrates that MindEye's CLIP fMRI embeddings are sufficiently general to work with multiple models, and that reconstruction quality will continue to improve as better generative models become available — a point the authors emphasize as a key advantage of the approach.

LAION-5B retrieval variant (Table 1): The mapping to the final layer of CLIP (rather than the last hidden layer) required for LAION-5B retrieval (because the CLIP Retrieval client has precomputed embeddings for the final layer) produces worse reconstructions across all metrics compared to the last-hidden-layer mapping used with Versatile Diffusion. Specifically, "MindEye (LAION)" achieves Incep = 86.9% vs. 93.8% for the full MindEye — a 6.9 p.p. drop — and CLIP = 86.1% vs. 94.1%. This is consistent with Table 2's finding that mapping to only the CLS token (which the final layer pooling effectively does) loses substantial decodable information. The LAION retrieval quality is thus limited by the need to use a less informative CLIP representation; if the CLIP Retrieval client supported last-hidden-layer embeddings, LAION-scale retrieval quality would likely improve.

ReST^EM revision model (not applicable): This paper is about fMRI-to-image, not about LLM self-correction. There are no revision model experiments.

Critical Assessment

The experiments in this paper are unusually thorough for a conference paper — the ablation studies systematically isolate the contributions of architecture depth, loss function, data augmentation, output dimensionality, objective decoupling, img2img strength, and generation model choice, with quantitative results for each. The main claims are well-supported by the evidence, though several gaps and interpretational caveats warrant attention.

Claim 1: "MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks" (Abstract, Table 1). This claim is strongly supported for the specific baselines, metrics, and dataset used. MindEye outperforms all four compared methods (Lin et al. [11], Takagi and Nishimoto [3], Gu et al. [28], Ozcelik and VanRullen [4]) on every reported metric in Table 1 except SSIM (where Brain-Diffuser achieves 0.356 vs. MindEye's 0.323 — a reversal of the trend where SSIM is slightly worse despite better PixCorr). The margin of improvement is substantial on most metrics (Inception: +6.6 p.p.; image retrieval: +72.5 p.p.). However, the claim's scope is narrower than it might appear: all comparisons are on the NSD dataset with MS-COCO natural scenes, all models use the same train/test splits for the same four subjects, and the baselines represent a specific evolutionary stage of fMRI-to-image methods (~2022–early 2023). It is possible that even simpler methods — e.g., ridge regression with more sophisticated voxel selection or feature engineering — could approach MindEye's performance, but this comparison is not tested. The paper does not compare against the contemporaneous Mind-Vis [59] (which used a different dataset and a masked brain model pretraining approach) or Mind-Video [60], so "state-of-the-art" should be understood as "state-of-the-art on NSD among published methods as of mid-2023."

Claim 2: "Specialized submodules for retrieval and reconstruction enable a single model to achieve state-of-the-art results across both tasks even though the objectives exhibit a tradeoff" (Main findings, Section 3.3). This claim is strongly supported by Table 4, which directly compares single-head vs. multi-head architectures. The "Only MLP Backbone" model's catastrophic failure at retrieval (13.3%) while achieving mediocre reconstruction (Incep = 81.5%) provides unambiguous evidence of the tradeoff. The full MindEye achieving both high retrieval (97.2%) and high reconstruction (CLIP = 95.0%) demonstrates that the decoupling works. However, the experiment that would fully validate the claim — testing whether a single-head model with careful loss balancing and hyperparameter tuning could approach the multi-head model's performance — is not performed. The "Only MLP Backbone" model uses equal weighting of contrastive and MSE losses with no tuning of the loss ratio; it's possible that a heavily MSE-weighted single-head model could achieve decent retrieval, or vice versa. The "Backbone + Prior" model (no separate projector) achieves 93.4% retrieval with strong reconstruction — nearly closing the gap to the full model — which suggests that a single diffusion prior head might be sufficient for most practical purposes, weakening the claim that specialized submodules are strictly necessary (as opposed to simply having at least the diffusion prior submodule).

Claim 3: "Mapping to a deep MLP with a parameter count orders of magnitude higher than previous methods does not produce overfitting and instead directly benefits model performance" (Main findings). This claim is supported by Table 2, which shows monotonic improvement in image retrieval with increasing depth and parameter count (873M → 907M → 940M). However, the evidence for "does not produce overfitting" is indirect — the paper shows that test-set performance improves with model size, but does not report training-set performance to explicitly quantify the generalization gap. A model with 940M parameters trained on 25K samples has ~37,600 parameters per training example, which is extraordinarily high. The fact that it generalizes suggests either that the effective dimensionality of the fMRI signal is much higher than the sample count (due to the structured, high-dimensional nature of voxel activation patterns) or that the aggressive regularization (dropout at 0.5 in the first layer, 0.15 in residual blocks, BiMixCo augmentation, soft targets) is doing heavy lifting. An ablation of the regularization components (e.g., dropout rates, mixup vs. no mixup, soft vs. hard targets) at different model sizes would clarify whether the large model succeeds because of or despite its size, but this is not reported. The comparison in Table 10 shows that MindEye (996M parameters combined) is comparable in total size to Ozcelik and VanRullen's 257 × 12M = ~3B parameter ridge regression system, so "orders of magnitude higher" is somewhat misleading — it's one order of magnitude higher than the per-model size of previous nonlinear methods (Lin et al.'s ~1.17M parameter convolutional model), but the linear methods used massive effective parameter counts through separate models per CLIP dimension.

Claim 4: "BiMixCo further improves model performance in this low-sample setting" (Main findings). This claim is supported by Table 3, which shows BiMixCo (CLIP + MixCo) achieving 88.4% image retrieval vs. 83.7% for CLIP loss alone, and Table 4 showing the BiMixCo-only variant achieving the highest retrieval (97.4%). The specific contribution of the Beta(0.15, 0.15) parameterization vs. other mixing distributions is not ablated, so the claim that this particular U-shaped distribution is important remains a theoretical argument rather than an empirically validated design choice. Similarly, the transition schedule from BiMixCo to SoftCLIP is justified by citation to prior work [21, 22] but not ablated against alternatives (e.g., what if you used BiMixCo for 50% or 75% of training? What if you used SoftCLIP throughout?). These are reasonable omissions given space constraints, but they leave open the question of whether the specific schedule is optimal or merely sufficient.

Claim 5: "Mapping to Stable Diffusion's VAE latent space yields state-of-the-art reconstructions for low-level image metrics" (Main findings). This claim is supported with qualifications by Table 1 and the Subject 1 comparison. On Subject 1, MindEye's low-level pipeline achieves PixCorr = 0.456 vs. Brain-Diffuser's 0.358 and SSIM = 0.493 vs. 0.437 — clear improvements. However, Brain-Diffuser's VDVAE-based low-level pipeline achieves substantially better high-level metrics on the same blurry reconstructions (AlexNet(2) = 97.7% vs. 87.1%, Incep = 77.0% vs. 61.6%), indicating that the VAE vs. VDVAE choice involves a semantic-perceptual tradeoff that the paper acknowledges but doesn't fully characterize. The "state-of-the-art" claim for low-level metrics is accurate, but the VDVAE alternative preserved more semantic information in the low-level pipeline — which might be advantageous in architectures that blend the two pipelines differently. The paper's img2img approach compensates for the VAE's semantic limitations by relying on the high-level pipeline for semantics, but a VDVAE-based low-level pipeline combined with MindEye's high-level pipeline might achieve even better overall results — this combination is not tested.

Missing ablation: Voxel selection and preprocessing. The paper uses preprocessed flattened fMRI voxels from the "nsdgeneral" region of interest with session-wise z-scored single-trial betas from GLMSingle. No ablation studies vary the voxel selection (e.g., using only early visual cortex vs. higher visual areas vs. whole brain), the preprocessing pipeline (e.g., without GLMSingle, without z-scoring, with different smoothing kernels), or the input dimensionality (e.g., PCA reduction of voxel space). This matters because Brain-Diffuser [4] and Takagi and Nishimoto [3] used different voxel selections for different model components — a design choice that MindEye doesn't explore. It's possible that some of MindEye's performance gain comes from the unified voxel space allowing the MLP to learn cross-region interactions that separate voxel selections would prevent, but this hypothesis is untested.

Missing baseline: Ablated CLIP-only retrieval ceiling. As noted earlier, the paper doesn't report a "CLIP image embedding nearest-neighbor" retrieval baseline to establish the ceiling. If CLIP image embeddings achieve, say, 98% retrieval on the 300-candidate task, then MindEye's 93.6% represents a 4.4 percentage point gap from ceiling — a more informative framing than the 72.5 p.p. gap from Brain-Diffuser. If CLIP image embeddings achieve only 93.6% (i.e., MindEye perfectly recovers all discriminative information in CLIP space), then the fMRI→CLIP mapping is essentially solved and future improvements must come from better target spaces, not better mapping architectures.

Missing evaluation: Statistical significance and confidence intervals. The paper reports point estimates for all metrics but no confidence intervals, standard errors, or statistical tests comparing MindEye to baselines. For the retrieval metrics, 30 random samplings provide some variability estimate, but this is reported only as a single averaged number. For the two-way identification metrics, which average across all 982 × 981 pairwise comparisons, the effective sample size is enormous and variance is presumably small, but this is not quantified. For metrics like PixCorr and SSIM computed per-image, subject-level standard deviations would help assess whether MindEye's advantages are consistent across images or driven by a subset of easy cases.

Missing evaluation: Reconstruction diversity and failure modes. The paper shows select reconstruction examples in Figures 1, 4, 5, and Appendix Figures, but does not systematically analyze failure modes. What kinds of images does MindEye reconstruct poorly? Are failures concentrated on images with multiple small objects, unusual viewpoints, rare categories, or high clutter? A per-image breakdown of reconstruction quality vs. image characteristics would provide neuroscientific insight (what visual features are poorly encoded in fMRI?) and practical guidance (what kinds of stimuli are unsuitable for fMRI-to-image decoding?). The retrieval confusion matrix — which specific images are most often confused with each other — would similarly illuminate the structure of the decoded representation space.

Missing comparison: Inter-subject decoding. The paper trains subject-specific models for each participant, consistent with prior work. No experiments test cross-subject generalization (training on Subject 1, testing on Subject 2) or multi-subject shared models. The subject-specific approach is well-motivated by known individual differences in cortical anatomy and functional organization, but establishing a lower bound on cross-subject performance would inform the practicality of deploying these models without per-subject calibration — the authors mention this as a future direction (Section 5) but provide no initial results.

The test set size and composition caveat: The 982-image test set, while standard for NSD evaluations, is relatively small for making broad claims about natural scene reconstruction. The images are drawn from MS-COCO, which has specific biases (object-centric, limited scene diversity compared to truly open-domain image collections). The qualitative LAION-5B retrieval results are encouraging but not quantitatively evaluated. The paper's results should be understood as specific to the MS-COCO image distribution as viewed through the NSD experimental paradigm (3-second passive viewing, specific MRI sequences, specific participants).

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Prohibitively Expensive for Practical Deployment

The assumption or constraint. The entire compute-optimal framework is built on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper’s method for doing so — generating 2048 samples per question and computing pass@1 (oracle) or averaging PRM final-answer scores (predicted) — requires substantially more computation than the test-time budgets being optimized. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The 2048-sample difficulty estimation step consumes 8× more generations than the largest budget studied in many experiments (256 generations) and 128× more than the budget at which the 4× efficiency gains are claimed (16 generations matching 64).

The consequence. The headline 4×4\times efficiency figure over best-of-N is computed after difficulty is known, excluding the cost of learning it. In any practical deployment, the true cost would be difficulty estimation plus strategy execution. If difficulty estimation costs 2048 generations, the total budget for a query that subsequently receives N=16N = 16 generations of compute-optimal search is 20642064 generations — making the approach less efficient than simply applying best-of-N with 512 generations uniformly, which would cost 512 generations total and likely achieve comparable or better accuracy on most problems. This means the reported efficiency gains are an upper bound on achievable efficiency under the unrealistic assumption of free difficulty estimates, not a realized deployment gain. For interactive applications where latency matters, the 2048-sample pre-computation would add tens of seconds of serial generation time before any answer is produced, making the approach entirely impractical regardless of throughput efficiency.

What evidence exists in the paper. The paper never includes difficulty estimation cost in any budget calculation (Figures 4, 8, 9). The predicted difficulty bins are generated using the same 2048-sample procedure, just with PRM scores instead of ground-truth correctness — the computational cost is identical. The paper reports that oracle and predicted difficulty bins yield similar performance (Figures 4, 8: curves "largely overlap"), confirming that the approach works without ground-truth labels, but this does nothing to address the computational cost of the estimation step. The paper never reports the total budget including difficulty estimation for any experiment.

Mitigation status. The paper acknowledges the issue explicitly in Section 3.2 and frames the difficulty estimation cost as "an exploration-exploitation tradeoff" to be addressed in future work. It suggests "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but develops no such model and provides no evidence that difficulty can be predicted accurately from question text alone without the 2048-sample procedure. A lightweight difficulty classifier is a natural next step — perhaps distilled from the PRM's average score distribution — but until such a classifier exists and its accuracy is validated, the compute-optimal framework exists in a regime where the cure (difficulty estimation) costs more than the disease (uniform allocation). The paper's own data provides a suggestive but incomplete starting point: if a smaller model (say, 32 samples) could estimate difficulty with sufficient accuracy to route problems to the correct bin, the overhead would drop from 2048 to 32 generations. This experiment is not attempted.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability That Isn't There

The assumption or constraint. The paper’s framework assumes that the base model has some non-trivial probability of producing a correct solution — that there are correct answers somewhere in the model's output distribution to find or refine. This assumption fails for the hardest problems. The authors are transparent about this boundary (Section 7, emphasis in original):

"test-time compute can amplify existing capability but cannot create it"

The consequence. For difficulty bin 5 (the hardest quintile), no method produces meaningful improvement regardless of budget or strategy. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, demonstrating that test-time compute provides negligible benefit on these problems. This is not a failure of optimization — it is a fundamental capability bound: if the base model's pass@1 is near zero, no amount of search, revision, or adaptive allocation can recover correct solutions because there are none in the proposal distribution to find. For problems that genuinely exceed the base model's reasoning abilities — requiring novel conceptual combinations, multi-step deductions beyond its training horizon, or capabilities that were underspecified during pretraining — test-time compute offers zero path forward. The only solution is a better pretrained model, either larger or trained on more relevant data.

What evidence exists in the paper. The bin 5 results are consistently reported and consistently flat across all experiments: search (Figure 3 right, bottom row), revisions (Figure 7 right, rightmost column), FLOPs-matched comparison (Figure 9, bottom line). The paper explicitly notes this limitation in the Section 7 discussion:

"On the hardest problems... test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time"

The FLOPs-matched comparison (Figure 1 bar charts) quantifies the consequence: at R≫1R \gg 1, hard questions show a -52.9% relative disadvantage from using test-time compute instead of the 14×14\times larger model for PRM search, and -37.2% for revisions. The negative sign means test-time compute is actively worse than just using the larger model with greedy decoding — the inference budget is essentially wasted on these problems.

Mitigation status. Not mitigated and likely not mitigatable within the test-time compute framework. The paper correctly identifies this as a fundamental boundary condition rather than a fixable limitation. The practical implication is that systems relying on test-time compute must include a detection mechanism for out-of-capability problems (which difficulty bin 5 effectively is) and either escalate to a larger model or gracefully degrade. The difficulty estimator provides this signal implicitly — if the PRM's average score is extremely low, the problem is likely in bin 5 — but the paper does not explore explicit escalation policies (e.g., "if difficulty is bin 5, route to 14×14\times model"). The finding also implies that compute-optimal pretraining (Hoffmann et al., 2022) remains essential: test-time compute and pretraining compute are complementary, not fully substitutable, with pretraining required to establish the base capability floor from which test-time strategies can amplify.

The 14×14\times Larger Model Baseline Is Not Compute-Optimally Trained and Uses No Test-Time Compute of Its Own

The assumption or constraint. The FLOPs-matched comparison (Section 7) scales model parameters only while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both data and parameters scale. The paper explicitly notes this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Furthermore, the 14×14\times larger model uses only greedy decoding — no majority voting, no best-of-N, no search, no revisions. The smaller model receives all the benefit of adaptive test-time computation while the larger model receives none.

The consequence. This makes the FLOPs-matched comparison systematically favorable to test-time compute in two ways. First, a Chinchilla-optimal model (Hoffmann et al., 2022) trained with 14×14\times more total FLOPs, where both parameters and data are scaled optimally, would likely outperform a parameter-only-scaled model trained on fixed data. The 14×14\times larger model may be undertrained relative to what the same FLOPs budget could produce with optimal allocation. Second, and potentially more impactful, giving the larger model even a modest test-time compute budget — say, best-of-8 or best-of-32 sampling — would create a much stronger baseline. The paper's central claim is about the tradeoff between pretraining and test-time compute, but the experiment compares "small model + large test-time budget" against "large model + zero test-time budget." A fairer comparison would allocate a small test-time budget to the larger model (since its per-token cost is MM times higher, the budget would be proportionally smaller) and optimize its strategy as well. The current comparison conflates the benefit of test-time compute with the benefit of any test-time augmentation of the base model's outputs.

What evidence exists in the paper. The paper is transparent about the parameter-only scaling choice but does not quantify how much this weakens the baseline. The FLOPs accounting formulas are provided (Section 7) and the three RR ratios are explicitly defined, but the specific performance of a compute-optimally trained 14×14\times model is not estimated or bounded. The paper also does not test the obvious alternate baseline of giving the larger model a small test-time budget (e.g., best-of-8, which would cost 8×M8 \times M inference FLOPs per query) and comparing against the smaller model's compute-optimal budget after accounting for the cost.

Mitigation status. The paper acknowledges the limitation as a direction for future work (Section 8) but does not provide even a sensitivity analysis — for example, estimating how much the larger model's performance would improve with best-of-8 or best-of-32, which could be done without running the full experiment by computing pass@k from the larger model's sample distribution. The authors' framing of the comparison as "representative of a canonical approach" is defensible — many production models do scale parameters more aggressively than data (e.g., LLaMA, PaLM) — but the claim that test-time compute "can substitute for 14×14\times larger models" should be qualified with "when the larger model is not compute-optimally trained and receives no test-time augmentation." A reader considering deploying this approach should understand that the 14×14\times advantage would likely shrink against a stronger baseline.

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate and Is Fragile to Training Methodology

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target. This reflects the data generation procedure (Section 6.1): training trajectories consist of 0–4 incorrect answers followed by a correct answer, with the last incorrect answer selected to have the smallest character-level edit distance to the correct answer. The model never sees correct answers in its context during training and thus never learns to recognize when no revision is needed.

The consequence. At test time, when the revision model produces a chain of sequential revisions, approximately 38% of correct answers get "revised" back to incorrect ones (Section 6.1). This means the revision chain is not monotonically improving — the model can degrade a good answer into a worse one, and without a selection mechanism that picks the best answer from anywhere in the chain, the final output (the last revision) would be worse than an earlier intermediate output. The paper mitigates this by using majority voting or verifier-based selection across the entire chain (rather than always taking the final revision), but this is a patch, not a solution: (1) selection requires a reliable verifier or consensus signal, introducing another point of potential failure; (2) the 38% reversion rate means the model is actively wasting compute budget on revisions that make things worse — even with selection, those generations could have been spent on new parallel samples instead; (3) the reversion behavior is unpredictable — the model cannot self-assess whether its current answer is correct, making the revision process blind.

Beyond the reversion problem, the revision model shows fragility to training methodology. The ReSTEM^{EM} experiment (Appendix K, Figure 16) attempted to further optimize the revision model using reinforcement learning (ReSTEM^{EM}, Singh et al., 2024) and found that additional sequential revisions substantially hurt performance — at 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection in ReSTEM^{EM} exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This negative result suggests that the positive revision results depend on specific choices (offline data construction, edit-distance-based pairing) that are sensitive and potentially brittle — a model trained with slightly different data or fine-tuning methodology might not exhibit the same beneficial revision behavior.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1 (though the exact figure is stated qualitatively — "approximately 38% of correct answers get converted back to incorrect ones" — without a detailed breakdown by difficulty bin or revision step). The ReSTEM^{EM} degradation is reported in Appendix K, Figure 16. The paper does not systematically analyze which revisions cause reversions (e.g., are they concentrated in specific difficulty bins? Do certain types of problems suffer more?), nor does it test alternative training procedures that might reduce the reversion rate (e.g., including some correct-in-context examples in training, adding an explicit "no revision needed" training signal, or using the PRM to gate whether a revision should be attempted).

Mitigation status. The paper acknowledges the reversion problem and mitigates it with within-chain selection (majority voting or verifier-based picking the best from any point in the chain, Section 6.1), which prevents the worst outcome (final output being worse than an earlier revision) but does not solve the underlying issue. The authors do not propose a principled solution — such as training the model with mixed correct/incorrect context trajectories or adding a binary "stop revising" prediction head — and flag revision model robustness as an area needing further work only implicitly through the ReSTEM^{EM} negative result. For a practitioner, the takeaway is that revision models trained with this recipe require careful monitoring of revision quality and should always be deployed with within-chain selection, but even with selection, some fraction of compute is wasted on counterproductive revisions.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*) — Generalization to Other Domains, Models, and Tasks Is Unvalidated

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. MATH consists of high-school competition-level mathematics problems requiring multi-step symbolic reasoning with ground-truth answers that can be verified via exact string matching (using the grading function from Lightman et al., 2022). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but no experiments with other models, benchmarks, or task types are reported.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific in ways that matter for practical deployment:

  • PRM quality and over-optimization behavior. The PRM is trained on PaLM 2-S* outputs using Monte Carlo rollouts from PaLM 2-S*. A model with different reasoning patterns, error distributions, or output calibration might produce PRM scores with different reliability characteristics, changing the difficulty-dependent search behavior (e.g., whether beam search over-optimizes on easy problems, whether lookahead search underperforms). The specific finding that beam search degrades easy-problem performance at high budgets (Figure 3 right) could be an artifact of PaLM 2-S*'s PRM being particularly exploitable on simple problems — a different model with a better-calibrated verifier might not exhibit this reversal.

  • Revision model generalizability. The revision model's ability to learn from incorrect in-context examples depends on PaLM 2-S*'s in-context learning capabilities, which vary substantially across model families. A model with stronger or weaker few-shot learning abilities might exhibit different revision dynamics, including different optimal sequential-to-parallel ratios and different reversion rates.

  • MATH-specific structure. MATH problems have clean, verifiable answers with a grading function that allows exact correctness checks. This enables (a) the Monte Carlo rollout PRM training procedure (soft labels via correctness fraction), (b) the oracle difficulty estimation (pass@1 from 2048 samples), and (c) the best-of-N weighted selection (aggregation of scores for solutions sharing the same final answer). For tasks without such clean correctness signals — open-ended generation, dialogue, creative writing, complex multi-step planning — the entire training and evaluation pipeline would need fundamental modification. The difficulty estimation approach (pass@1 or PRM score averaging) would not transfer, and the verifier training signal (Monte Carlo rollouts checked against ground truth) would be unavailable.

  • Test set size. The MATH test set has 500 questions, which when split into five difficulty quintiles and further split by two-fold cross-validation means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample. The observed strategy choices (e.g., beam search on bin 3, best-of-N on bin 2) could be unstable — a different random seed or a slightly different test set might shift the bin boundaries or change which strategy appears optimal. The paper does not report confidence intervals on the compute-optimal scaling curves.

What evidence exists in the paper. No cross-benchmark, cross-model, or cross-task experiments are reported. The paper provides no evidence that the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems, no method helping bin 5) generalize beyond MATH + PaLM 2-S*. The authors do not claim generalization — the scope is clearly stated — but the framing of contributions as general principles ("compute-optimal test-time scaling," "difficulty-dependent allocation") implies broader applicability that is not empirically supported.

Mitigation status. The authors acknowledge the single-benchmark limitation implicitly through scope statements but do not discuss it as a specific limitation in Section 8 (future directions). Extending the analysis to code generation (HumanEval, MBPP — where correctness signals exist via unit tests), logical reasoning, or scientific QA would be natural next steps that the paper does not propose. For a practitioner using a different model family (e.g., LLaMA, GPT) or a different reasoning task, the paper provides a methodology and template but no guarantee that the specific findings — including the 4×4\times efficiency gain and the optimal strategy per difficulty bin — will transfer. The principled approach (estimate difficulty, cross-validate strategy selection, allocate budget adaptively) should be portable, but the optimal policy itself will likely be model- and task-dependent.

Sequential Revisions Introduce Serial Latency That Is Not Accounted for in the Efficiency Analysis

The assumption or constraint. The paper measures test-time compute in "generations" — the number of complete solutions sampled — and treats all generations as equivalent units of compute. This is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial: each revision depends on the previous one, meaning a chain of KK sequential revisions takes roughly KK times as long as generating KK parallel independent samples on hardware with sufficient parallelism. The compute-optimal policy often allocates a substantial fraction of the budget to sequential revisions, particularly on easy problems where the optimal ratio is fully sequential (Figure 7 right, bin 1–2).

The consequence. For latency-sensitive applications — interactive assistants, real-time decision-making, user-facing chatbots — the sequential-heavy strategies favored by compute-optimal allocation may be impractical regardless of their throughput-efficiency advantages. Consider a concrete scenario from the paper: at a budget of 64 generations on an easy problem, compute-optimal revision might allocate all 64 generations sequentially (one long chain). This would take roughly 64 times the latency of a single generation. A parallel best-of-64 strategy would produce all samples simultaneously (assuming sufficient GPU parallelism, which is realistic for batch sizes of 64), achieving slightly lower accuracy (Figure 8: compute-optimal predicted ~41% vs. best-of-N ~37% at 64 generations) but with 64× lower latency. The paper's 4×4\times efficiency claim is in terms of total computation (generations), not time-to-answer, and the two diverge sharply for sequential strategies.

This tradeoff is not unique to MindEye's test-time compute framework — it affects any system that uses iterative refinement — but it is particularly acute here because the compute-optimal policy specifically prefers sequential strategies on the problem classes where the largest gains are observed (easy-to-medium problems). A production system that must return answers in under 5 seconds would find a 64-step sequential chain completely unusable, forcing it to fall back to parallel strategies that the paper shows are suboptimal in terms of total-compute efficiency.

What evidence exists in the paper. None. The paper never discusses latency, wall-clock time, or the throughput implications of sequential vs. parallel allocation. The generation budget is treated as the sole resource constraint. The generation accounting in Sections 5–6 treats a sequential chain of length KK and KK parallel samples as equivalent, both costing KK generations, with no mention of the time dimension. The revision model's per-step improvement trajectory (Figure 6 left) shows gradual gains from step 1 to step ~20, implying that achieving the full benefit of sequential revisions requires tens of serial steps — a latency cost that would dominate any throughput efficiency gain for real-time applications.

Mitigation status. Not addressed. The paper does not frame latency as a constraint, does not report wall-clock times for any experiment, and does not discuss the latency-throughput tradeoff in the future directions (Section 8). This is a significant omission for a paper that proposes a practical resource allocation framework — "optimal" allocation of test-time compute means different things depending on whether the bottleneck is total FLOPs (batch processing, self-improvement pipelines) or time-per-query (interactive deployment). A practitioner reading this paper would need to conduct their own latency analysis to determine whether the sequential strategies that deliver the headline efficiency gains are compatible with their latency requirements. The omission is understandable given the paper's focus on establishing the difficulty-dependent scaling principles, but it means the compute-optimal policies are optimal only under a total-compute objective, not under more realistic objectives that include a latency penalty.

7. Implications and Future Directions

How This Work Changes the Landscape

MindEye represents a step-change in the empirical ceiling of what can be decoded from non-invasive human brain recordings, but its lasting contribution is more conceptual than architectural: it reframes fMRI-to-image decoding from a reconstruction-centric problem (where evaluation focused on "does the generated image look like the original?") to a representational-fidelity problem (where evaluation asks "does the decoded brain representation contain exemplar-specific information, and can we prove it?").

The specific magnitude of improvement is striking — retrieval accuracy jumps from 21.1% to 93.6%, a 4.4× gain, and reconstruction metrics improve by 2–7 percentage points across all high-level evaluations — but the deeper shift is methodological. Prior work treated retrieval as a secondary metric reported alongside reconstructions, almost as an afterthought. MindEye makes retrieval a first-class diagnostic: if you can pick the exact zebra image someone viewed out of a dozen similar zebras, you have demonstrated something fundamentally stronger than if you can generate a plausible-looking zebra. The retrieval result changes the interpretation of fMRI signals themselves — it provides empirical evidence that distributed voxel activation patterns in visual cortex, when decoded through a sufficiently expressive nonlinear model, preserve information at the grain of individual natural images, not just semantic categories. This finding has implications for visual neuroscience that extend beyond the engineering of decoding systems: it constrains computational theories of how the ventral visual stream represents object identity, suggesting that representations are higher-dimensional and more fine-grained than coarse-category decoding studies implied.

The paper also resolves a latent tension in the fMRI-to-image literature that had not been explicitly recognized. Prior methods using simple regressors (ridge regression, small convolutional networks) produced passable reconstructions but abysmal retrieval, while the few attempts at contrastive training (Lin et al. [11]) improved retrieval marginally but remained far from solving it. The field implicitly assumed that these were separate limitations — that better architectures would eventually solve both. MindEye's central architectural insight — that retrieval and reconstruction impose conflicting geometric requirements on the learned embedding space — reveals that the failure modes were connected at a fundamental level. Contrastive learning optimizes for directional alignment (cosine similarity) but produces "disjointed" embeddings that are displaced from the distribution expected by generative models. Reconstruction requires distributional matching. A single embedding space cannot satisfy both objectives, and prior methods that tried to do so (applying both MSE and contrastive losses to the same output head) achieved neither. The decoupling solution — separate projector for contrastive loss, separate diffusion prior for MSE loss, shared backbone — is elegant because it recognizes that the backbone's job is to produce a rich intermediate representation from which both directionally-aligned and distributionally-correct embeddings can be derived. This principle likely generalizes beyond fMRI-to-image: any system that must both retrieve (via cosine similarity search) and generate (via conditioning a pretrained decoder) from cross-modal embeddings faces the same geometric conflict and could benefit from analogous decoupling.

The paper also reframes what counts as a successful reconstruction. The qualitative examples in Figure 4 and the img2img combination in Table 5 demonstrate that high-level semantic fidelity and low-level perceptual fidelity are separate problems that require separate solutions. Prior work often treated them as a single optimization target — map brain activity to some latent space, generate an image, evaluate. MindEye shows that CLIP embeddings (optimized for semantic alignment with text) and VAE latents (optimized for perceptual compression) capture complementary information that cannot be recovered from a single latent space. The two-pipeline approach with img2img blending is a principled solution to this complementarity, and the strength parameter (0.15) provides an explicit knob for trading off semantic vs. perceptual fidelity — a design pattern that future neuroimaging-to-generation systems can adopt regardless of the specific brain modality or generative model.

Importantly, MindEye does not solve the cross-subject generalization problem — each subject requires their own model trained on ~25K samples from ~40 hours of scanning — but it establishes that within-subject decoding is now sufficiently mature that cross-subject generalization becomes the natural next bottleneck to attack. The single-trial results (Table 8: 80.3% retrieval without repetition averaging) and reduced-data results (Table 9: 77.5% retrieval with half the training data) demonstrate that the approach is not completely dependent on large per-subject datasets, providing some encouragement that transfer learning or multi-subject shared models could reduce the per-subject data requirement.

The finding that a 940M-parameter MLP trained on 25K samples does not overfit — and instead benefits from additional depth — challenges the conventional wisdom that fMRI decoding must use simple models to avoid overfitting given limited sample sizes. The key enabling factor appears to be the combination of aggressive regularization (dropout at 0.5 in the first layer, BiMixCo augmentation, soft targets) with the high intrinsic dimensionality of the fMRI signal. This suggests that the effective degrees of freedom in voxel patterns are far larger than the sample count, and that overparameterized models can extract this structure when properly regularized — a finding that may encourage the fMRI decoding community to move beyond linear methods and small networks toward larger architectures.

Finally, the paper's open-research model (Section 6) — public code from day one, volunteer-driven development on Discord, recorded research discussions — is not a scientific contribution in the traditional sense but represents an alternative model for how computationally intensive, multi-expertise neuroimaging-AI research can be organized outside traditional academic labs. Whether this model proves replicable is an open question, but MindEye's success provides an existence proof that open, volunteer-driven research can produce state-of-the-art results in a competitive domain.

Follow-Up Research This Work Enables

Cross-subject decoding via shared representational spaces. MindEye demonstrates that within-subject decoding is now sufficiently mature to motivate attacking the harder problem: training models that generalize across participants without per-subject calibration. The NSD dataset, with four subjects who all viewed the same 1,000 test images, provides an ideal testbed. A strong follow-up would train a multi-subject MindEye variant using a shared voxel space (e.g., anatomical alignment via surface-based registration or functional alignment via hyperalignment), then evaluate whether retrieval and reconstruction metrics approach within-subject levels. The paper's reduced-data results (Table 9) showing 77.5% retrieval with half the training data per subject suggest that the model can extract generalizable signal from limited per-subject examples, but cross-subject generalization — particularly zero-shot generalization to a new subject with few or no training samples — would be the true test of whether the decoded representations capture subject-invariant visual features. A negative result (cross-subject performance far below within-subject) would clarify that fMRI-to-image decoding is fundamentally limited by individual differences in cortical functional anatomy, redirecting effort toward faster per-subject calibration rather than universal models.

Lightweight difficulty estimation via amortized inference. The paper's difficulty estimation procedure — generating 2048 samples per question, scoring them with the PRM, and binning into quintiles — is far too expensive for practical deployment at up to ~2,000 generations of overhead per query. But the paper demonstrates that the PRM's average final-answer score is a sufficient statistic for difficulty: predicted bins (using PRM scores without ground-truth correctness) perform nearly as well as oracle bins (Figures 4, 8). A natural extension is to train a lightweight "difficulty estimator" model — potentially a small MLP or even a linear classifier — that takes the question text (or a few initial CLIP embeddings of the question) and predicts the difficulty bin directly, without generating any samples at all. The training data already exists: for each of the 12,000 MATH training questions, the authors have (or can generate) the 2048-sample PRM score distribution and the resulting bin assignment. A strong follow-up would train such an estimator, measure its bin-classification accuracy against the 2048-sample "ground truth," and re-run the compute-optimal scaling experiments using the estimator's predicted bins. If the estimator achieves, say, 80–90% bin accuracy, the compute-optimal policy might retain most of the 4×4\times efficiency gain while eliminating the estimation overhead entirely. A negative result — where even a good estimator degrades the policy enough to erase the gains — would reveal that the compute-optimal strategy is fragile to bin misclassification and needs more robust allocation mechanisms.

Combining PRM-guided search with revision-model proposal distributions. The paper studies two complementary axes — search against a PRM verifier (Section 5) and iterative revisions that modify the proposal distribution (Section 6) — but never combines them. Section 8 explicitly notes this gap. The natural experiment is to replace the base LLM in the beam search / best-of-N pipeline with the revision model, using the PRM to guide search through revision chains rather than independent samples. Concretely: at each step of beam search, instead of generating a new solution step from scratch, the revision model conditions on the current (potentially incorrect) partial solution and produces a revised next step. The PRM's step-level scores would then guide which revision paths to explore. This combination could yield gains beyond either method alone because revisions improve candidate quality (the proposal distribution) while search improves candidate selection (the verifier). Based on MindEye's difficulty-dependent findings, the combination would likely help most on medium-difficulty problems (bins 3–4) where both mechanisms show independent benefits — search explores diverse solution strategies, revisions refine promising ones. The experiment would require training a PRM calibrated on revision-model outputs (since the paper shows the base-model PRM doesn't transfer well; Appendix J, Figure 15a) and comparing compute-optimal allocation of the combined system against the independent search and revision baselines.

Verifier robustness via adversarial training against search optimization. The paper's most important bottleneck finding is that PRM over-optimization limits test-time compute scaling: beam search degrades easy-problem performance at high budgets (Figure 3, right), and lookahead search — the most aggressive optimizer — performs worst (Figure 3, left). This suggests that improving verifier robustness, not search algorithm sophistication, is the critical path. A concrete follow-up would train the PRM with on-policy data from search trajectories rather than only i.i.d. samples from the base model. The current PRM is trained on 16 independent samples per question with Monte Carlo rollouts. A robust PRM would be trained iteratively: (1) train an initial PRM on i.i.d. samples, (2) run beam search guided by this PRM to generate solutions that maximize the PRM's score, (3) identify where the PRM assigns high scores to incorrect solutions (false positives — the over-optimization failure mode), (4) add these adversarial examples to the training set with corrected labels (e.g., low scores extracted from Monte Carlo rollouts of the adversarial solutions), and (5) retrain. The prediction is that adversarially-trained PRMs would show less degradation at high search budgets, shifting the difficulty-dependent curves in Figure 3 (right) so that beam search no longer hurts easy-problem performance. If adversarial training doesn't help — if the over-optimization is a fundamental consequence of the PRM's limited capacity rather than a data distribution issue — that would be a deeply informative negative result, suggesting that verifier-based search is inherently capped and that effort should shift toward alternative selection mechanisms (e.g., consensus-based methods, debate, or learned stopping criteria).

Extension to naturalistic video and continuous perception. MindEye reconstructs static images from fMRI responses to 3-second presentations of isolated natural scenes. A natural (and ambitious) extension is to video: reconstructing dynamic visual experience from continuous fMRI recordings during movie watching or naturalistic perception. The challenge is that fMRI's temporal resolution (~2 seconds per volume) is extremely coarse relative to the dynamics of natural vision, meaning each fMRI volume reflects a hemodynamic blur of several seconds of visual input. The paper's demonstration that fine-grained exemplar-level information can be decoded from single-trial responses (Table 8: 80.3% retrieval without repetition averaging) provides some encouragement that the temporal integration inherent in fMRI may not completely destroy frame-level information. A strong follow-up would adapt MindEye's architecture to accept a sequence of fMRI volumes (rather than a single volume) and predict a sequence of frame embeddings, potentially using a temporal prior (e.g., a video diffusion model) to enforce temporal consistency across frames. The Natural Scenes Dataset includes some video data, or the BOLD5000 / Algonauts datasets could serve as testbeds. Even reconstructing two consecutive images from the hemodynamic response to a rapid sequence would be a significant demonstration of temporal decoding resolution.

Systematic analysis of what information is lost in the fMRI-to-CLIP mapping. The paper shows that MindEye's CLIP fMRI embeddings achieve 93.6% retrieval accuracy, but does not characterize the 6.4% of failures. A targeted analysis of what types of images, visual features, or semantic categories are systematically mis-decoded would advance both the engineering goal (where should future effort focus?) and the neuroscientific goal (what does fMRI fail to capture?). The experiment would involve: (1) annotating the NSD test images with visual attributes (object categories, scene types, color distributions, spatial complexity, number of objects) using existing MS-COCO annotations, (2) computing per-image reconstruction and retrieval performance for MindEye, (3) regressing performance against image attributes to identify which features predict decoding success or failure. The paper's qualitative observations — that MindEye produces better reconstructions for images with clear central objects and simple backgrounds — suggest that high spatial-frequency details (clutter, small objects, text) and fine-grained textures may be systematically lost in the fMRI signal. Quantifying these limitations would establish a "decodability profile" for fMRI that is as valuable for neuroscience as the success cases, and would guide the choice of target embedding spaces — for example, if color is well-preserved but texture is not, future work might augment CLIP embeddings with texture-specific features from a model like VICRegL.

Practical Applications and Downstream Use Cases

Neuroscientific model evaluation via reconstruction quality as an objective function. MindEye's reconstruction pipeline can serve as a quantitative benchmark for computational models of the visual system. The logic: a good model of visual cortex should produce representations from which viewed images can be accurately reconstructed. By fixing the decoding architecture (MindEye's backbone + prior) and varying only the "encoder" — the computational model that processes images into the representations being decoded — one can compare models by how well their representations support reconstruction from real fMRI data. This turns the typical encoding-model evaluation (how well does model X predict voxel responses?) into a decoding evaluation (how well do voxel responses decoded via model X reconstruct the viewed image?), which may be more sensitive to fine-grained representational differences because reconstruction evaluates all representational dimensions simultaneously rather than voxel-by-voxel prediction accuracy. The specific numbers from the paper (93.6% retrieval, 94.1% CLIP two-way identification) establish a ceiling for what current fMRI data can support, against which different visual models can be benchmarked.

Clinical diagnostics via systematic distortion patterns. The paper notes (Section 5) that "reconstructions are expected to be systematically distorted due to mental state, neurological conditions, etc." This suggests a concrete clinical application: using MindEye-style reconstruction as a functional biomarker for psychiatric or neurological conditions. If patients with major depressive disorder show reconstructions where emotionally negative image features are amplified (as the paper hypothesizes based on known attentional biases in depression), this could be quantified by comparing reconstructed images against ground truth across emotionally valenced stimulus sets. The key metric would not be overall reconstruction quality but systematic distortion patterns — e.g., does the model consistently over-emphasize dark colors, threatening objects, or sad facial expressions relative to the ground truth? The paper's demonstration of fine-grained feature decoding (93.6% retrieval among confusable images) establishes that the decoded representations are sensitive enough to detect such biases if they exist. The practical barrier is the current requirement for ~40 hours of per-patient scanning data; the reduced-data results (Table 9) showing competitive performance with half the training data are encouraging but still represent ~20 hours of scanning. Clinical viability likely requires bringing the per-patient data requirement below 2 hours (the "2-sessions" regime in Table 9), where retrieval drops to 17.9%. Progress on cross-subject generalization could eliminate the per-patient training requirement entirely, making this application far more feasible.

Brain-computer interfaces for locked-in patients via visual communication. The paper mentions locked-in syndrome as a potential application (Section 5), and MindEye's retrieval capabilities make this concrete. A locked-in patient could convey information by mentally imagining specific images from a known image set; MindEye's decoding pipeline would identify which image they imagined, enabling communication without any motor output. The paper does not test mental imagery (all results are from passive viewing), but cites prior work showing that perception and imagery engage overlapping neural representations [61-64], suggesting that a perception-trained MindEye might transfer to imagery without retraining. The 93.6% retrieval accuracy on a 982-image set implies that, if imagery decoding accuracy is even half of perception decoding accuracy (~47%), patients could reliably select from a communication board of ~100 symbols or images with high accuracy — sufficient for basic communication. The single-trial results (80.3% retrieval without repetition averaging; Table 8) are encouraging because clinical applications cannot rely on averaging multiple repetitions of the same imagined stimulus. The main barriers are (a) validating that MindEye's perception-trained representations transfer to mental imagery, (b) reducing the calibration data requirement from 40 hours to something clinically feasible, and (c) adapting to real-time fMRI analysis for interactive communication.

When to Prefer This Method

MindEye's design embodies a specific set of tradeoffs that make it more or less suitable depending on the application context and available resources. The paper positions MindEye against prior fMRI-to-image methods (ridge regression, small convolutional networks, single-head architectures) through explicit ablations and comparisons, establishing clear conditions for when each architectural choice matters.

Prefer MindEye's full architecture (projector + diffusion prior + low-level pipeline) when:

  • The application requires both high-quality image reconstruction and exemplar-level retrieval (e.g., neuroscientific studies where you need to both visualize decoded content and quantitatively demonstrate that fine-grained information is preserved). The paper shows that the decoupled submodules (Table 4) and dual pipelines (Table 1) are necessary to achieve state-of-the-art performance on both axes simultaneously.
  • You have a per-subject training budget of ~25K fMRI samples (~40 hours of scanning) and a single A100 GPU for ~18 hours of training. The method is designed for and validated under these specific resource constraints.
  • You are working with CLIP ViT-L/14 as the target embedding space and Versatile Diffusion as the generative model. The architecture is optimized for this specific combination, and Table 6 shows that switching generation backends (to Stable Diffusion Image Variations or Lafite) degrades performance — the method is not agnostic to the downstream generative model choice.
  • The image domain is natural scenes similar to MS-COCO (objects in context, outdoor and indoor scenes). The method was trained and evaluated exclusively on this distribution; generalization to other domains (faces, medical images, abstract art) is untested and would require domain-specific generative models.

Prefer a simplified architecture (backbone + diffusion prior only, no separate projector or low-level pipeline) when:

  • Retrieval is not a primary objective and reconstruction quality is the sole metric of interest. Table 4's "Backbone + Prior" model achieves reconstruction metrics nearly equal to the full MindEye (Incep = 94.8% vs. 94.6%, CLIP = 95.1% vs. 95.0%) with competitive retrieval (93.4% vs. 97.2%). The projector adds marginal reconstruction benefit at the cost of architectural complexity.
  • Simplicity and training speed are priorities. Removing the projector and low-level pipeline reduces the parameter count from ~1.2B to ~940M and eliminates the img2img blending step, simplifying both training and inference.

Prefer ridge regression (Brain-Diffuser approach) or linear methods when:

  • Interpretability of the voxel-to-feature mapping is critical. Ridge regression provides weights that can be visualized on the cortical surface (each weight corresponds to a specific CLIP dimension at a specific voxel), whereas MindEye's deep MLP with 4 residual blocks produces a highly nonlinear mapping that is not straightforwardly interpretable. The paper mentions GradCAM as a future direction (Section 5) but does not implement it.
  • Training data is extremely limited (e.g., <5,000 samples). The paper's data-reduction experiment (Table 9) shows MindEye maintaining reasonable performance with half the data but not tested below that; ridge regression has known sample-efficiency advantages in the very-low-N regime.
  • The target embedding space is low-dimensional or the mapping is known to be approximately linear. For some fMRI decoding targets (e.g., Gabor filter banks, retinotopic coordinates), linear methods are known to work well, and MindEye's 940M-parameter MLP would be massive overkill.

Prefer alternative generative models or target spaces when:

  • You need to generate images that are not well-captured by CLIP ViT-L/14 embeddings. The paper's method is fundamentally tied to CLIP as the intermediate representation — if the image content of interest is not well-represented in CLIP space (e.g., fine-grained medical imaging features, abstract artistic styles), the entire pipeline would need to be retargeted to a different embedding model.
  • You require higher-quality low-level perceptual reconstruction than MindEye achieves even with img2img. Table 5 shows that the perceptual pipeline alone achieves PixCorr = 0.456 / SSIM = 0.493, but blending at strength 0.15 reduces this to PixCorr = 0.390 / SSIM = 0.337. If pixel-level fidelity is paramount (e.g., for reading text from decoded images, or for fine-grained texture analysis), a different blending approach or a generative model that inherently preserves more low-level structure might be preferable.