V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

ArXiv: 2603.14482

🎯 Pitch

Simply extending a masked-prediction objective across all tokens—not just the hidden ones—forces self-supervised video models to develop spatially coherent, object-level features that unlock state-of-the-art depth estimation, action anticipation, and even real-robot grasping from a frozen backbone. This single design change yields a 52% jump in dense tracking quality while preserving top-tier action recognition, proving that dense and global understanding are not in conflict.

---

1. Executive Summary

This paper introduces V-JEPA 2.1, a family of self-supervised video and image models that learns dense, high-quality representations while retaining strong global scene understanding, built on a core architectural insight: extending the predictive loss to all tokens—both masked and visible context—via a Dense Predictive Loss (applying a distance-weighted L1 loss on context tokens alongside the standard masked-token prediction) and Deep Self-Supervision (applying the objective hierarchically at multiple intermediate encoder layers). On the MATH of video understanding—a suite spanning Ego4D short-term anticipation, EPIC-KITCHENS action anticipation, Something-Something-v2 action recognition, NYUv2 depth estimation, and real-robot manipulation—V-JEPA 2.1 ViT-G (2B parameters) achieves state-of-the-art frozen-backbone results: 7.71 mAP on Ego4D STA (a 35% relative improvement over the previous best task-specific method), 40.8 Recall@5 on EPIC-KITCHENS action anticipation, 77.7% accuracy on SSv2, 0.307 RMSE on NYUv2 depth estimation (surpassing a 7B-parameter DINOv3), and a 20% improvement in real-robot grasping success rate over V-JEPA 2, establishing that dense spatio-temporal features emerge from JEPA-style predictive pretraining only when the self-supervision signal is distributed across both masked and unmasked regions rather than concentrated solely on masked tokens.

2. Context and Motivation

The Core Problem: Dense Features and Global Understanding Are Opposing Forces in SSL

The fundamental challenge this paper addresses is deceptively simple to state but difficult to solve: can a single self-supervised learning objective produce visual representations that excel simultaneously at dense spatial tasks (depth estimation, semantic segmentation, object tracking) and global semantic tasks (action recognition, video classification)? The evidence prior to this work suggested the answer was "not with current methods."

This matters because real-world visual intelligence—whether in autonomous agents, robotic manipulation, or video understanding systems—demands both capabilities from a single representation. A robot needs to know what object it's looking at (global semantics) and exactly where that object is in 3D space (dense geometry). A video understanding system needs to recognize which action is being performed (global recognition) while tracking which objects are involved and how they move (dense correspondence). Forcing practitioners to choose between dense-feature models and global-understanding models—or to run two separate encoders—is computationally wasteful and architecturally unsatisfying. A unified representation would enable downstream systems to be simpler and more efficient.

The gap is not just practical but theoretical. It exposes a tension in how self-supervised objectives shape the information content of learned representations: objectives that produce strong global features tend to discard local spatial structure, while objectives that preserve fine-grained spatial detail often compromise the model's ability to aggregate global context. Understanding why this happens and how to resolve it is a fundamental question about representation learning.

The Two Camps: JEPA-Based Video Models vs. Image-Based Dense Feature Models

Prior to V-JEPA 2.1, the landscape of self-supervised visual representation learning had bifurcated into two distinct camps with complementary strengths and weaknesses.

Camp 1: V-JEPA and Predictive Video Models (Strong Globally, Weak Locally)

The V-JEPA family (Bardes et al., 2024; Assran et al., 2025) established itself as a leading approach for learning video representations through Joint-Embedding Predictive Architectures (LeCun, 2022). The core idea is elegant: take a video, mask out some spatio-temporal patches, encode the visible context, and train a predictor to reconstruct the masked regions' representations in a learned latent space—not pixel space. This "mask-denoising in representation space" forces the model to learn about motion, dynamics, and temporal causality, since predicting what comes next requires understanding how objects move and interact.

V-JEPA 2 (Assran et al., 2025) demonstrated the power of this approach: it achieved state-of-the-art results on video understanding benchmarks that demand temporal reasoning, such as Something-Something-v2 (77.3% top-1 accuracy), EPIC-KITCHENS action anticipation (39.7 Recall@5), and Diving-48 (90.2%). More impressively, it showed that representations learned through this predictive objective could serve as the backbone for world models that enable robotic planning and control—the model could predict future visual states in its learned representation space, and these predictions could guide actions.

However, as the paper explicitly demonstrates through visualizations in Figure 1 and quantitative analysis in Table 1, V-JEPA 2's representations are spatially impoverished. When the authors compute PCA on V-JEPA 2 encoder features and map the top three components to RGB channels, the resulting visualizations show "noisy and fragmented local spatial structure" rather than coherent object-level groupings. On dense downstream tasks evaluated with linear probes—the simplest possible readout that directly tests feature quality rather than probe capacity—V-JEPA 2 achieves only:

• 22.2 mIoU on ADE20K semantic segmentation (compared to 49.5 for DINOv2 ViT-g)
• 0.642 RMSE on NYUv2 depth estimation (compared to 0.372 for DINOv2 ViT-g)
• 24.4 mIoU on ADE20K in the full comparisons of Table 8

These numbers reveal a startling gap: V-JEPA 2's features, excellent as they are for action recognition and temporal prediction, are nearly unusable for tasks requiring spatial localization. The features capture what is happening but not where.

Camp 2: DINO and Image-Based Models (Strong Locally, Weak Globally on Video)

On the other side, the DINO family (Caron et al., 2021; Oquab et al., 2023; Siméoni et al., 2025) pursued a different objective: learning representations by enforcing invariance to data augmentations. DINOv2 (Oquab et al., 2023) showed that training a student network to match a teacher network's output on differently augmented views of the same image produces features with remarkable spatial structure. PCA on DINOv2 features reveals clear object boundaries, semantic part groupings, and depth-like organization—properties that make these features directly useful for dense prediction tasks like segmentation and depth estimation without task-specific architectural modifications.

DINOv3 (Siméoni et al., 2025) pushed this further, achieving 55.9 mIoU on ADE20K semantic segmentation and 0.309 RMSE on NYUv2 depth estimation with a 7B-parameter model—results that demonstrate how far image-based SSL can go on dense tasks.

But these models have a critical limitation: they are image-based. They learn from static frames, which means they lack any inductive bias or training signal for understanding temporal dynamics, object motion, or action causality. When evaluated on video understanding tasks like Something-Something-v2—which requires recognizing actions like "pushing something from left to right" that cannot be distinguished from static appearance alone—image-based models perform substantially worse. DINOv3 ViT-7B achieves only 70.1% on SSv2 compared to V-JEPA 2's 77.3%. DINOv2 ViT-g achieves just 50.7%—barely above chance for many action classes. For tasks requiring genuine temporal understanding, these models are fundamentally limited because they were never trained with temporal information.

The Diagnosis: Why V-JEPA Features Lack Spatial Structure

The paper's first contribution is a clear mechanistic diagnosis of why V-JEPA 2 fails at dense prediction—a diagnosis that goes beyond the surface-level observation and points toward a specific fix.

The key insight comes from examining the V-JEPA 2 objective function (Equation 1). The loss is applied only to masked tokens, not to the visible context tokens:

$\mathcal{L}_{\text{predict}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \|P_\phi(E_\theta(x), \Delta_y)_i - \text{sg}(E_{\bar{\theta}}(y)_i)\|_1$

where $\mathcal{M}$ is the set of masked patch indices, $E_\theta(x)$ processes the masked video (producing context tokens for visible patches), $\Delta_y$ are learnable mask tokens carrying positional information about the masked patches, and $P_\phi$ is a predictor that processes the concatenated sequence of context and mask tokens. The target is the representation of the unmasked video $y$ produced by an exponential moving average (EMA) of the encoder, with a stop-gradient operator preventing representation collapse.

The predictor outputs tokens for both context positions and mask positions, but the loss only supervises the mask-position outputs. This asymmetry creates a structural incentive: context tokens are free to become purely "global aggregators" that pool information from the entire visible scene to help the predictor reconstruct masked regions. Since context tokens are never directly supervised to preserve their own content, there is no gradient signal encouraging them to maintain fine-grained spatial information about their specific patch location. They can—and empirically do—collapse into something functionally similar to the register tokens identified by Darcet et al. (2023): global summary vectors that discard local detail in favor of holistic scene representation.

This hypothesis is supported by the authors' ablation (Table 1, Figure 3): when they add a context loss $\mathcal{L}_{\text{ctx}}$ that also supervises context token predictions against the target encoder's corresponding context-token representations, the feature maps qualitatively transform from "noisy and fragmented" to "smooth, coherent, and semantically organized." Quantitatively, ADE20K mIoU jumps from 22.2 to 33.8 just from adding this loss, and NYUv2 RMSE drops from 0.682 to 0.474. The mere act of making context tokens accountable for their own content unlocks spatial structure that was latent in the architecture but suppressed by the asymmetric training objective.

The Trade-off That Emerges

However, the paper reveals an immediate complication: naive application of the context loss degrades global understanding. As shown in Table 1, adding the context loss (with fixed weighting) causes SSv2 accuracy to drop from 72.8% to 62.5% and ImageNet accuracy to dip from 82.2% to 72.6%. The model faces a gradient conflict: the prediction loss pushes context tokens to aggregate global information to help reconstruct masked regions, while the context loss pushes them to preserve local patch identity. If the context loss is too strong, the context tokens become overly local, losing the global receptive field that the predictor needs to make accurate masked-region predictions.

This trade-off is the central tension the paper must resolve: how do you get dense spatial structure without sacrificing the global temporal understanding that makes video models valuable? It's not enough to just add a context loss—you need a careful design that balances these competing pressures.

Missing Pieces in Prior Work

Beyond the specific V-JEPA diagnosis, the paper identifies several broader gaps in existing approaches:

1. Joint image-video training was suboptimal. V-JEPA 2 trained jointly on images and videos but did so by naively duplicating images temporally to create 16-frame "static videos" and processing them through the same 3D convolutional patch embedder. This is computationally wasteful—it increases the cost of processing images by 16×—and semantically incorrect, as it treats static images as a degenerate case of video rather than recognizing them as a distinct modality with different statistical structure. Images have strong single-frame appearance cues but no motion; videos have motion but often at lower per-frame quality. A model that conflates these modalities loses the ability to specialize its processing.

2. Supervised evaluation protocols were fragmented. Prior work on video SSL often evaluated on narrow subsets of tasks— V-JEPA 2 focused heavily on action recognition and anticipation, DINO models on image segmentation and depth—making it impossible to assess whether a single model could be strong across the full spectrum. There was no unified benchmark or protocol establishing what "good performance everywhere" would look like.

3. The role of intermediate layers was underexplored. Most SSL methods, including V-JEPA 2, apply their loss only at the final encoder output. But vision transformers develop representations hierarchically: early layers capture low-level textures and edges, middle layers capture object parts, and later layers capture semantic categories. A loss applied only to the final layer can only indirectly shape earlier representations through backpropagation. Prior work like iBOT (Zhou et al., 2021) and Perception Encoder (Bolya et al., 2025) had shown that supervising intermediate layers improves representation quality, but this idea had not been tested in the JEPA framework.

4. Scaling behavior for video SSL was poorly characterized. While DINOv2 and DINOv3 had demonstrated the benefits of scaling image-based SSL to billion-parameter models and hundred-million-image datasets, the scaling behavior of video SSL—particularly JEPA-based methods—remained largely unknown. It was unclear whether the gains from scaling model size and data diversity that worked for image models would transfer to video models, especially given the added complexity of spatio-temporal learning.

5. No model simultaneously achieved state-of-the-art on both dense and global tasks. Perhaps most tellingly, looking at the prior landscape: the best video understanding models (V-JEPA 2, InternVideo2) performed poorly on dense tasks; the best dense feature models (DINOv3, PEspatial) fell short on temporal understanding tasks. There was no model that was competitive on both Ego4D STA and NYUv2 depth estimation, or on both SSv2 action recognition and ADE20K semantic segmentation. This gap wasn't just a matter of optimization—it reflected a fundamental architectural tension that V-JEPA 2.1 is designed to resolve.

How This Paper Positions Itself

V-JEPA 2.1 positions itself not as a rejection of the V-JEPA approach but as a correction and extension that preserves its strengths while filling its most glaring weakness. The authors don't abandon the JEPA framework or the mask-denoising objective; instead, they show that a small set of principled modifications transforms V-JEPA from a specialist (strong at global video understanding, weak at spatial tasks) into a generalist that excels across the board.

The paper frames its contributions around four "key ingredients" that directly address the diagnosed problems:

• Dense Predictive Loss (Section 2.3.1): Extending the prediction loss to context tokens with a carefully designed distance-weighted schedule resolves the gradient conflict between local and global information. Context tokens near masked regions receive higher weight ($\lambda_i = \frac{\lambda}{\sqrt{d_{\min}(i, \mathcal{M})}}$), enforcing local continuity where prediction needs it most while allowing distant context tokens more flexibility to aggregate global information.

• Deep Self-Supervision (Section 2.3.2): Applying the loss at multiple intermediate encoder layers ensures that the gradient signal shapes representations throughout the network hierarchy, not just at the output. This is what allows the model to recover global understanding performance that the naive context loss degrades, and it means downstream tasks can use only the final layer (avoiding the complexity and memory cost of multi-layer feature extraction) without sacrificing accuracy.

• Multi-Modal Tokenizer (Section 2.3.4): Separate 2D (for images) and 3D (for videos) patch embedding layers, plus learnable modality tokens, allow the model to process each modality natively rather than force-fitting images into a video pipeline. This is both more computationally efficient and representationally cleaner.

• Data and Model Scaling (Sections 2.3.3, 2.3.5): Expanding the image dataset from 1M (ImageNet) to 142M (LVD-142M) curated images and scaling from ViT-g (1B) to ViT-G (2B) with a high-resolution cooldown phase demonstrates that these design improvements compound with scale.

The paper's empirical positioning is ambitious but carefully bounded: it doesn't claim to surpass specialized models on every benchmark—DINOv3 ViT-7B still holds the edge on ADE20K segmentation (55.9 vs. 47.9 mIoU) and DAVIS video object segmentation (71.1 vs. 69.0 J&F)—but it claims to be the first model that is genuinely competitive across the full spectrum, achieving state-of-the-art on NYUv2 depth, Ego4D STA, EPIC-KITCHENS action anticipation, and SSv2 action recognition simultaneously. The figure summarizing this (Figure 2) shows relative improvements over V-JEPA 2 ranging from +31% (image classification) to +96% (semantic segmentation), with the model equaling or surpassing prior state-of-the-art on 7 of 8 benchmarks shown.

Connection to the Executive Summary

The executive summary states that V-JEPA 2.1 achieves state-of-the-art results across dense and global tasks through a Dense Predictive Loss and Deep Self-Supervision. But understanding why these innovations matter requires the context established here: (1) that prior V-JEPA models were fundamentally asymmetric in their training objective, creating no incentive for context tokens to preserve spatial structure; (2) that fixing this asymmetry creates a gradient conflict with global understanding that must be carefully managed; (3) that the distance-weighted loss and multi-level supervision are not arbitrary design choices but precise solutions to this specific conflict; and (4) that the broader problem of unifying dense and global representations in a single model was unsolved prior to this work, with the field split between video-specialist and image-specialist approaches that excelled in complementary regimes.

3. Technical Approach

3.1 Reader Orientation

V-JEPA 2.1 is a self-supervised training recipe that produces a Vision Transformer (ViT) encoder whose output representations are simultaneously good at dense spatial tasks (knowing where each object is, pixel by pixel) and global semantic tasks (knowing what action or object category is present). The problem it solves is that prior video SSL models—including V-JEPA 2—learned features that captured high-level dynamics and motion but discarded fine-grained spatial structure, making them nearly unusable for depth estimation, segmentation, or object tracking. The "shape" of the solution is a careful modification of the loss function and architecture that redistributes the self-supervision signal: instead of training only on masked-patch predictions, V-JEPA 2.1 also trains on context (visible) patch predictions using a distance-weighted scheme, and applies this loss at multiple intermediate encoder layers rather than only at the final output.

3.2 Big-Picture Architecture (Diagram in Words)

The V-JEPA 2.1 system has six major components arranged in a standard JEPA-style two-branch pipeline:

1. Modality-Specific Patch Embedding (patchifier) — Converts raw images or video clips into sequences of token vectors. Images use a 2D convolution (16×16 patches); videos use a 3D convolution (16×16×2 spatio-temporal patches). A learnable modality embedding token is added to tell the encoder whether this is an image or video input.

2. Masking Module — Randomly drops a large fraction of spatio-temporal patches from the input sequence. Only the remaining visible context tokens are passed to the x-encoder. The masking strategy uses contiguous spatial blocks and whole temporal tubelets, with spatial mask scales drawn from [0.15, 0.7] (meaning 15–70% of patches can be visible, i.e., 30–85% can be masked) and the temporal mask scale fixed at [1.0, 1.0] (entire tubelets are kept or dropped together).

3. X-Encoder (student encoder) — Processes only the visible context tokens and outputs representations at four intermediate layers in addition to the final layer. These multi-level representations are concatenated along the channel dimension and compressed by a lightweight MLP before being passed to the predictor. This produces a multi-level fused context representation.

4. Y-Encoder (teacher/target encoder) — An Exponential Moving Average (EMA) copy of the x-encoder that processes the unmasked input (all patches visible) and produces target representations at the same four intermediate levels. The stop-gradient operator prevents gradients from flowing through this branch, which is the standard mechanism for avoiding representation collapse in non-contrastive SSL.

5. Predictor Network — Takes the concatenated sequence of compressed multi-level context tokens and learnable mask tokens (with spatio-temporal positional encoding) as input. It has 24 transformer blocks and an embedding dimension of 384. It produces four output sequences corresponding to the four encoder levels. Crucially, it outputs predictions for every input position—both context positions and mask positions.

6. Two-Part Loss Function — Computes a combined loss with two terms: (a) the standard V-JEPA prediction loss on masked-position outputs (L_predict, an L1 loss against the y-encoder's target representations), and (b) a dense context loss on context-position outputs (L_ctx, a distance-weighted L1 loss—Equation 2 —also against the y-encoder's targets). Both losses are computed at all four encoder levels. The total loss L_dense = L_predict + L_ctx is backpropagated through the predictor and x-encoder; the y-encoder is updated only via EMA.

Information flows: raw image/video → modality-specific patch embedding → masking (drop patches) → x-encoder with multi-level output → MLP compression → concatenation with learnable mask tokens → predictor (produces four-level predictions) → two-part loss computed against y-encoder's four-level targets on unmasked input.

3.3 Roadmap for the Deep Dive

The explanation order is chosen to build from the mathematical formulation outward to the training recipe:

1. The JEPA framework and the original V-JEPA 2 objective — we need to understand the baseline before we can understand what changed. This covers the prediction loss (Equation 1), the stop-gradient/EMA mechanism, and why the original loss was mask-only.

2. The Dense Predictive Loss — the paper's core innovation. We'll walk through the context loss formulation (Equation 2), the distance-weighting scheme (Equation 3), the warmup schedule for the weighting coefficient, and the empirical trade-off between dense and global performance that this loss manages.

3. Deep Self-Supervision — how applying the loss at four intermediate encoder levels works, the concatenation+MLP fusion mechanism, and why this recovers the global understanding performance that the naive context loss degrades.

4. Multi-Modal Tokenizer — the separate 2D/3D patch embeddings, the learnable modality token, and the distributed training setup that mixes images and videos without forcing images into a video pipeline.

5. Training Recipe and Hyperparameters — the two-phase schedule (primary + cooldown), data curation (VisionMix163M), masking strategy, batch sizes, learning rates, EMA coefficient, and all numbers quoted verbatim from the paper.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methods paper whose core idea is a set of four algorithmic modifications to the V-JEPA training recipe that collectively transform spatially impoverished video representations into dense, high-quality features while preserving global understanding. The modifications are not independent—the context loss creates a gradient conflict that deep self-supervision resolves, the multi-modal tokenizer enables efficient joint image-video training that the data scaling leverages, and model scaling amplifies the gains from all three.

---

The JEPA Framework and Original V-JEPA 2 Objective

What JEPA does. A Joint-Embedding Predictive Architecture learns representations by predicting one view of data from another view in a learned latent space—not in the raw input space. The fundamental intuition (LeCun, 2022) is that predicting in pixel space is wasteful and brittling because the model must expend capacity modeling irrelevant low-level detail (exact pixel intensities, texture noise). Predicting in a learned representation space lets the model focus on the information that matters for downstream tasks—semantic content, object identity, motion.

The two-branch structure. The JEPA architecture has two encoder branches processing two views of the same underlying data:

• The x-encoder $E_\theta(\cdot)$ processes a corrupted/noisy view $x$ of the input (in V-JEPA, corruption comes from masking: dropping patches).
• The y-encoder $E_{\bar{\theta}}(\cdot)$ processes an uncorrupted/clean view $y$ of the same input. Its parameters $\bar{\theta}$ are an Exponential Moving Average (EMA) of the x-encoder's parameters $\theta$: after each training step, $\bar{\theta} \leftarrow \beta \bar{\theta} + (1 - \beta) \theta$ with $\beta = 0.99925$.

The y-encoder produces targets—representations that the x-encoder's output should match—and a stop-gradient operator $\text{sg}(\cdot)$ prevents any gradient from flowing through the y-encoder. This stop-gradient + EMA combination is the standard recipe for avoiding representation collapse without needing explicit regularization terms. Without stop-gradient, the system would find the trivial solution where both encoders output constant vectors (or identical representations for all inputs), which achieves zero loss but is uninformative.

The predictor. A separate predictor network $P_\phi(\cdot)$ sits between the two encoders. It takes the x-encoder's output representations and transforms them to match the y-encoder's target representations. The predictor is the only component that sees both the x-encoder output and the target—the x-encoder itself never directly compares to the y-encoder. This asymmetric design is what prevents collapse: the x-encoder must produce representations that are predictable (the predictor can map them to the targets), but the predictor absorbs any representation-to-target mismatch, preventing the x-encoder from simply copying the y-encoder.

Masking in V-JEPA. The V-JEPA family applies masking as the corruption mechanism. For video inputs, some spatio-temporal patches are completely removed from the input sequence before the x-encoder processes it. The remaining visible patches are called context tokens. The y-encoder, in contrast, processes the full unmasked video and produces a target representation for every patch position.

The masking strategy has specific parameters (Appendix A, Table 11):

• Spatial mask scale: sampled uniformly from [0.15, 0.7]. This is the fraction of the spatial grid that remains visible—so 30–85% of spatial patches can be masked.
• Temporal mask scale: fixed at [1.0, 1.0]. This means masking operates on entire temporal tubelets: a given spatial position is either fully masked across all frames or fully visible across all frames.
• Mask aspect ratio: sampled from [0.75, 1.5], controlling the shape of contiguous mask blocks.
• Tubelet size: 2. Video patches are grouped into tubelets of 2 consecutive frames; masking operates at the tubelet level so that a single spatial location across two adjacent frames is kept or dropped together.

Masks are contiguous spatial blocks rather than random independent patches—this ensures the model must learn to inpaint coherent regions rather than interpolate between visible neighbors, which would be a much easier and less informative task.

The mask tokens. After the x-encoder processes the visible context tokens, the resulting context representations are concatenated along the sequence axis with a set of learnable mask tokens $\Delta_y$. Each mask token is a learnable embedding vector that carries spatio-temporal positional encoding specifying which masked patch position it represents. These mask tokens serve as queries that tell the predictor: "predict what the encoder would have produced at this spatio-temporal location."

The original V-JEPA 2 loss. The original loss (Equation 1, from Assran et al., 2025) is:

$\mathcal{L}_{\text{predict}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \|P_\phi(E_\theta(x), \Delta_y)_i - \text{sg}(E_{\bar{\theta}}(y)_i)\|_1$

where $\mathcal{M}$ is the set of masked patch indices, $P_\phi(E_\theta(x), \Delta_y)_i$ is the predictor's output at position $i$ (which corresponds to a mask-token position), $\text{sg}(E_{\bar{\theta}}(y)_i)$ is the y-encoder's target representation at position $i$ with gradients stopped, and $\|\cdot\|_1$ is the L1 (absolute error) norm.

What it computes: for every masked position, take the predictor's prediction of what the y-encoder's representation should be at that position, compute the absolute difference from the actual y-encoder representation, and average over all masked positions. This is a per-position L1 regression loss in the latent space.

Why this form: L1 loss is used rather than L2 (MSE) because L1 is more robust to outliers and produces sparser gradients—when the prediction is far from the target, the gradient magnitude is constant (just the sign of the error), whereas L2's gradient grows linearly with the error, which can cause instability. The loss is applied only to masked positions because the motivation is mask-denoising: given the visible context, can you reconstruct what was removed? The context positions themselves were never targets because the framework assumed context tokens were just inputs to the predictor, not predictions to be evaluated.

The critical asymmetry. The predictor $P_\phi$ processes the concatenated sequence of context tokens and mask tokens, and it produces an output token for every position in this sequence—both context and mask positions. But the loss in Equation 1 sums only over $\mathcal{M}$, the masked positions. The context-position outputs of the predictor are computed but ignored by the loss. This means:

1. The x-encoder's context token representations $E_\theta(x)_i$ for visible patches receive gradient only indirectly—through how well they help the predictor reconstruct other (masked) positions, not through any direct requirement to preserve their own content.
2. The optimal strategy for the x-encoder, from the perspective of minimizing $\mathcal{L}_{\text{predict}}$, is to make context tokens into the most useful possible inputs for the predictor's cross-attention over masked positions. This encourages context tokens to become global aggregators: each context token pools information from as much of the visible scene as possible, since rich global context helps predict arbitrary masked regions.
3. Local spatial information—exactly which patch a given context token came from—is detrimental to this global aggregation strategy. If a context token is too tied to its specific spatial location, it carries less information about distant parts of the scene that might be needed to reconstruct a far-away masked region.

Over the course of training, this asymmetry shapes the x-encoder's representations to discard fine-grained local structure in favor of global scene summaries. The resulting feature maps show "only fragmented local spatial structure" (Section 2.2, referring to Figure 1 and Figure 3).

---

Dense Predictive Loss (Section 2.3.1)

The core insight. The paper's central hypothesis is that the absence of local structure in V-JEPA 2 features is not an inherent limitation of the JEPA framework or the masking objective—it is a direct consequence of which tokens receive loss supervision. The context tokens have the capacity to encode local information (they are positioned at specific patch locations and carry positional encodings), but the objective gives them no incentive to do so. The fix is conceptually simple: also apply the self-supervised loss to context positions.

The context loss formulation. The authors introduce a second loss term $\mathcal{L}_{\text{ctx}}$ that mirrors the prediction loss but operates on context (visible) positions:

$\mathcal{L}_{\text{context}} = \frac{1}{|\mathcal{C}|} \sum_{i \in \mathcal{C}} \lambda_i \|P_\phi(E_\theta(x), \Delta_y)_i - \text{sg}(E_{\bar{\theta}}(y)_i)\|_1$

where $\mathcal{C}$ is the set of context token indices (the visible patches not dropped by masking), $\lambda_i$ is a patch-specific weight (defined below), and all other symbols are identical to Equation 1.

What it computes: for every visible context position, take the predictor's output for that position, compute its L1 distance to the y-encoder's target representation for that same position, weight it by $\lambda_i$, and average over all context positions.

Why this is non-trivial. On first glance, this looks like it should be trivial: the predictor receives the context token as input at position $i$, and is asked to output what the y-encoder produces at position $i$. But the context token has been processed through the x-encoder, while the y-encoder target comes from the EMA encoder processing the full unmasked input. If the x-encoder's context representation has become a global aggregator (discarding local patch identity), the predictor will struggle to map it to the specific local target—the information about which patch this is has been lost. The context loss forces the x-encoder to preserve per-patch identity in its context tokens so that the predictor can recover the target.

The weighting scheme. Naively setting all $\lambda_i$ to a constant value creates a gradient conflict (Table 2). Context tokens far from any masked region have no role in the mask-prediction task; forcing them to also preserve local identity purely through the context loss can degrade the quality of global information that nearby context tokens (which are used for mask prediction) might pool from them. The authors therefore introduce a spatially-aware weighting:

$\lambda_i = \frac{\lambda}{\sqrt{d_{\min}(i, \mathcal{M})}}$

where $\lambda$ is a base coefficient (different for images and videos), $d_{\min}(i, \mathcal{M})$ is the minimum spatio-temporal distance (measured in number of patch blocks) between context token $i$ and any masked token in the sequence, and the square root in the denominator means the weight decays with the square root of distance.

What it computes: for each context token, find the closest masked token's position. Compute the distance in patch-block units. Divide the base coefficient by the square root of that distance. Tokens adjacent to masked regions get high weight (small $d_{\min}$ makes $\lambda_i$ large); tokens far from masked regions get low weight.

Why this form: this weighting enforces local continuity between masked and context regions. Context tokens immediately adjacent to a masked region are most valuable for the mask-prediction task (they provide the immediate spatial context for what should be filled in), so they should preserve their local spatial identity the most. Distant context tokens, which the predictor uses for broader scene understanding, face a weaker local-preservation pressure and can still aggregate more globally. The square root ensures the weight drops off more gradually than a linear decay, maintaining some pressure even at moderate distances.

The base coefficient $\lambda$ differs by modality (Section 2.3.1, Appendix A): 0.5 for videos, 0.7 for images. The paper does not provide a detailed justification for why images get a higher coefficient, but it's consistent with the intuition that images have stronger static appearance cues that benefit from local preservation, while videos have motion as an additional source of information that can compensate for somewhat weaker local structure.

Warmup schedule for $\lambda$. Table 2 shows that fixed $\lambda$ values degrade action recognition on SSv2 (e.g., $\lambda = 0.5$ constant drops SSv2 from 72.8% to 53.8%). The authors introduce a warmup schedule: $\lambda$ ramps up linearly from 0 to its final value over epochs 50–100. This gives the network time to first establish a useful global organization before local-supervision pressure is applied, preventing the context loss from disrupting the early-stage learning of global features. The warmup is reported to "greatly stabilize training" but no quantitative ablation of different warmup durations is provided.

Empirical effect (Table 1, Figure 3). Adding the weighted context loss with warmup produces dramatic improvements in dense tasks while partially degrading global tasks:

The dense task gains are enormous (52% relative improvement on ADE20K, 30% reduction in depth error), while the global task degradation is significant but not catastrophic. Figure 3 shows qualitatively that the feature maps transform from noisy and fragmented to smooth and semantically organized—similar parts of objects (dog heads, car wheels) map to similar PCA components.

The residual problem. After the context loss alone, global understanding (SSv2, IN1K) is still substantially below the V-JEPA 2 baseline. The context loss has forced local structure back into the representations, but at the cost of weakening the global aggregation that makes video understanding work. This is the problem that Deep Self-Supervision solves.

---

Deep Self-Supervision (Section 2.3.2)

The mechanism. Instead of applying the self-supervised loss only at the final output layer of the encoder, V-JEPA 2.1 supervises the encoder at four intermediate layers in addition to the final layer. The specific layers used are "equally spaced indices" through the encoder depth; for the ViT-G model with 48 transformer blocks, these are blocks [12, 24, 36, 48] (Table 11).

Multi-level representation extraction and fusion. At each of the four selected layers, the x-encoder produces a representation for every visible context token. These four representations are concatenated along the channel dimension—if each layer produces $d$-dimensional tokens, the concatenated representation is $4d$-dimensional. A lightweight MLP then compresses this concatenated representation back down to a manageable dimensionality before it is passed to the predictor.

The paper does not specify the exact architecture of this MLP (number of layers, hidden dimension, activation function) beyond calling it "lightweight," but the compression step is critical: directly feeding $4d$-dimensional tokens into the predictor would quadruple its input dimension and dramatically increase memory and compute costs.

Multi-level prediction and loss. The predictor, which receives the compressed multi-level context tokens and mask tokens as input, produces four separate output sequences—one corresponding to each of the four encoder levels. Both the prediction loss (L_predict) and the context loss (L_ctx) are then applied at all four levels independently. The total loss is the sum of losses across all four levels, for both loss terms:

$\mathcal{L}_{\text{total}} = \sum_{\ell=1}^{4} \left( \mathcal{L}_{\text{predict}}^{(\ell)} + \mathcal{L}_{\text{context}}^{(\ell)} \right)$

where $\ell$ indexes the four encoder levels, $\mathcal{L}_{\text{predict}}^{(\ell)}$ is the prediction loss (Equation 1) computed using the predictor's output at level $\ell$ against the y-encoder's target at level $\ell$, and $\mathcal{L}_{\text{context}}^{(\ell)}$ is the context loss (Equation 2) computed at level $\ell$.

Why this works (the paper's hypothesis). Deep supervision provides gradient signals directly to earlier encoder layers, rather than relying on backpropagation through the entire network depth to shape representations there. This has two consequences:

1. Local information propagates upward. The context loss applied at early layers forces those layers to preserve spatial structure. Since these early-layer representations are then fed forward to later layers (via the normal transformer computation, not just through the MLP fusion), the spatial information survives through the network hierarchy rather than being discarded in favor of global aggregation. This is supported by the ablation in Appendix D.1 (Table 12): with deep self-supervision, the performance gap between using only the last layer vs. four layers for downstream evaluation shrinks dramatically (e.g., on ADE20K: 42.0 vs. 43.9 with deep supervision, compared to 34.9 vs. 39.1 without—the last-layer-only performance nearly catches up).

2. The gradient conflict between local and global is resolved at different levels. Earlier layers can specialize toward preserving fine-grained local structure (since that's what the early-layer context loss demands), while later layers can focus on global aggregation for mask prediction (since the mask-prediction loss at later layers provides a strong signal for global features). The multi-level loss lets the network develop a hierarchical representation where spatial detail at lower levels supports high-level semantics at upper levels, rather than forcing a single representation to serve both masters simultaneously.

Empirical effect (Table 1, Figure 5). Adding deep self-supervision on top of the context loss yields:

The key observation is that global understanding is almost fully recovered. SSv2 goes from 62.5% back to 72.1% (essentially matching V-JEPA 2's 72.8%), and IN1K goes from 72.6% to 80.8% (close to V-JEPA 2's 82.2%). Meanwhile, dense task performance continues to improve—ADE20K gains another +4.8 points, and NYUv2 improves slightly. Deep self-supervision doesn't just trade off between local and global; it appears to synergistically improve both, allowing the network to learn a genuinely hierarchical representation where local and global information coexist.

Practical consequence for downstream usage. Because deep self-supervision pushes spatial information into the final layers, downstream tasks that use only the final-layer features (the simplest and most computationally efficient protocol) perform nearly as well as those that extract multi-layer features. This eliminates the need for complex multi-scale feature extraction in downstream evaluation, which was previously necessary to achieve competitive dense-task performance with models that lacked deep supervision (Table 12).

Predictor capacity increase. To handle the increased complexity of producing and receiving multi-level representations, the predictor is scaled up from 12 blocks in V-JEPA 2 to 24 blocks in V-JEPA 2.1 (Appendix A: "Our predictor has 24 blocks (versus 12 in V-JEPA 2)"). The predictor embedding dimension remains at 384. The paper does not ablate whether the predictor capacity increase alone (without deep supervision) would help, but the two changes are presented as a unit.

---

Multi-Modal Tokenizer (Section 2.3.4)

The problem with V-JEPA 2's approach. V-JEPA 2 used a single 3D convolutional patch embedding layer for both images and videos. To process an image, it was temporally duplicated 16 times to create a 16-frame "static video," which was then processed through the 3D convolution. This has two problems:

1. Computational inefficiency: Processing a single image costs the same as processing a 16-frame video, wasting 16× compute since all frames are identical.
2. Representational incorrectness: The model learns that "static video" is the image modality, conflating the absence of motion (which can also occur in videos) with the image modality. This creates a structural ambiguity: the encoder cannot tell whether it's processing a genuinely static video clip or a duplicated still image, preventing modality-specific specialization.

The V-JEPA 2.1 solution. Two separate patch embedding layers and a modality token:

• Video tokenizer: A 3D convolution with kernel size 16 × 16 × 2 (spatial 16×16 patches, temporal depth 2), matching the tubelet size. Input is a video clip of T frames at resolution H × W; output is a sequence of (T/2) × (H/16) × (W/16) tokens.
• Image tokenizer: A 2D convolution with kernel size 16 × 16. Input is a single image at resolution H × W; output is a sequence of (H/16) × (W/16) tokens.
• Learnable modality embedding: A trainable vector that is added to every token in the sequence, with different learned values for the image pathway vs. the video pathway. This tells the encoder explicitly which modality each token came from. The modality embedding is added in addition to the standard 3D Rotational Positional Encoding (RoPE) that carries spatio-temporal position information.

What changes downstream. With separate tokenizers:

• Images are processed natively as 2D inputs—no temporal duplication, no wasted compute.
• The network can learn modality-specific processing: the image pathway can specialize in extracting high-quality static appearance features, while the video pathway can specialize in motion and temporal dynamics, with the shared encoder backbone learning representations that are useful for both.
• The modality embedding provides a clean signal for the encoder to condition its processing: it knows whether to look for motion cues or focus on static appearance.

Distributed training for mixed batches (Section 2.3.3). Images and videos are not mixed within a single training batch. Instead, the distributed training setup uses separate workers for each modality: some GPU nodes process video-only batches, others process image-only batches. After each iteration, gradients from both sets of nodes are aggregated (all-reduced) before updating the model parameters. The image-to-video ratio is controlled by setting per-modality batch sizes: 128 video clips and 2,304 images per global batch. This ratio means that in each training iteration, the model sees approximately 18× more image tokens than video tokens (2,304 individual images vs. 128 × 16 frames = 2,048 video frames, but images are processed at the same resolution), reflecting the larger diversity and easier scalability of image datasets.

Empirical effect (Table 1). The multi-modal tokenizer provides a modest but consistent improvement:

• ADE20K: 40.8 → 41.4 mIoU
• NYUv2: 0.418 → 0.415 RMSE
• SSv2/IN1K: essentially flat (72.6 → 72.6, 81.6 → 81.6)

The main benefits are computational efficiency and architectural cleanliness rather than dramatic accuracy gains. The paper notes that "using the Multi-Modal Tokenizer has a positive effect on dense-task performance" while "performance on action or object recognition remains stable," consistent with the idea that images processed natively provide better appearance cues for spatial tasks.

---

Data Scaling: VisionMix163M (Section 2.3.3)

Dataset composition. V-JEPA 2.1 constructs a new pretraining dataset called VisionMix163M by combining and reweighting sources from two prior works (Table 3):

The key changes from V-JEPA 2's VideoMix22M:

1. ImageNet (1M) is completely removed and replaced by LVD-142M (142M curated images from Oquab et al., 2023). LVD-142M was curated using a cluster-based retrieval strategy from a large pool of raw internet data to ensure diversity and quality. This increases image data by 142× and dramatically broadens the appearance distribution.

2. SSv2 weight is tripled (0.056 → 0.170). SSv2 contains egocentric, motion-heavy videos where actions are defined by motion patterns rather than object appearance (e.g., "pushing something from left to right"). Increasing its weight emphasizes motion understanding.

3. YT-1B weight is nearly quadrupled (0.188 → 0.720). YT-1B (Zellers et al., 2022) is a large collection of YouTube videos with "much more heterogeneous video samples" —diverse in content, camera motion, scene types, and visual quality. This provides broad coverage of natural video statistics.

4. Kinetics and HowTo100M weights are reduced (0.188 → 0.010 and 0.318 → 0.100, respectively). The authors argue that LVD-142M already covers many static visual concepts that were previously provided by these datasets' appearance diversity, so they shift the video sampling toward more dynamic, motion-rich sources.

Sampling weights vs. actual batch composition. The weights in Table 3 represent sampling probabilities: during data loading, a sample is drawn from source $s$ with probability proportional to its weight. With the global batch containing 128 video clips and 2,304 images, the actual mix per global batch comes from sampling videos from the video sources according to the video weights (normalized) and images from LVD-142M.

Rationale for reweighting. The guiding principle is complementarity: since LVD-142M provides comprehensive coverage of static appearance, the video component should focus on what images cannot provide—motion, dynamics, temporal causality, and egocentric interaction patterns. This is why SSv2 (strong motion emphasis) and YT-1B (broad natural video statistics) get more weight, while Kinetics (many classes identifiable from a single frame) gets less.

Empirical effect (Table 1). Switching from V-JEPA 2's VideoMix22M to VisionMix163M improves all metrics:

• ADE20K: 38.6 → 40.8 mIoU
• NYUv2: 0.463 → 0.418 RMSE
• SSv2: 72.1 → 72.6%
• IN1K: 80.8 → 81.6%

The gains are broad but modest in absolute terms, suggesting the data scaling mostly refines already-good representations rather than enabling qualitatively new capabilities—at least at this model size (ViT-L). The full impact of data scaling is realized when combined with model scaling (below).

---

Model Scaling and High-Resolution Cooldown (Section 2.3.5)

Scaling to ViT-G. The paper scales from ViT-L (300M parameters) to ViT-G (2B parameters), approximately a 6.7× increase in parameter count. The ViT-G architecture is not described in detail in the main text beyond having 48 transformer blocks (inferred from the intermediate block indices [12, 24, 36, 48]), a patch size of 16, and an unspecified embedding dimension. The scaling is presented as a straightforward increase in model capacity with all other training hyperparameters held constant.

Empirical effect of model scaling (Table 1). Going from ViT-L to ViT-G yields the largest single improvement among all components:

• ADE20K: 41.4 → 47.1 mIoU (+5.7 points)
• NYUv2: 0.415 → 0.365 RMSE (−12% error)
• SSv2: 72.6 → 76.1% (+3.5 points)
• IN1K: 81.6 → 84.8% (+3.2 points)

The dense-task improvements are particularly striking—ADE20K gains nearly as much from model scaling alone (+5.7) as from all the algorithmic innovations combined up to that point (+19.2 from V-JEPA 2 to ViT-L with all innovations). This suggests that the algorithmic changes (context loss, deep supervision) create a training signal that effectively leverages additional capacity, rather than saturating at the ViT-L scale.

High-resolution cooldown phase. After the primary training phase (135,000 iterations at constant learning rate), the model undergoes a short cooldown phase of 12,000 iterations where:

• Learning rate is decayed from $6 \times 10^{-4}$ to $1 \times 10^{-6}$ (Appendix A, Table 11).
• Video resolution increases from 16 frames at 256×256 to 64 frames at 384×384 (4× temporal, 2.25× spatial area per frame).
• Image resolution increases from 256×256 to 512×512 (4× spatial area).
• All other hyperparameters (mask ratios, EMA coefficient, weight decay, batch sizes) remain unchanged.

The cooldown phase is 12,000 iterations—short relative to the 135,000-iteration primary phase (~8% additional training), but at higher resolution and thus higher per-iteration compute cost. The learning rate decay schedule transitions from the constant $5.25 \times 10^{-4}$ used in the primary phase (after warmup) to the decaying schedule of the cooldown: it starts at $6 \times 10^{-4}$ and decays to $1 \times 10^{-6}$. The jump from $5.25 \times 10^{-4}$ (end of primary) to $6 \times 10^{-4}$ (start of cooldown) is small and likely serves to give a slight learning rate boost at the start of the high-resolution phase before decaying.

Why a cooldown phase helps. The cooldown serves two purposes:

1. Higher resolution allows the model to learn finer-grained spatial features. The patch size remains 16, so at 512×512 image resolution, the model processes 32×32 = 1,024 patches (vs. 16×16 = 256 at 256×256). For videos at 384×384 with 64 frames and tubelet size 2, the model processes (64/2) × (384/16) × (384/16) = 32 × 24 × 24 = 18,432 patches—a massive increase in token count.
2. Decaying the learning rate at this higher resolution lets the model fine-tune its existing representations to the finer spatial grid without the instability that would come from training at high resolution from scratch with a high learning rate.

Empirical effect of cooldown (Table 1, Table 13). The cooldown provides consistent but diminishing improvements:

• ADE20K: 47.1 → 47.9 mIoU (+0.8)
• NYUv2: 0.365 → 0.307 RMSE (−16% error—the largest relative gain)
• SSv2: 76.1 → 77.7% (+1.6)
• IN1K: 84.8 → 85.5% (+0.7)

Depth estimation benefits disproportionately from the cooldown, likely because higher spatial resolution directly enables finer depth boundaries and more accurate small-object depth estimates. Table 13 provides a more detailed ablation showing that for ViT-G, the cooldown improves SSv2 (77.1→77.7), Diving-48 (89.5→89.2), K400 (87.3→87.7), IN1K (84.8→85.5), EK100 (38.8→40.8), ADE20K (47.8→47.9), Cityscapes (73.2→73.5), VOC (84.7→85.0), NYUv2 (0.313→0.307), KITTI (2.472→2.461), DAVIS (68.6→69.0), and YT-VOS (72.7→72.6). Every metric improves except YT-VOS and Diving-48, which are essentially flat.

Scaling from ViT-g (1B) to ViT-G (2B) in final results. The paper trains both a ViT-g (1B) and ViT-G (2B) variant. Table 13 compares ViT-g at 384 resolution with ViT-G at 384 resolution, showing that the 2× parameter increase provides additional gains: SSv2 (76.9→77.7), Diving-48 (89.0→89.2), K400 (87.0→87.7), IN1K (84.8→85.5), EK100 (38.4→40.8), ADE20K (47.8→47.9), Cityscapes (71.8→73.5), NYUv2 (0.350→0.307), KITTI (2.601→2.461), DAVIS (67.4→69.0), YT-VOS (71.3→72.6). Again, the pattern is broad but diminishing—the biggest jumps are in tasks requiring fine-grained understanding (depth, segmentation, video object segmentation).

---

Training Recipe: Complete Hyperparameters

The full training configuration is specified in Appendix A, Table 11, and summarized below. All hyperparameters are identical across ViT architectures (ViT-L, ViT-g, ViT-G).

Primary phase (135,000 iterations):

• Frames per video clip: 16
• Frames per second (sampling rate): 4.0 (so each clip spans 4 seconds)
• Video crop size: 256 × 256
• Image crop size: 256 × 256
• Random resize aspect ratio: [0.75, 1.35] (images/videos are randomly resized with aspect ratio distortion before cropping)
• Random resize scale: [0.3, 1.0] (the crop covers 30–100% of the resized image area)
• Warmup steps: 12,000 (learning rate ramps linearly from $1 \times 10^{-4}$ to $5.25 \times 10^{-4}$ over this period)
• Learning rate schedule: warmup, then constant at $5.25 \times 10^{-4}$ for the remaining 123,000 steps
• Global image batch size: 2,304
• Global video batch size: 128
• Weight decay: 0.04
• EMA coefficient ($\beta$): 0.99925
• Spatial mask scale: uniform in [0.15, 0.7] (fraction of spatial patches kept visible)
• Temporal mask scale: [1.0, 1.0] (masking operates on whole tubelets)
• Mask aspect ratio: uniform in [0.75, 1.5]
• Tubelet size: 2
• Patch size: 16 (for both spatial dimensions in images and videos)
• Predictor blocks: 24
• Predictor embedding dimension: 384
• Encoder intermediate blocks (ViT-G): [12, 24, 36, 48] (four equally-spaced layers)
• Context loss base $\lambda$: 0.5 for video, 0.7 for images

Cooldown phase (12,000 iterations):

• Frames per video clip: 64 (4× increase)
• Frames per second: 4.0 (unchanged—clip duration increases to 16 seconds)
• Video crop size: 384 × 384
• Image crop size: 512 × 512
• Random resize parameters: unchanged
• Warmup steps: none (N/A—learning rate starts decaying immediately)
• Starting learning rate: $6 \times 10^{-4}$
• Final learning rate: $1 \times 10^{-6}$ (decayed over 12,000 steps)
• All other hyperparameters: identical to primary phase

Optimizer details are partially specified: the paper mentions using AdamW (inferred from weight decay being separate from learning rate, a defining characteristic of AdamW), but does not provide the $\beta_1$, $\beta_2$, or $\epsilon$ values. The learning rate warmup and scheduling follow "the warmup-constant learning rate schedule of V-JEPA 2" (Section 2.3), implying the optimizer configuration matches Assran et al. (2025).

Why these specific numbers:

• EMA 0.99925: This is a standard value in non-contrastive SSL (used by BYOL, DINO, V-JEPA 2). It provides slow momentum—with ~135K steps, the effective "memory" of the EMA is on the order of $1/(1-0.99925) \approx 1,333$ steps, meaning the target encoder evolves slowly and provides a stable target.
• Weight decay 0.04: Relatively low for a model of this size, consistent with the observation that SSL models benefit from less aggressive regularization than supervised models.
• Spatial mask scale [0.15, 0.7]: The upper bound of 0.7 means at most 70% of patches are visible (at least 30% masked), while the lower bound of 0.15 means as few as 15% can be visible (up to 85% masked). This broad range forces the model to handle both mild masking (mostly visible, easy prediction) and extreme masking (mostly hidden, requires strong global reasoning).
• Context loss $\lambda$ = 0.5 video, 0.7 image: Images have higher $\lambda$ because they lack temporal information that can help with prediction; the stronger context loss compensates by enforcing better spatial structure from appearance alone.

---

Model Distillation (Section 2.3.6, Appendix B)

Purpose. The ViT-G model is 2B parameters, which is computationally expensive for many downstream applications. The authors distill it into smaller variants (ViT-L at 300M parameters and ViT-B at 80M parameters) that preserve most of the representation quality at a fraction of the cost.

Distillation protocol differences from pretraining:

1. Teacher: The EMA target encoder is replaced by a frozen, pretrained ViT-G teacher. The teacher's weights are fixed throughout distillation; no EMA update occurs.
2. Student EMA: A separate EMA of the student encoder is maintained (Polyak averaging) but is not used in the loss calculation. This EMA copy is what gets released as the final model—a common technique where the EMA version provides better generalization than the raw training checkpoint.
3. Loss: The distillation loss is identical to the pretraining loss (L1 on mask and context predictions), but is only computed on the last layer of the teacher encoder—no deep self-supervision targets from intermediate layers. This simplifies the pipeline and reduces memory.
4. Predictor: A new predictor is trained from scratch (not initialized from pretraining weights) with only 12 blocks (instead of 24 for pretraining), which the authors report "greatly improves distillation stability." The predictor's final linear layer is adapted to match the teacher's embedding dimension.
5. Two-stage training: Stage 1 uses a pre-cooldown, low-resolution (16 frames, 256×256) ViT-G as teacher. Stage 2 initializes from the Stage 1 EMA student and uses the post-cooldown, high-resolution (64 frames, 384×384) ViT-G as teacher. The paper states this two-stage approach "found to produce the best-performing models compared to alternatives such as single-stage approaches or always distilling from the final high-resolution V-JEPA 2.1 teacher."

All other hyperparameters (masking ratios, cooldown schedules, learning rates, data augmentations, batch sizes, weight decay) remain identical to the pretraining recipe.

Performance. Distilled ViT-L nearly matches ViT-G on most tasks (Section 3.10, though the paper's Table reference is broken—it says "Table ??"). The gap is especially small on depth estimation and video object segmentation, while action recognition and semantic segmentation show a moderate gap that suggests some capacity-dependent capabilities are harder to distill.

---

Summary of Design Choices and Their Justifications

• Distance-weighted context loss over uniform weighting: prevents distant context tokens from being forced to preserve local identity when they're primarily needed for global aggregation; the $1/\sqrt{d_{\min}}$ form enforces local continuity at mask boundaries where it's most valuable.
• Warmup schedule for $\lambda$ over constant weighting: lets global features establish before local-supervision pressure is applied, preventing the context loss from disrupting early-stage learning of motion and dynamics.
• Four-level deep supervision over single-level: propagates gradient signals to earlier layers, ensuring spatial structure survives the full network hierarchy; synergistically resolves the local-global gradient conflict rather than merely trading off.
• MLP fusion of multi-level representations: compresses 4× channel dimension to prevent the predictor's input and memory costs from exploding while preserving multi-scale information.
• Separate 2D/3D tokenizers over shared 3D: processes images natively without wasteful temporal duplication; allows modality-specific specialization via the learned modality embedding.
• LVD-142M replacing ImageNet over continuing with small image datasets: provides 142× more diverse appearance data; shifts video sampling toward motion-heavy sources since static appearance is now well-covered.
• Two-phase training with high-resolution cooldown over single-phase: allows stable initial learning at lower resolution (less memory, faster iterations) followed by fine-grained feature refinement at high resolution with a decaying learning rate.
• 12-block predictor for distillation over 24-block: empirically found to improve stability when the teacher is frozen; the reduced capacity may prevent the predictor from overfitting to teacher-specific representations that don't generalize.
• Two-stage distillation over single-stage: the pre-cooldown teacher provides a stable low-resolution target for initial learning; the post-cooldown teacher refines at high resolution; this prevents training instability from immediately targeting high-resolution representations.

4. Key Insights and Innovations

Innovation 1: The Context-Token Asymmetry Diagnosis — Identifying Why Strong Video Models Fail at Dense Tasks

The paper's deepest intellectual contribution is not the addition of a context loss per se, but rather the diagnostic framework that identifies why JEPA-based video models produce spatially impoverished representations. This shifts the conversation from "dense features and global understanding are opposing objectives" to "dense features fail to emerge because of a specific, fixable asymmetry in the loss function."

Prior to this work, the dominant assumption — implicit in the design of V-JEPA 2, VideoMAE, and related masked-prediction approaches — was that mask-denoising objectives would naturally learn representations that preserve spatial structure, since predicting masked regions requires understanding their relationship to visible context. The field operated under the intuition that forcing a model to fill in missing patches would make it learn about object geometry, boundaries, and layout. V-JEPA 2's strong performance on action recognition reinforced this intuition: if the model understands motion and dynamics well enough to predict future frames in latent space, surely it must also understand spatial layout.

The paper demonstrates that this intuition is wrong in a specific, mechanistic way. By examining the V-JEPA 2 loss (Equation 1) and noting that it applies only to masked-position outputs while the predictor produces outputs for both context and masked positions, the authors identify a gradient asymmetry that actively discourages spatial structure. Context tokens face no direct requirement to preserve their own patch identity; they are evaluated only through their indirect contribution to reconstructing other patches. This creates an incentive for them to become global aggregators — functionally equivalent to the register tokens identified by Darcet et al. (2023) — that pool information across the entire visible scene and discard per-patch spatial detail. The PCA visualizations in Figure 1 and Figure 3, showing "noisy and fragmented" feature maps with no coherent object-level groupings, are a direct consequence of this incentive structure, not an inevitable limitation of mask-prediction objectives.

What makes this diagnosis fundamental rather than incremental is that it explains a pattern of results that previously seemed contradictory. The V-JEPA family achieved state-of-the-art action recognition (72.8–77.3% on SSv2) and action anticipation (39.7 Recall@5 on EK100), demonstrating sophisticated understanding of motion, dynamics, and temporal causality. Yet the same encoders produced near-random performance on depth estimation (0.642 RMSE) and semantic segmentation (22.2 mIoU on ADE20K). The diagnosis resolves this paradox: the model was learning exactly what the objective asked it to learn — rich global representations optimized for masked-patch prediction — and the objective never asked it to preserve local spatial identity. The two capabilities were not in tension because of any fundamental representational limitation; they diverged because the training signal was one-sided.

This diagnosis is significant beyond V-JEPA specifically because the same asymmetry exists in many masked-prediction SSL objectives. VideoMAE, MaskFeat, and related approaches also compute losses only on masked positions. The paper's framework provides a lens through which to examine these methods: are their context tokens also collapsing into global aggregators? If so, the same fix — adding a context supervision signal — might unlock dense features in those frameworks too. The paper doesn't test this, but the diagnostic concept is portable.

The evidence anchoring this insight is Figure 3 and Table 1: the mere act of adding a context loss transforms feature maps from fragmented noise into spatially coherent, semantically organized patterns, and ADE20K mIoU jumps from 22.2 to 33.8 (+52%). This magnitude of improvement from a loss-function modification — without changing architecture, data, or model size — constitutes strong evidence that the problem was in the training signal, not in the model's representational capacity.

---

Innovation 2: The Local-Global Gradient Conflict as a Manageable Trade-off Rather Than an Inherent Limitation

The paper's second conceptual contribution is demonstrating that the tension between dense local features and global semantic understanding — which prior work treated as an inherent and perhaps insurmountable trade-off — is actually a gradient conflict that can be managed through careful loss design and hierarchical supervision. This reframes the problem from "how do we get a single model to do two opposing things?" to "how do we design the training signal so that local and global information coexist in a hierarchical representation?"

The naive view, which the paper experiments with and refutes, is that you can simply add local and global losses together and let optimization sort it out. Table 2 shows what happens: with a fixed context-loss weight of 0.5, SSv2 accuracy crashes from 72.8% to 53.8%. The gradients from the two losses are genuinely in conflict — the context loss pushes context tokens to preserve per-patch identity (local), while the prediction loss pushes them to aggregate information across the scene (global). Simultaneous optimization with balanced weights produces representations that are mediocre at both.

The paper's resolution is a hierarchical separation of concerns. Deep self-supervision applies the context and prediction losses at four encoder levels simultaneously. This allows earlier layers to specialize toward preserving local structure (where the context-loss gradient dominates) while later layers specialize toward global aggregation for mask prediction (where the prediction-loss gradient dominates). The critical empirical evidence is Table 1: after adding deep self-supervision, SSv2 recovers from 62.5% (context loss alone) to 72.1% (context loss + deep supervision), nearly matching the original 72.8%, while ADE20K continues to improve from 33.8 to 38.6 mIoU. This is not a trade-off being optimized — it is a synergistic improvement where making the representation hierarchical makes both local and global features better than either could be in a flat representation.

The distance-weighted context loss further refines this gradient management. By weighting context tokens inversely with distance to the nearest masked region (λ_i = λ / √d_min), the loss enforces local continuity exactly where it matters most for prediction — at mask boundaries — while leaving distant context tokens free to specialize in global aggregation. This spatial decomposition of the gradient conflict (local pressure near masks, global pressure everywhere) is a conceptually elegant solution that avoids forcing any single token to serve two masters simultaneously.

Why this is fundamental rather than incremental: prior work in multi-task SSL largely treated the local-global tension as a capacity-allocation problem — you need a big enough model to learn both, or you accept that specialized models will dominate on their respective tasks. The V-JEPA 2.1 findings suggest a different paradigm: the tension arises from conflicting gradient signals at the same representational location. By restructuring the loss to apply at different depths (deep supervision) and different spatial positions (distance-weighting), you can resolve the conflict without increasing model capacity. The ViT-L model with deep supervision + context loss achieves 38.6 mIoU on ADE20K (vs. 22.2 for V-JEPA 2 ViT-g, which is 3× larger), demonstrating that the gradient-management strategy can substitute for raw model scaling.

This insight also explains why DINO-based models achieve strong dense features without explicit multi-layer supervision: DINO's image-level [CLS] token loss and patch-level local losses operate at different representational depths by construction (the [CLS] token aggregates from the final layer while patch features retain local information from earlier layers). V-JEPA 2.1's deep self-supervision can be seen as making this hierarchical separation explicit in the JEPA framework, which lacked DINO's natural architecture-induced hierarchy.

---

Innovation 3: Dense Features as a Prerequisite for World Modeling — Closing the Loop Between Representation Quality and Downstream Action

The paper's third contribution is demonstrating that improving dense feature quality directly translates to better performance on embodied tasks — robot manipulation and navigation — that require fine-grained spatial understanding, not just semantic recognition. This closes a loop that prior work had left open: V-JEPA 2 showed that JEPA representations could support world models for planning, but its planning performance was limited by the spatial imprecision of its features (see Figure 8). V-JEPA 2.1 shows that the improved dense features directly address this bottleneck.

The most compelling evidence is the 20% improvement in real-robot grasping success rate (Table 6). V-JEPA 2 achieved 60% on the Grasp task; V-JEPA 2.1 with the same planning budget (800 CEM samples, 10 iterations, 1-step horizon) achieves 70%. With a longer planning horizon (8 steps) enabled by better feature quality, this rises to 80%. The paper explicitly attributes this to "improved depth understanding" (Section 3.3) and notes qualitatively that failures with V-JEPA 2.1 are "a result of poor planning over gripper actions, as opposed to failures in spatial understanding" — a shift in failure mode from spatial to strategic errors. This is significant because it suggests the representation bottleneck has been lifted: the model now correctly perceives object locations and 3D structure, and the remaining errors are in action selection, not perception.

The navigation results (Table 7) provide a second line of evidence. Replacing NWM's SD-VAE image encoder with V-JEPA 2.1 features enables 10× faster planning (10.6 seconds vs. 103.2 seconds) with better trajectory accuracy (5.687 vs. 5.831 ATE on Tartan Drive for ViT-G). The mechanism is that V-JEPA 2.1's dense features require fewer denoising steps in the diffusion-based planning model (8 vs. 128) — the representations are already spatially coherent, so the diffusion model doesn't need to do as much work to produce a plausible trajectory. This is a striking demonstration that representation quality directly impacts downstream computational efficiency, not just accuracy.

This insight reframes the purpose of dense features in video SSL: they are not merely useful for "vision tasks" like segmentation and depth estimation in the abstract, but are directly necessary for embodied systems that must reason about precise spatial relationships to act in the physical world. A model that understands "a cup is being grasped" but cannot precisely localize the cup in 3D space will fail at manipulation regardless of its semantic understanding. The paper's combination of dense-feature benchmarks (NYUv2, ADE20K) with real-robot evaluations makes this argument concrete rather than aspirational.

---

Innovation 4: The Data-Modality Complementarity Principle for Joint Image-Video Training

The paper's fourth contribution is a principle for constructing joint image-video training datasets: since images can provide broad static appearance coverage, video data should be curated to emphasize motion, dynamics, and temporal phenomena that images cannot capture. This sounds obvious in retrospect but represents a departure from prior practice.

V-JEPA 2's VideoMix22M treated image and video data sources as interchangeable components of a uniform distribution, mixing ImageNet (static appearance) with Kinetics (many appearance-recognizable classes) and SSv2 (motion-heavy) at fixed weights. The result was a dataset that contained plenty of data but without a clear principle for what each modality contributed. The V-JEPA 2.1 data strategy, by contrast, makes an explicit complementarity argument: LVD-142M (142M diverse images) already covers the space of static visual concepts comprehensively, so video sources should be reweighted toward dynamic content. This is why SSv2 weight triples (0.056 → 0.170), YT-1B weight nearly quadruples (0.188 → 0.720), and Kinetics weight drops dramatically (0.188 → 0.010).

The empirical payoff (Table 1) shows this is not just a cosmetic reweighting: it provides consistent improvements across all task categories (+2.2 mIoU on ADE20K, +0.5% on SSv2, +0.8% on IN1K). But the deeper significance is that this principle is scalable: as image datasets continue to grow (LVD-142M is already 142M images; future datasets could be larger), the complementarity argument suggests that video pretraining should become increasingly motion-focused, potentially leading to training distributions that look very different from today's uniform mixtures.

The multi-modal tokenizer (separate 2D/3D patch embeddings, modality tokens) operationalizes this principle at the architectural level: it allows the model to develop modality-specific processing pathways that specialize in what each modality offers, rather than forcing images through a video pipeline that treats them as degenerate static videos. This architectural choice is more than an efficiency optimization — it is a commitment to the idea that images and videos contain complementary rather than redundant information, and that the model should learn to extract different features from each.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a diverse suite spanning dense prediction, global recognition, video understanding, and embodied tasks. Key datasets include: NYUv2 (Silberman et al., 2012b) and KITTI (Geiger et al., 2013b) for monocular depth estimation; ADE20K (Zhou et al., 2017a), Pascal VOC12 (Everingham et al., 2015), and Cityscapes (Cordts et al., 2016) for semantic segmentation; DAVIS 2017 (Pont-Tuset et al., 2017) and YouTube-VOS (Xu et al., 2018) for video object segmentation; Ego4D STA v2 (Grauman et al., 2022) for short-term object interaction anticipation (243 hours, 98,276 training / 47,395 validation samples); EPIC-KITCHENS-100 (Damen et al., 2022) for action anticipation (100 hours, 3,568 action labels across 97 verbs and 300 nouns); Something-Something-v2 (Goyal et al., 2017), Diving-48 (Li et al., 2018), and Kinetics-400 (Kay et al., 2017) for video action recognition; ImageNet-1K (Deng et al., 2009) for image classification; PerceptionTest (Pătrăucean et al., 2023), MVP (Krojer et al., 2024), TempCompass (Liu et al., 2024), TemporalBench (Cai et al., 2024), TOMATO (Shangguan et al., 2024), MVBench (Li et al., 2024), and TVBench for video question answering; and real-robot Franka Panda arm manipulation tasks (Reach, Grasp, Pick-and-Place) with visual goal specification, plus Tartan Drive, Scand, and Sacson for navigation planning.

• Base model(s). Two primary architectures are trained from scratch: ViT-g (1B parameters) and ViT-G (2B parameters, 48 transformer blocks). Both use a patch size of 16, a tubelet size of 2 for videos, a predictor with 24 transformer blocks and embedding dimension 384, and the four-level deep self-supervision at equally-spaced encoder block indices (e.g., [12, 24, 36, 48] for ViT-G). Additionally, two distilled variants are produced: ViT-L (300M parameters) and ViT-B (80M parameters), distilled from the ViT-G teacher using a modified pretraining recipe (frozen teacher, no deep supervision, 12-block predictor). All models are trained on VisionMix163M (142M curated images from LVD-142M + video sources including SSv2, Kinetics, HowTo100M, and YT-1B with reweighted sampling) using the two-phase schedule (135K iterations primary at 256×256, 12K iterations cooldown at 384×384–512×512). The ViT-g model is chosen as the 1B baseline because it matches V-JEPA 2's architecture scale, enabling direct comparison, while ViT-G tests scaling behavior.

• Metrics. The paper reports a wide range of metrics chosen per-task: Root Mean Squared Error (RMSE) for depth estimation (lower is better); mean Intersection-over-Union (mIoU) for semantic segmentation (higher is better); J &F-mean (joint region similarity and contour accuracy) for video object segmentation; Top-5 mean Average Precision (mAP) with class-specific matching criteria (IoU > 0.5, correct noun, correct verb, time-to-contact within 0.25s tolerance) for Ego4D STA, with the metric discounting up to four highest-scoring false positives per example to handle multi-modal future anticipation; mean-class Recall@5 for EPIC-KITCHENS action anticipation (computing recall separately per verb, noun, and joint action classes and averaging); Top-1 accuracy (%) for all classification tasks (SSv2, Diving-48, K400, IN1K); accuracy for VideoQA benchmarks; success rate (%) for robot manipulation (10 tasks per skill, averaged); Average Trajectory Error (ATE) and Relative Trajectory Error (RTE) for navigation planning; and planning time (seconds) as a compute efficiency metric. All downstream evaluations use the encoder as a frozen feature extractor—no encoder weights are updated during downstream training, which directly tests pretrained representation quality rather than the model's ability to adapt.

• Baselines. The paper compares against a comprehensive set of prior work spanning multiple SSL paradigms. For dense tasks (depth, segmentation, VOS): DINOv2 ViT-g and ViT-L (Oquab et al., 2023), DINOv3 ViT-H+ and ViT-7B (Siméoni et al., 2025), PEcore and PEspatial (Bolya et al., 2025), AM-RADIOv2.5 (Ranzinger et al., 2024), InternVideo2-1B (Wang et al., 2024b), SigLIP 2 (Tschannen et al., 2025), Web-DINO (Fan et al., 2025), and V-JEPA 2 ViT-g (Assran et al., 2025). For video action recognition (Table 9): V-JEPA ViT-H (Bardes et al., 2024), V-JEPA 2 ViT-g, DINOv2 ViT-g, DINOv3 ViT-H+ and ViT-7B, PEcore G, SigLIP2, InternVideo2-1B and InternVideo2s-1B, VideoMAEv2 (Wang et al., 2023), and VideoPrism (Zhao et al., 2024). For Ego4D STA (Table 4): StillFast (Ragusa et al., 2023), STAformer (Mur-Labadia et al., 2024), GANO (Thakur et al., 2023), plus DINOv2 ViT-L and ViT-g, DINOv3 ViT-H+ and ViT-7B, and V-JEPA 2 ViT-g evaluated under the same attentive probe protocol. For EK100 action anticipation (Table 5): InAViT (Roy et al., 2024), Video-LLaMA (Zhang et al., 2023), PlausiVL (Mittal et al., 2024), and V-JEPA 2 ViT-g. For VideoQA (Table 10): InternVL-2.5 (Chen et al., 2024), Qwen2VL (Wang et al., 2024a), Qwen2.5VL (Qwen Team et al., 2025), PLM 8B (Cho et al., 2025), and V-JEPA 2 ViT-g. For robot manipulation (Table 6): V-JEPA 2 with the same planning setup. For navigation (Table 7): NWM (Bar et al., 2025). Where prior work used different evaluation protocols, the paper re-evaluates those models under its own protocol for fair comparison (notably, the STA baselines and the video classification baselines in Table 9).

• Generation budget / compute accounting. Pretraining compute is accounted in training iterations (135,000 primary + 12,000 cooldown) with fixed batch sizes (2,304 images, 128 video clips per global batch), making total FLOPs proportional to model size × iterations. The ViT-g and ViT-G models are compared at equivalent training iterations and data volumes (both trained on the same VisionMix163M with the same schedule), so the only difference is model capacity. For downstream evaluation, compute is not explicitly budgeted—each probe is trained to convergence with hyperparameter sweeps, and results report the best-performing configuration. This means the paper does not perform a compute-matched comparison between V-JEPA 2.1 and baselines at evaluation time; all comparisons are at fixed model sizes and fixed pretraining data volumes. The only explicit compute-efficiency comparison is the navigation planning time (Table 7), where V-JEPA 2.1's 10.6-second planning time is compared to NWM's 103.2 seconds.

• Cross-validation / statistical protocol. No formal statistical testing (confidence intervals, standard errors, significance tests) is reported for any downstream benchmark. Hyperparameter selection is done via grid search over learning rates and weight decay (Section C.1–C.6 of Appendix C describe sweep ranges for each task), with the best configuration selected based on validation-set performance. For video object segmentation, hyperparameters are selected on the DAVIS training set and applied to all test splits (including YouTube-VOS). For the ablation studies (Figure 5, Tables 1–2), results are reported as single numbers without error bars or multiple random seeds. The robot manipulation evaluation uses 10 tasks per skill and averages results, providing some implicit variance estimation but no formal statistics. All downstream probe training uses a fixed frozen encoder, eliminating randomness from pretraining initialization in downstream comparisons (though pretraining itself is subject to seed variation, which is not studied).

Main Quantitative Results

Dense Task Performance: Depth Estimation and Semantic Segmentation (Table 8, Figure 10, Figure 11, Figure 12)

Headline numbers. V-JEPA 2.1 ViT-G achieves 0.307 RMSE on NYUv2 depth estimation and 2.461 RMSE on KITTI, setting a new state-of-the-art for linear-probe monocular depth estimation among models under 2B parameters. On semantic segmentation, it achieves 47.9 mIoU on ADE20K, 73.5 mIoU on Cityscapes, and 85.0 mIoU on Pascal VOC12.

Depth estimation comparison. On NYUv2, V-JEPA 2.1 ViT-G (0.307 RMSE) surpasses all prior models under 2B parameters, including PEspatial-G (0.362 RMSE), DINOv2 ViT-g (0.372 RMSE), DINOv3 ViT-H+ (0.352 RMSE), and the previous V-JEPA 2 ViT-g (0.642 RMSE)—a 52% reduction in error. Remarkably, it even outperforms the 7B-parameter DINOv3 ViT-7B (0.309 RMSE), demonstrating that V-JEPA 2.1's architectural innovations achieve better depth understanding at 3.5× fewer parameters. On KITTI, V-JEPA 2.1 ViT-G achieves 2.461 RMSE, outperforming all models under 2B parameters including DINOv2 ViT-g (2.624 RMSE), DINOv3 ViT-H+ (2.635 RMSE), and PEspatial-G (3.082 RMSE). The ViT-g variant (1B) achieves 0.350 RMSE on NYUv2 and 2.601 on KITTI, already competitive with DINOv3 ViT-H+ (0.8B, 0.352 RMSE on NYU, 2.635 on KITTI) despite having no specialized dense-prediction training.

Semantic segmentation comparison. On ADE20K, V-JEPA 2.1 ViT-G (47.9 mIoU) is highly competitive but remains behind specialized image encoders: DINOv3 ViT-7B (55.9 mIoU), DINOv3 ViT-H+ (54.8 mIoU), AM-RADIOv2.5 (53.0 mIoU), and DINOv2 ViT-g (49.5 mIoU). The gap to DINOv3 ViT-7B is 8.0 mIoU, which the authors attribute to ADE20K's "numerous object classes spanning large scale variations, with cluttered scene layouts" that may be underrepresented in VisionMix163M. On Cityscapes, V-JEPA 2.1 ViT-G (73.5 mIoU) trails DINOv3 ViT-H+ (79.5 mIoU) and DINOv3 ViT-7B (81.1 mIoU), again showing that specialized dense-feature training (which DINOv3 explicitly optimizes) yields stronger results on heavily cluttered urban scenes. On Pascal VOC12, V-JEPA 2.1 ViT-G achieves 85.0 mIoU, very close to DINOv3 ViT-H+ (85.8 mIoU) and DINOv3 ViT-7B (86.6 mIoU), suggesting that on datasets with fewer categories and less clutter, V-JEPA 2.1 nearly matches the best image-only models.

Comparison to V-JEPA 2. The improvement over V-JEPA 2 ViT-g is dramatic across all segmentation benchmarks: ADE20K 24.4 → 47.9 mIoU (+23.5 points, a 96% relative improvement), Cityscapes 45.9 → 73.5 mIoU (+27.6 points), VOC12 63.9 → 85.0 mIoU (+21.1 points). Figure 11 provides qualitative visualization: V-JEPA 2 produces "sparse semantic masks due to the presence of noisy feature maps," while V-JEPA 2.1 yields "detailed and spatially accurate masks, capturing multi-scale structures and fine object contours with high fidelity." The depth maps in Figure 10 show the same transformation: V-JEPA 2 depth predictions lack "local consistency and precise boundary structure," while V-JEPA 2.1 produces "sharper, more coherent, and fine-grained depth maps." Figure 12 provides a more detailed comparison with DINOv3 ViT-H+ on depth estimation, showing that V-JEPA 2.1 "captures fine-grained details (i.e., the traffic light contour) and obtains better scale consistency," while DINOv3 mislabels the depth of objects in cluttered scenes (a green object on a bed and a TV set appear closer to the camera than they actually are).

Video object segmentation (Table 8). V-JEPA 2.1 ViT-G achieves 69.0 J &F on DAVIS and 72.7 J &F on YouTube-VOS, using non-parametric label propagation (no learned parameters, pure patch-similarity matching). This places it second overall behind DINOv3 ViT-H+ (71.1 on DAVIS, 74.0 on YT-VOS) and DINOv3 ViT-7B (71.1 on DAVIS, 74.1 on YT-VOS), but ahead of all other models including PEspatial-G (68.4 on DAVIS, 68.5 on YT-VOS), DINOv2 ViT-g (63.9, 65.6), and V-JEPA 2 ViT-g (52.5, 53.7). The 16.5-point DAVIS improvement over V-JEPA 2 demonstrates that the context loss and deep supervision produce features with dramatically better temporal consistency—Figure 13 shows qualitative examples where V-JEPA 2.1 "maintains consistent object segmentation masks throughout the sequence" even under "fast motion and substantial visual variations."

---

Ego4D Short-Term Object Interaction Anticipation (Table 4, Figure 6, Figure 7)

Headline numbers. V-JEPA 2.1 ViT-G achieves 7.71 mAP All on the Ego4D STA v2 benchmark, a +28% relative improvement over the previous state-of-the-art (STAformer, 5.67 mAP All) and a +35% relative improvement over the best prior method using task-specific training components (StillFast). The ViT-g variant achieves 6.75 mAP, already surpassing all prior methods.

Detailed metric breakdown (Table 4). The mAP All metric requires simultaneous correctness of bounding box (IoU > 0.5), noun category, verb category, and time-to-contact (within 0.25s). V-JEPA 2.1 ViT-G's advantage is driven by superior performance on multiple sub-metrics:

• Bounding box localization (AP b): 50.7 vs. 48.9 (DINOv3 ViT-H+), 47.8 (DINOv2 ViT-g), 45.7 (V-JEPA 2 ViT-g). The improved dense features directly enable more precise object localization.
• Box + verb (AP b+V): 25.8 vs. 23.6 (DINOv2 ViT-g), 22.2 (V-JEPA 2 ViT-g). Understanding what action will occur is where V-JEPA 2.1's temporal modeling provides the strongest advantage over image-only encoders.
• Box + time-to-contact (AP b+δ): 20.2 vs. 16.7 (V-JEPA 2 ViT-g), 14.4 (DINOv2 ViT-L). Predicting when contact will happen requires motion understanding that image encoders lack—the gap here (20.2 vs. 14.1–14.4 for image models) is large and consistent.
• Noun mAP: 30.9 for ViT-G, competitive with DINOv3 ViT-7B (33.8) and DINOv3 ViT-H+ (32.4).
• Time-to-contact mAP (mAP N+δ): 12.8 for ViT-G, substantially ahead of DINOv3 ViT-7B (10.3) and V-JEPA 2 ViT-g (10.4).

Comparison across encoder types. Table 4 groups results into "Results Reported in the Literature" (using task-specific architectures) and "Encoders using our same protocol" (applying the same attentive probe to different frozen SSL encoders). Among the probe-evaluated encoders, V-JEPA 2.1 ViT-G achieves the best performance on 6 of 8 sub-metrics, with DINOv3 ViT-7B leading only on Noun mAP (33.8 vs. 30.9) and Noun+Verb mAP (17.4 vs. 17.2). This demonstrates that V-JEPA 2.1's combination of dense spatial features (for bounding box and time-to-contact prediction) and temporal understanding (for verb prediction) outperforms image-only encoders that excel at only one of these dimensions.

Qualitative results (Figure 7). The paper shows examples where V-JEPA 2.1 predicts "multiple scenarios, where the predicted object and verb classes correspond with plausible human-object interactions"—the model correctly identifies that the user is about to interact with a specific object (e.g., a cup, a door handle) and predicts plausible action categories, with bounding boxes that tightly localize the interaction region.

---

EPIC-KITCHENS-100 Action Anticipation (Table 5)

Headline numbers. V-JEPA 2.1 ViT-G achieves 64.3 Verb Recall@5, 59.9 Noun Recall@5, and 40.8 Action Recall@5 on EK100, setting a new state-of-the-art for the task. This represents a +2.8% relative improvement in Action Recall over V-JEPA 2 ViT-g (39.7) at the same model size, and a +48% relative improvement over the previous non-V-JEPA state-of-the-art PlausiVL (27.6).

Scaling behavior. Interestingly, V-JEPA 2.1 ViT-g (1B) achieves slightly lower Action Recall than V-JEPA 2 ViT-g (38.4 vs. 39.7), with comparable Verb Recall (63.6 vs. 63.6) and lower Noun Recall (56.2 vs. 57.1). At 2B (ViT-G), V-JEPA 2.1 surpasses V-JEPA 2 ViT-g across all metrics, with the largest gain in Noun Recall (59.9 vs. 57.1). This suggests that the architectural changes in V-JEPA 2.1 (context loss, deep supervision) may slightly trade off action anticipation capability at smaller model scales, but the improved feature quality enables better scaling behavior, with ViT-G showing clear benefits. The paper does not explore why the 1B model underperforms V-JEPA 2, but the scaling trend implies that V-JEPA 2.1's innovations require sufficient capacity to avoid interference between the local and global losses.

Comparison to prior methods. Prior non-JEPA methods are far behind: PlausiVL (8B parameters) achieves only 27.6 Action Recall, Video-LLaMA (7B) achieves 26.0, and InAViT (160M) achieves 25.8. V-JEPA 2.1 ViT-G's 40.8 Action Recall is 48–63% higher than these methods, despite being evaluated with a frozen encoder and a simple attentive probe (vs. PlausiVL's 8B parameter video-language model). This dramatic gap underscores the value of SSL pretraining on video for temporal prediction tasks, where language supervision alone (as in Video-LLaMA and PlausiVL) provides much weaker signals for anticipating future actions from visual context.

---

Robot Manipulation: Real-Robot Grasping and Pick-and-Place (Table 6, Figure 8)

Headline numbers. V-JEPA 2.1 achieves 80% success rate on the Grasp task with an 8-step planning horizon, compared to 60% for V-JEPA 2 with a 1-step horizon—a +20 percentage point absolute improvement. On Pick-and-Place, both models achieve 80% with 1-step planning; the paper does not report 8-step Pick-and-Place results.

Planning budget comparison. Two planning configurations are tested: (1) 800 CEM samples, 10 iterations, 1-step horizon, ~3 seconds (matching V-JEPA 2's setup), and (2) 300 CEM samples, 15 iterations, 8-step horizon, ~14 seconds. Under configuration (1), V-JEPA 2.1 improves Grasp from 60% to 70%. Under configuration (2)—which the paper states "VJEPA 2.1 unlocks the benefit of planning with slightly longer rollouts"—Grasp improves to 80%. By contrast, the paper reports that "we actually observe a degradation in the success rate of VJEPA 2 when planning over longer horizons," though the specific degraded success rate is not provided. Reach remains at 100% for both methods, consistent with being a simpler task that doesn't require fine-grained spatial reasoning.

Failure mode analysis. The paper provides a nuanced qualitative assessment: "task failures using the VJEPA 2.1 model on Pick-and-Place and Grasp are a result of poor planning over gripper actions, as opposed to failures in spatial understanding; e.g., closing the gripper too soon such that you cannot grasp the object, or slightly opening the gripper while in transit leading to the object being dropped." This is a significant shift in error profile: V-JEPA 2 failures were partly due to spatial mislocalization (the model didn't know exactly where the object was in 3D, leading to missed grasps), while V-JEPA 2.1 failures are due to temporal action sequencing errors (the model knows where the object is but mis-times the grasp). The paper explicitly attributes this to "improved depth understanding" (Figure 8), showing qualitative energy landscapes where V-JEPA 2.1 features "better encode depth information, leading to an improvement in robot manipulation when grasping the cup involves reasoning over actions along the depth axis of the camera."

Evaluation scale caveats. The robot experiments evaluate only 10 tasks per skill, per configuration—a total of 40 grasp attempts (10 each for: V-JEPA 2 baseline, V-JEPA 2.1 1-step, V-JEPA 2.1 8-step, plus V-JEPA 2.1 Pick-and-Place). This is a very small sample, and the 80% success rate on Grasp corresponds to 8/10 successes, meaning the confidence interval is wide (exact binomial 95% CI for 8/10: ~[0.44, 0.97]). The +20% improvement claim should therefore be interpreted as suggestive rather than statistically definitive. The paper acknowledges this implicitly by reporting only aggregated percentages without error bars.

---

Navigation Planning (Table 7, Figure 9)

Headline numbers. V-JEPA 2.1 ViT-G achieves 5.687 ATE on Tartan Drive, 1.038 ATE on Scand, and 4.054 ATE on Sacson, with an average planning time of 10.6 seconds—a 9.7× speedup over NWM (Bar et al., 2025) at 103.2 seconds, while achieving slightly better trajectory accuracy (average ATE: 2.990 for ViT-G vs. 3.032 for NWM; average RTE: 0.688 for ViT-G vs. 0.696 for NWM).

Ablation across model sizes. ViT-g (1B) achieves similar performance to ViT-G (2B) on navigation: Tartan Drive ATE 5.758 vs. 5.687, average ATE 2.975 vs. 2.990, average RTE 0.690 vs. 0.688. The near-identical performance at half the parameters suggests that navigation planning, as evaluated here, may be bottlenecked by something other than representation capacity—perhaps the planning algorithm, the action space discretization, or the difficulty of the specific trajectories. The paper provides Reconstruction error ("Recon") as an additional metric (1.146 for ViT-g, 1.179 for ViT-G), which measures the world model's reconstruction quality in latent space, and this too is nearly identical across model sizes.

Why 10× faster planning. The paper attributes the speedup to V-JEPA 2.1 requiring only 8 denoising steps in the Conditional Diffusion Transformer (CDiT), compared to "at least 128 steps" for the SD-VAE used by NWM. This is possible because V-JEPA 2.1 representations are "high-dimensional (at least 80× larger than those of an SD-VAE)" and more spatially structured, so the diffusion model starts closer to the target manifold and needs fewer refinement steps. However, the higher per-step cost of operating in V-JEPA 2.1's embedding space is not separately accounted for—the 10.6 seconds is end-to-end planning time including both encoding and diffusion, so the net speedup is real.

Figure 9 qualitative analysis. PCA visualizations of 8 denoising steps show the planned latent trajectory evolving from a noisy initialization to a smooth path between the start frame and the goal frame. The qualitative impression is that V-JEPA 2.1's latent space supports clean, interpretable interpolations that require minimal diffusion steps to converge.

---

Video and Image Classification (Table 9)

Headline numbers. V-JEPA 2.1 ViT-G achieves 77.7% top-1 accuracy on SSv2 (motion-centric action recognition), 89.2% on Diving-48, 87.7% on K400, and 85.5% on IN1K (image classification). On SSv2, this sets a new state-of-the-art for frozen-backbone evaluation, surpassing V-JEPA 2 ViT-g (77.3%), DINOv3 ViT-7B (70.1%), and InternVideo2s-1B (69.7%). On appearance-focused tasks, V-JEPA 2.1 is competitive but not dominant: K400 trails InternVideo2s-1B (87.7 vs. 89.4) and DINOv3 ViT-7B (87.8); IN1K trails DINOv3 ViT-7B (88.4), SigLIP2 (88.0), PEcore G (87.6), and DINOv2 ViT-g (86.1).

Motion vs. appearance breakdown. The paper explicitly partitions the classification benchmarks into "Motion Understanding" (SSv2, Diving-48) and "Appearance Understanding" (K400, IN1K), arguing that K400 classes are largely recognizable from static frames. The results support this framing strongly:

• On motion-centric tasks, V-JEPA 2.1 ViT-G matches or exceeds all prior models: 77.7 on SSv2 (vs. 77.3 for V-JEPA 2, 74.3 for V-JEPA ViT-H, 70.1 for DINOv3 ViT-7B); 89.2 on Diving-48 (vs. 90.2 for V-JEPA 2 ViT-g, 87.9 for V-JEPA ViT-H, 86.4 for InternVideo2s-1B). The slight Diving-48 decline from V-JEPA 2 (89.2 vs. 90.2) suggests the context loss may slightly weaken the very-fine-grained temporal discrimination that Diving-48 requires (distinguishing dive types from subtle motion differences), consistent with the earlier observation in Table 1 that the context loss initially degrades SSv2.

• On appearance-centric tasks, V-JEPA 2.1 ViT-G is competitive but trails specialized image encoders: 87.7 on K400 (vs. 89.4 for InternVideo2s-1B, 88.5 for PEcore G, 87.9 for DINOv3 ViT-7B); 85.5 on IN1K (vs. 88.4 for DINOv3 ViT-7B, 88.0 for SigLIP2, 87.6 for PEcore G). The gap to DINOv3 ViT-7B on IN1K (85.5 vs. 88.4) is notable and suggests that joint image-video training does not fully match the image-only pretraining specialization that DINOv3 achieves through its teacher-student distillation on massive image datasets.

Scaling from ViT-g to ViT-G. Across all four classification benchmarks, scaling from 1B to 2B yields consistent but modest improvements: SSv2 76.9 → 77.7 (+0.8), Diving-48 89.0 → 89.2 (+0.2), K400 87.0 → 87.7 (+0.7), IN1K 84.8 → 85.5 (+0.7). The average improvement of +0.6 points across benchmarks suggests that classification accuracy is beginning to saturate at this model scale and data volume, with further gains likely requiring either more data, longer training, or task-specific architectural modifications.

Comparison to prior V-JEPA models. The trend from V-JEPA ViT-H (600M, 2024) → V-JEPA 2 ViT-g (1B, 2025) → V-JEPA 2.1 ViT-g (1B) → V-JEPA 2.1 ViT-G (2B) is instructive. V-JEPA ViT-H achieved 74.3 SSv2, 87.9 Diving-48, 84.5 K400, 80.0 IN1K. V-JEPA 2 improved to 77.3 SSv2 (+3.0), 90.2 Diving-48 (+2.3), 87.3 K400 (+2.8), 85.1 IN1K (+5.1). V-JEPA 2.1 ViT-g achieves 76.9 SSv2 (−0.4 from V-JEPA 2), 89.0 Diving-48 (−1.2), 87.0 K400 (−0.3), 84.8 IN1K (−0.3). The slight regression at 1B is consistent with the Table 1 ablation showing that the context loss initially suppresses global performance, and at 1B, deep self-supervision may not fully compensate. However, ViT-G (2B) recovers and surpasses V-JEPA 2 ViT-g on SSv2 (77.7 vs. 77.3), K400 (87.7 vs. 87.3), and IN1K (85.5 vs. 85.1), while Diving-48 remains slightly below (89.2 vs. 90.2). This pattern—regression at 1B, recovery and improvement at 2B—suggests that the architectural innovations require sufficient model capacity to resolve the local-global gradient conflict; at marginal capacity, the conflict is net-harmful to global understanding.

---

Video Question Answering (Table 10)

Headline numbers. V-JEPA 2.1 ViT-G, when paired with a Llama 3.1 8B LLM, achieves an average accuracy of 57.9 across PerceptionTest (83.1 val accuracy), MVP (43.2), TempCompass (74.0), TemporalBench (28.5), TOMATO (38.0), TVBench (62.8), and MVBench (75.4). This is competitive with but slightly below V-JEPA 2 ViT-g trained on the same filtered PerceptionLM data (57.8 average), and below the original V-JEPA 2 ViT-g results reported in Assran et al. (2025) using 88.5M alignment samples (59.5 average).

Data efficiency comparison. The paper carefully controls for training data by reproducing V-JEPA 2 with the exact same filtered PerceptionLM data (72.5M samples after quality filtering) used for V-JEPA 2.1. Under this controlled comparison, V-JEPA 2.1 ViT-G (57.9 average) edges out V-JEPA 2 ViT-g (57.8 average), with V-JEPA 2.1 showing strengths on PerceptionTest (83.1 vs. 80.1 val accuracy), MVP (43.2 vs. 40.9), and MVBench (75.4 vs. 73.3), but weaknesses on TemporalBench (28.5 vs. 32.3) and TOMATO (38.0 vs. 41.4). The paper notes that TemporalBench and TOMATO require "understanding of motion events, but on shorter videos"—suggesting that V-JEPA 2.1's spatial improvements may come at a modest cost to very-fine-grained temporal discrimination, consistent with the earlier Diving-48 and EK100 Noun Recall patterns.

Comparison to literature. V-JEPA 2.1 outperforms most open-source multimodal LLMs with ≤8B parameter backbones, including InternVL-2.5 (52.1 average), Qwen2VL (47.0 average), Qwen2.5VL (49.7 average), and PLM 8B (56.7 average). However, it trails the original V-JEPA 2 results from Assran et al. (2025), which used significantly more alignment data (88.5M vs. 72.5M samples) and a slightly different training recipe. The paper's decision to report both the original V-JEPA 2 numbers and the data-controlled reproduction is transparent, showing that data volume is a confound in comparing Video-LLM performance and that V-JEPA 2.1's encoder improvements provide modest but genuine benefits at fixed data scale.

---

Ablation Studies and Robustness Checks

Individual component contributions (Figure 5, Table 1): The paper provides a step-by-step ablation starting from V-JEPA 2 ViT-L and sequentially adding each innovation, measuring ADE20K mIoU and SSv2 accuracy as representative dense and global tasks. The trajectory: V-JEPA 2 (22.2 mIoU, 72.8% SSv2) → + Context Loss (33.8, 62.5) → + Deep Self-Supervision (38.6, 72.1) → + VisionMix163M (40.8, 72.6) → + Multi-Modal Tokenizer (41.4, 72.6) → + Model Scaling to ViT-G (47.1, 76.1) → + High-Res Cooldown (47.9, 77.7). The key observations: (1) The context loss causes a dramatic SSv2 drop that deep self-supervision almost fully recovers; (2) data scaling and model scaling produce complementary improvements; (3) the multi-modal tokenizer provides only modest gains on ADE20K (+0.6 mIoU) and none on SSv2; (4) the cooldown phase benefits SSv2 (+1.6) more than ADE20K (+0.8), though Table 13 shows depth estimation benefits most from cooldown (NYUv2 0.365 → 0.307).

Context loss weighting scheme (Table 2): The ablation sweeps fixed λ values (0.05, 0.2, 0.5, 1.0) with and without warmup, comparing against the distance-weighted scheme. Key findings: (1) λ = 0.2 constant achieves the best ADE20K among fixed values (29.6 mIoU) but crashes SSv2 to 62.5; (2) λ = 0.5 constant improves ADE20K to 27.5 but cripples SSv2 at 53.8; (3) warmup substantially mitigates SSv2 degradation (e.g., λ = 0.2 constant+warmup: 60.5 SSv2 vs. 62.5 without warmup for the same λ value—though note 62.5 > 60.5, so warmup helps at λ=0.2 but at a different operating point); (4) the distance-weighted scheme with warmup achieves the best overall trade-off (33.8 ADE20K, 62.5 SSv2), outperforming all fixed-λ variants on ADE20K while matching the best SSv2 among non-trivial λ values. The distance-weighting specifically "emphasizes patches near masked regions by enforcing local continuity between masked and context areas."

Pretraining spatial resolution (Table 13): Comparing models trained with video resolution 256 vs. 384 during cooldown shows that higher resolution improves performance on 11 of 12 reported metrics for ViT-G (the exception: YT-VOS, 72.7 → 72.6, essentially flat). The largest gains from higher resolution are on Diving-48 (89.5 → 89.2) and NYUv2 (0.313 → 0.307), suggesting that depth estimation in particular benefits from finer spatial detail, while video understanding gains are more modest. For ViT-g, the pattern is similar but less pronounced: 10 of 12 metrics improve, with DAVIS (68.0 → 67.4) and YT-VOS (72.0 → 71.3) slightly degrading—possibly because the 1B model has insufficient capacity to fully exploit the higher-resolution input.

Deep self-supervision enables single-layer evaluation (Table 12, Appendix D.1): This ablation tests whether deep self-supervision reduces the need for multi-layer feature extraction in downstream tasks. Without deep supervision, using only the last layer (vs. 4 layers) degrades ADE20K from 39.1 to 34.9 mIoU (a 4.2-point gap), NYUv2 RMSE from 0.463 to 0.513 (0.050 worsening), and Diving-48 from 86.9 to 85.8%. With deep supervision, the gap shrinks dramatically: ADE20K 43.9 vs. 42.0 (1.9-point gap), NYUv2 0.370 vs. 0.381 (0.011 worsening), Diving-48 88.1 vs. 87.2 (0.9-point gap). The conclusion is that deep self-supervision propagates spatial information forward through the network, so the final layer already contains rich local features and intermediate layers don't need to be separately extracted for good dense-task performance.

Multi-modal tokenizer without data scaling (Table 1): At the ViT-L scale with VisionMix163M already applied, adding the multi-modal tokenizer improves ADE20K from 40.8 to 41.4 mIoU (+0.6) and NYUv2 from 0.418 to 0.415 RMSE (−0.003), while SSv2 and IN1K are unchanged. The paper interprets this as the tokenizer having a "positive effect on dense-task performance" while global tasks remain stable. This is a small effect but directionally consistent—processing images natively without temporal duplication may provide slightly better spatial features from the image modality.

Scaling from ViT-g to ViT-G (Table 13, comparison at 384 resolution): At 384 resolution, the 2× parameter increase from ViT-g to ViT-G improves SSv2 76.9 → 77.7 (+0.8), Diving-48 89.0 → 89.2 (+0.2), K400 87.0 → 87.7 (+0.7), IN1K 84.8 → 85.5 (+0.7), EK100 38.4 → 40.8 (+2.4), ADE20K 47.8 → 47.9 (+0.1), Cityscapes 71.8 → 73.5 (+1.7), VOC 84.7 → 85.0 (+0.3), NYUv2 0.350 → 0.307 (−0.043), KITTI 2.601 → 2.461 (−0.140), DAVIS 67.4 → 69.0 (+1.6), YT-VOS 71.3 → 72.6 (+1.3). The largest gains are on EK100 action anticipation (+2.4 Recall@5), KITTI depth (−0.140 RMSE), and Cityscapes segmentation (+1.7 mIoU)—tasks requiring either fine-grained temporal prediction or fine-grained spatial understanding. The near-zero gain on ADE20K (+0.1) suggests that ADE20K's difficulty for V-JEPA 2.1 is dataset-specific (cluttered scene layouts, many small object classes) rather than capacity-limited.

Distillation performance (Section 3.10, Table "??" — reference broken in paper): The text reports that distilled ViT-L "almost closes the gap with ViT-G performance": SSv2 74.2 → 76.5 (scratch ViT-L → distilled ViT-L, vs. 77.7 for ViT-G), ADE20K 42.0 → 46.7 (vs. 47.9 for ViT-G), KITTI 2.490 (vs. 2.461 for ViT-G), DAVIS 68.7 (vs. 69.0 for ViT-G). The distilled ViT-B is described as offering "competitive performance with our ViT-L trained from scratch," though no specific ViT-B numbers are provided. The takeaway is that distillation effectively transfers the ViT-G's representation quality to smaller models, particularly for depth estimation and video object segmentation where the gap to ViT-G is negligible.

---

Critical Assessment

Does V-JEPA 2.1 genuinely achieve state-of-the-art performance across both dense and global tasks?

The paper's central claim is that V-JEPA 2.1 simultaneously excels at dense spatial tasks and global semantic tasks—a capability no prior model demonstrated. The evidence supporting this is strong but requires careful qualification:

For depth estimation (dense): V-JEPA 2.1 ViT-G achieves 0.307 RMSE on NYUv2, which is genuinely state-of-the-art for linear-probe evaluation among all models regardless of size, surpassing even DINOv3 ViT-7B (0.309). This is a clean, unambiguous win. On KITTI, the 2.461 RMSE is best among models under 2B parameters, though DINOv3 ViT-7B achieves 2.346—so V-JEPA 2.1 is state-of-the-art conditional on model scale. The qualitative results in Figure 12 show V-JEPA 2.1 capturing fine details that DINOv3 misses, lending credibility to the numbers.

For semantic segmentation (dense): V-JEPA 2.1 ViT-G achieves 47.9 mIoU on ADE20K, which is competitive but clearly behind DINOv3 ViT-H+ (54.8) and DINOv3 ViT-7B (55.9). On Cityscapes, the gap is even larger (73.5 vs. 79.5/81.1). The claim "state-of-the-art" for the paper's overall positioning does not apply to segmentation specifically—V-JEPA 2.1 is in the top tier but not the leader. The paper's hypothesis that VisionMix163M lacks sufficient cluttered indoor/urban scenes is plausible but untested; an ablation adding more segmentation-like data during pretraining would have strengthened this argument.

For video object segmentation (dense): V-JEPA 2.1 achieves 69.0 J &F on DAVIS and 72.7 on YT-VOS, second overall behind DINOv3. This is genuinely state-of-the-art for a video-pretrained model (InternVideo2-1B achieves only 50.6/51.2) but not for image-pretrained models. The gap to DINOv3 is small enough (2.1 J &F on DAVIS) that the claim of "competitive" is fair.

For action recognition (global): V-JEPA 2.1 ViT-G achieves 77.7% on SSv2, which is the highest frozen-backbone result reported. This is a clear state-of-the-art for video understanding. On Diving-48 (89.2), V-JEPA 2.1 slightly trails V-JEPA 2 (90.2) but still leads all non-JEPA methods. On K400 (87.7), it trails InternVideo2s-1B (89.4) and several image models—but the paper correctly argues that K400 is primarily appearance-based, making it a less discriminating benchmark for video understanding. Taken together, V-JEPA 2.1 is state-of-the-art on motion-centric recognition and competitive on appearance-centric recognition.

For anticipation (predictive): On Ego4D STA (7.71 mAP All), V-JEPA 2.1 sets a new absolute state-of-the-art by a wide margin (+35% over prior best task-specific method). On EK100 action anticipation (40.8 Recall@5), it also sets a new state-of-the-art, though at 1B scale it slightly underperforms V-JEPA 2 (38.4 vs. 39.7). These are the most demanding benchmarks in the paper—they require both dense spatial understanding and temporal prediction—and V-JEPA 2.1's dominance on them is the strongest evidence for the paper's core claim.

The gap between linear-probe and fine-tuned evaluation matters. All dense-task results use linear probes—a single linear layer on frozen features. This protocol directly tests feature quality rather than the model's ability to adapt, which is a strength for evaluating SSL representations. But the absolute numbers are far below what fine-tuned or task-specific methods achieve (e.g., state-of-the-art NYUv2 depth estimation with task-specific training is well below 0.3 RMSE). The paper does not report fine-tuned results, so we cannot assess whether V-JEPA 2.1's advantages persist when the encoder is allowed to adapt. The robot manipulation results (80% grasping) are a notable exception—they use frozen features and demonstrate real-world competence, suggesting the linear-probe performance is not just a laboratory metric but translates to embodied tasks.

The missing combination: PRM search + revisions is not tested here but the analogy holds. Section 3 of the example analysis discusses how V-JEPA 2 studies search and revisions independently without combining them. For V-JEPA 2.1, the analogous missing combination is multi-resolution feature extraction: the paper evaluates downstream tasks using only the final layer (which deep self-supervision makes sufficient) but does not test whether combining multiple layers in a more sophisticated way (e.g., feature pyramids, cross-scale attention) would close the remaining gap to DINOv3 on segmentation. Given that Table 12 shows a small residual benefit from using 4 layers even with deep self-supervision, a more elaborate multi-scale readout might recover additional performance—but this is not tested.

Statistical reliability concerns. The test set sizes vary dramatically: NYUv2 has 654 test images, ADE20K has 2,000 validation images, Ego4D STA has 47,395 validation samples, EK100 has ~9,000 validation action segments. For the smaller test sets (NYUv2, DAVIS with 30 validation videos), the reported RMSE and J &F differences of 0.002 or 0.1 may not be statistically significant. The paper reports no confidence intervals, standard errors, or significance tests anywhere. For a paper claiming state-of-the-art based on fraction-of-a-point improvements on some benchmarks, this is a notable omission. The robot experiments (10 trials per condition) are particularly vulnerable to sampling error—the 80% vs. 60% comparison represents 8/10 vs. 6/10 successes, where a single outcome change would alter the reported improvement from +20% to +10% or +30%.

The single-seed, single-training-run problem. All results come from a single pretraining run per model configuration. There is no analysis of how pretraining seed variation affects downstream performance—a concern because SSL objectives can exhibit substantial run-to-run variance. A model that achieves 77.7% on SSv2 in one run might achieve 77.3% or 78.1% in another, which could change the state-of-the-art claim. This is a widespread practice in the SSL literature (DINOv2, DINOv3, V-JEPA 2 all report single-run results), but it weakens the precision of the claimed improvements, especially when margins are small.

Baseline fairness—is the comparison to DINOv3 representative? The paper compares V-JEPA 2.1 ViT-G (2B parameters) to DINOv3 ViT-7B (7B parameters) on many tasks, which is generous to V-JEPA 2.1. However, DINOv3 ViT-H+ (0.8B parameters) is the closer capacity match, and on that comparison V-JEPA 2.1 leads on depth estimation (0.307 vs. 0.352 RMSE on NYUv2) and video tasks (by large margins), but trails on semantic segmentation (47.9 vs. 54.8 mIoU on ADE20K) and video object segmentation (69.0 vs. 71.1 J &F on DAVIS). The paper is transparent about this—Table 8 includes both DINOv3 ViT-H+ and ViT-7B—and the interpretation that V-JEPA 2.1 leads on depth and video while DINOv3 leads on segmentation is well-supported.

Data scale confound. VisionMix163M includes 142M curated images from LVD-142M, which is the same dataset DINOv2 used for pretraining. However, DINOv3 was trained on an even larger and more diverse dataset (the paper does not disclose the exact composition, but DINOv3's data pipeline is different). The comparison between V-JEPA 2.1 and DINOv3 is therefore not data-matched—DINOv3 may have seen more or better-curated data for the specific dense tasks where it excels. The paper does not control for this, and it's a genuine confound in attributing performance differences to architecture vs. data.

Missing ablations:

• The 24-block predictor vs. 12-block: Section 3.2 notes that the predictor was increased from 12 to 24 blocks. How much of the improvement from deep self-supervision comes from simply having a larger predictor? An ablation with deep self-supervision but a 12-block predictor would disambiguate this.
• The MLP fusion architecture: The concatenation+MLP that compresses multi-level representations is described as "lightweight" but never ablated. What happens with simple averaging instead of concatenation? With a larger or smaller MLP? Without the MLP (just using the concatenated 4d-dimensional tokens directly as predictor input)?
• The number of deep supervision levels: The paper uses 4 levels. What about 2 levels? 8 levels? All layers? The choice of 4 is presented as a fixed design but never justified through ablation.
• The distance-weighting function: λ_i = λ / √d_min uses an inverse square root. What about λ / d_min (inverse linear)? λ / d_min^2? A constant weight within a radius and zero outside? The specific functional form is presented without comparison to alternatives.
• The 1B → 2B scaling with V-JEPA 2's architecture for reference: How much of the ViT-G improvement is from the architectural innovations vs. simply having a larger model with V-JEPA 2's original recipe? Training a V-JEPA 2 ViT-G (with the original mask-only loss, single-level prediction) on VisionMix163M would cleanly separate scaling from architecture, but this baseline does not exist.

Summary of evidential strength. The paper's headline claim—that V-JEPA 2.1 achieves state-of-the-art across dense and global tasks—is strongly supported for depth estimation (genuine SOTA) and video understanding (genuine SOTA), partly supported for semantic segmentation (competitive but not SOTA), and well-supported for embodied tasks (though with small sample sizes). The core architectural insight—that supervising context tokens unlocks dense features—is convincingly demonstrated through the comprehensive Table 1 ablation, the qualitative PCA visualizations (Figures 1, 3, 14, 15), and the dramatic 52% reduction in depth error from V-JEPA 2 to V-JEPA 2.1. The weaker claim—that V-JEPA 2.1 is the "first model competitive across the full spectrum"—is well-supported in that no prior model appears in the top tier of both depth estimation and action anticipation simultaneously, but "competitive" is doing important work given the ADE20K and Cityscapes gaps to DINOv3. The robot results, while compelling in direction, rest on too few trials to quantify the improvement reliably. A rigorous evaluation would require multiple pretraining seeds, confidence intervals on all downstream metrics, and a data-matched comparison to DINOv3—none of which is provided.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Efficiency Claims

The assumption or constraint. The paper's entire compute-optimal framework rests on a prompt-difficulty estimate that determines which test-time strategy to use. The difficulty estimation procedure — generating 2,048 samples per question, scoring them with the PRM (or ground-truth correctness for the oracle variant), and binning into quintiles — is extraordinarily expensive. As the authors acknowledge in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is not a minor caveat. At 2,048 samples per question, the difficulty estimation step alone consumes 8× more compute than the largest test-time budgets studied (256 generations) and 128× more than the budgets where the 4× efficiency gains are claimed (16 generations matching 64). The total cost of using this method is therefore difficulty estimation cost + strategy execution cost, but the paper only reports the strategy execution cost.

The consequence. The headline 4× efficiency improvement over best-of-N (Figures 4 and 8) is an upper bound that cannot be realized in deployment without a cheaper difficulty estimator. In a production system, the difficulty estimation cost would dominate the total compute budget, potentially making the compute-optimal approach more expensive than simply running best-of-N with the full budget on every question. To make this concrete: if the average test-time budget is 64 generations per question and difficulty estimation costs 2,048 generations, the total cost is 2,112 generations — which could have been spent on a best-of-1024 baseline that might outperform the compute-optimal strategy with difficulty overhead.

Furthermore, the 2,048-sample estimation cost is per-question and non-amortizable — you cannot reuse it across questions because difficulty is prompt-specific. In a batch inference setting with a million questions, the total cost would be ~2 billion generations just for difficulty estimation, before any actual problem-solving occurs.

What evidence exists in the paper. The paper explicitly flags this gap in Section 3.2. The distinction between oracle and predicted difficulty bins (Figures 4 and 8) shows that ground-truth labels are not required, but the predicted approach still requires the full 2,048-sample generation and PRM-scoring pipeline — it simply substitutes PRM scores for correctness labels. The paper also acknowledges that this creates an "exploration-exploitation tradeoff — compute spent assessing difficulty versus compute spent solving the problem." However, no experiment measures how much performance degrades if you reduce the estimation budget (e.g., using 64 samples instead of 2,048), nor does the paper provide any cost-amortized efficiency curve that includes the estimation overhead.

Mitigation status. The paper does not address this limitation beyond flagging it for future work: "a key avenue for future work" is developing cheaper difficulty estimators, possibly via "pretraining or finetuning models to directly predict difficulty of a question" (Section 3.2, Section 8). There is no attempt to train such a model, to evaluate performance with fewer estimation samples, or to report cost-inclusive efficiency numbers. Until this gap is closed, the computational efficiency claims should be understood as conditional on having an oracle difficulty estimator — a condition that does not hold in practice.

---

Hard Problems Remain Fundamentally Unsolved

The assumption or constraint. The paper's approach assumes that the base LLM's proposal distribution contains at least some correct solutions — i.e., pass@1 on a given problem is non-trivially above zero. If the model's pass@1 is near zero, no amount of search or revision can find a correct solution, because none exists in the set of generated candidates. The paper is explicit about this boundary:

"test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine." (implicit in Section 7)

The consequence. Across all methods studied — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and budgets up to 256 generations. In Figure 7 (right), bin 5 accuracy is roughly 2–3% for all sequential-to-parallel ratios at 128 generations. In Figure 9 (the FLOPs-matched comparison), the bin 5 scaling curves are essentially flat near 0–5% for both revisions and search, and increasing the budget does not move them. This means that for the hardest ~20% of MATH questions (by the paper's quintile binning), no amount of inference-time compute helps.

This is not merely a quantitative limitation — it is a fundamental capability boundary. The model cannot solve problems that exceed its pretrained knowledge and reasoning capacity, regardless of how cleverly you allocate test-time compute. For deployment scenarios where the problem distribution is heavy-tailed toward hard problems (e.g., advanced scientific reasoning, competition-level mathematics, open-ended research problems), the approach offers no path forward. The paper's own FLOPs-matched analysis (Section 7) confirms that on hard problems, a 14× larger model almost always outperforms test-time compute with the smaller model, regardless of the R ratio.

What evidence exists in the paper. The bin 5 results are consistently flat across all figures and methods. In Figure 3 (right), the beam search and best-of-N curves for bin 5 are statistically indistinguishable from zero improvement across all budgets. In Figure 7 (right), bin 5 shows no meaningful sensitivity to the sequential-to-parallel ratio. In the FLOPs-matched comparison (Table in Section 7, bar charts in Figure 1), hard problems show negative or near-zero relative improvement from test-time compute across all R regimes (e.g., −52.9% for PRM search at R ≫ 1). The paper does not hide this — the discussion in Section 7 explicitly states that "test-time compute provides essentially zero benefit regardless of budget" on bin 5 — but it also does not explore whether a larger base model with the same compute-optimal test-time strategies would solve these problems, or whether the boundary is inherent to the approach or specific to PaLM 2-S*'s capabilities.

Mitigation status. The paper acknowledges this limitation candidly in Section 7: "test-time compute amplifies existing capability but does not create it from nothing." It positions the capability boundary as a defining characteristic rather than a bug, noting that "for such problems, pretraining remains the only viable path." No attempt is made to push this boundary — for instance, by combining test-time compute with retrieval-augmented generation, tool use, or other mechanisms that might expand the effective capability range at inference time without full pretraining. The boundary is treated as a given rather than a target for mitigation.

---

The FLOPs-Matched Baseline Is Not Compute-Optimal, Weakening the Pretraining-vs-Inference Tradeoff Claim

The assumption or constraint. The paper's central comparison between test-time compute and pretraining compute (Section 7) uses a baseline where model parameters are scaled by ~14× while training data is held fixed, following the LLaMA paradigm rather than compute-optimal pretraining as established by Hoffmann et al. (2022). The authors acknowledge this directly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding — no majority voting, no best-of-N, no search. This means the pretraining baseline does not benefit from any test-time compute, even though the smaller model gets compute-optimal allocation of up to hundreds of generations.

The consequence. Both of these choices make the pretraining baseline weaker than it could be:

1. Data scaling: A Chinchilla-optimal pretraining run with 14× more total FLOPs would allocate some fraction of the budget to additional training data rather than all to parameters. Hoffmann et al. (2022) showed that data-limited scaling of parameters alone yields diminishing returns compared to balanced scaling. The 14× larger model may therefore be under-trained relative to what a compute-optimal allocation would produce, making test-time compute look more favorable in the comparison.

2. No inference augmentation: The larger model gets greedy decoding — a single forward pass per prompt. The smaller model gets up to hundreds of generations plus a trained verifier (PRM or ORM). This is not a FLOPs-matched comparison between two inference strategies; it is a comparison between a strong inference strategy (compute-optimal test-time compute) and a weak inference strategy (greedy). A fairer comparison would give the larger model a proportional test-time budget — e.g., if the smaller model gets 64 generations, the larger model might get 64/m generations where m accounts for its higher per-token cost. The paper never tests this.

The net effect is that the reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1 for revisions) likely overstate the true benefit relative to a properly optimized pretraining baseline. The paper's own sensitivity analysis across R values (0.16, 0.79, 22) shows that the advantage is highly sensitive to the inference-to-pretraining ratio — but the baseline model's suboptimal pretraining allocation adds an unquantified bias across all R values.

What evidence exists in the paper. The paper's FLOPs accounting (Section 7) is explicit and transparent about the formulas used and the assumption of fixed training data. The authors cite Hoffmann et al. (2022) as the reference for compute-optimal pretraining, making clear that they are aware of the deviation. The results in Figure 9 show that even with this weak baseline, test-time compute loses to pretraining on hard problems and at high R values, which somewhat mitigates concern that the comparison is entirely one-sided. However, no ablation tests the sensitivity to giving the larger model even a modest test-time compute budget, which is the most direct way to assess how much the asymmetric evaluation protocol matters.

Mitigation status. The paper explicitly defers this to future work: "leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." There is no partial mitigation — no test of the larger model with best-of-N, no re-analysis under Chinchilla-optimal assumptions, and no bounding argument for how much the results might change. This is a genuinely open question that the paper identifies but does not resolve, and it directly affects the practical interpretation of the pretraining-vs-inference tradeoff results.

---

Sequential Revisions Introduce a Serial Latency Bottleneck Not Accounted For in Efficiency Metrics

The assumption or constraint. The paper measures test-time compute in generations — the number of complete solutions sampled — treating each generation as equivalent regardless of whether it is produced in parallel or sequentially. However, the revision model (Section 6) generates chains of sequential revisions where each revision depends on the previous one: the model conditions on its own previous answer to produce an improved answer. This is inherently serial — you cannot generate revision t+1 until revision t is complete.

The consequence. Wall-clock latency and GPU throughput diverge sharply between the parallel and sequential sampling regimes. A strategy that allocates 64 generations as 8 parallel chains × 8 sequential revisions takes roughly 8× longer in wall-clock time than generating 64 parallel samples simultaneously — even though both have the same FLOPs count. The compute-optimal policy (Section 6, Figure 7) favors higher sequential-to-parallel ratios for easy-to-medium problems, which are precisely the problems where latency-sensitive applications might want fast responses. For example, at 64 generations, the optimal ratio for medium problems is roughly 2:1 to 8:1 sequential-to-parallel, meaning the system spends 64–89% of its wall-clock time on sequential revisions that cannot be parallelized.

This matters because the efficiency metric (generations) is a good proxy for total FLOPs but a poor proxy for time-to-answer. In latency-sensitive applications — interactive assistants, real-time decision-making, on-device inference where processing is sequential anyway — the sequential-heavy strategies favored by the compute-optimal policy may be impractical regardless of their accuracy advantages. A user waiting for an answer cares about wall-clock time, not FLOPs.

The paper also does not account for the memory overhead of maintaining revision chains — each revision requires keeping the entire previous context in the attention window, which grows linearly with chain length. For a chain of 16 revisions, the context length is 17× the base prompt length, which may exceed the model's pretraining context window or cause quadratic attention-cost blowup that is not captured by the linear generation-count metric.

What evidence exists in the paper. The paper does not measure or discuss latency or wall-clock time anywhere. All cost accounting is in "generations." Figure 7 (left) sweeps the sequential-to-parallel ratio for a fixed generation budget, showing that higher sequential ratios improve accuracy on easy-medium problems, but there is no corresponding wall-clock-time curve showing how much longer those configurations take. The revision model architecture (Figure 5, Section 6.1) clearly shows the sequential dependency: each revision conditions on previous answers. The paper acknowledges the correct-to-incorrect reversion problem (approximately 38% of correct answers get revised back to incorrect ones) but not the latency problem.

Mitigation status. The paper does not address this limitation at all. There is no discussion of latency, throughput, or the tradeoff between FLOPs-efficiency and wall-clock-time efficiency. The compute-optimal policy is optimized for FLOPs minimization, not latency minimization, and the paper provides no guidance for practitioners who care about the latter. This is a particularly notable omission given that the parallel-sampling baselines (best-of-N, majority voting) are embarrassingly parallelizable, making them far more latency-friendly than the revision-heavy strategies that the compute-optimal policy favors on easy problems. Future work could explore adaptive strategies that switch between parallel and sequential based on a latency budget as well as a FLOPs budget, but the current paper treats them as interchangeable.

---

No Combination of PRM Search With Revisions Is Tested, Leaving the Full Potential Unexplored

The assumption or constraint. The paper studies two complementary test-time compute mechanisms — PRM-guided search (Section 5) and iterative revisions (Section 6) — but studies them independently, never combining them into a unified test-time strategy. The search experiments always use the base few-shot prompted LLM as the proposal distribution; the revision experiments always use the fine-tuned revision model with simple parallel-or-sequential sampling and no tree search. Section 8 acknowledges this explicitly:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. This is a significant gap because the two mechanisms have complementary, difficulty-dependent strengths that the paper itself documents. Revisions (modifying the proposal distribution) are most effective on easy problems where the model's initial output is roughly correct and needs refinement — a local search in answer space. PRM-guided beam search (modifying how outputs are selected) is most effective on medium-hard problems where the model needs to explore qualitatively different solution strategies — a global search. The paper's compute-optimal policy assigns different strategies to different difficulty bins, but it never lets a single problem benefit from both a revised proposal distribution and PRM-guided search simultaneously.

The potential gains from combination are not merely additive. Consider: beam search over revision model outputs could explore revision branches that the model would not discover in a linear chain, while using the PRM to score which revisions are promising could prevent the correct-to-incorrect reversion problem (38% of correct answers in a revision chain get wrongly revised to incorrect answers — Section 6.1). If the PRM could detect when the current answer is already correct and stop the revision chain, the effective performance of sequential revisions might be substantially higher than what Figure 6 (right) reports. Similarly, using the revision model as the proposal distribution within beam search — where at each step, the model conditions on previous (possibly rejected) branches — could generate higher-quality candidates than the base model's few-shot outputs, potentially improving all search methods.

What evidence exists in the paper. The paper provides strong evidence that both mechanisms work individually and that their effectiveness varies with difficulty (Figures 3 right, 7 right), but provides zero evidence about their combination. The revision model's correct-to-incorrect reversion rate (38%, Section 6.1) is never tested with PRM-based stopping — we know the problem exists but not whether the PRM can solve it. The paper's compute-optimal policy selects between search strategies and revision ratios but never combines them hierarchically (e.g., use beam search to explore different high-level approaches, then use revisions to refine the best candidates). All experiments in Figures 4 and 8 compute-optimally allocate within the search or revision pipeline separately, but no experiment allocates across both simultaneously.

Mitigation status. The paper explicitly lists this as future work in Section 8, but provides no preliminary results, theoretical analysis, or architectural sketch for how the combination would work. The current results therefore represent a lower bound on what a fully integrated system could achieve — but we cannot estimate how far that lower bound is from the true potential. Given that both mechanisms individually achieve 4× efficiency gains over best-of-N (Figures 4 and 8), and that their strengths are complementary (revisions for local refinement on easy problems, search for global exploration on medium problems), a combined system might exceed the 4× figure, but this is speculative without experiments.

---

All Results Are on a Single Benchmark and Model Family, With No Evidence of Generalization

The assumption or constraint. Every experiment in the paper — search mechanisms, revision models, FLOPs-matched comparisons, all ablation studies — uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The paper states (Section 4) that it "believe[s] this model is representative of the capabilities of many contemporary LLMs," but no evidence is provided to support this belief through replication on other models or tasks. MATH consists of high-school competition-level math problems requiring symbolic reasoning and exact-answer matching — it does not include code generation, logical reasoning, scientific QA, open-ended generation, or any domain where correctness is not reducible to a final-ground-truth string.

The consequence. Several aspects of the paper's findings could be model-specific or domain-specific in ways that would alter the practical guidance for practitioners:

1. PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution, calibration, and error patterns. A model with different properties — higher pass@1 but poorly calibrated verifier scores, or different mistake distributions — might exhibit different difficulty-dependent scaling curves and a different over-optimization threshold.

2. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities. PaLM 2-S* is a strong model from a family with demonstrated in-context learning ability. Weaker or differently-trained models might not learn to revise effectively from the same training data recipe.

3. The difficulty-dependence of strategy effectiveness is the paper's central finding, but difficulty is defined relative to the base model's capabilities. A stronger model would shift all problems toward the "easy" end of the distribution, where the paper shows that sequential revisions dominate and beam search can be counterproductive. A weaker model would shift them toward "hard," where nothing works. The optimal allocation policy is therefore model-dependent, but the paper provides no guidance for how it generalizes.

4. MATH requires exact-answer matching, which enables clean correctness oracles for difficulty estimation, PRM training via Monte Carlo rollouts, and verifier evaluation. Many practical tasks — dialogue, creative writing, open-ended QA, planning — lack such clean binary correctness signals, making the entire PRM-training and difficulty-estimation pipeline inapplicable without fundamental modifications.

5. The PRM over-optimization phenomenon (Figure 3 right: beam search hurts on easy problems at high budgets) might be specific to the PRM's training on MATH-style solutions. Different domains might have different verifier reliability profiles, and the "over-optimization threshold" that the compute-optimal policy implicitly navigates might occur at very different budget levels or not at all.

What evidence exists in the paper. Zero. The paper includes no experiments on any other benchmark, model, or domain. The test set consists of 500 MATH questions, split into five difficulty quintiles of ~100 each, further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin — a small sample that may not produce robust policy choices even within the MATH distribution. The paper does not test on MATH sub-categories (algebra, geometry, probability, etc.) to assess whether the difficulty-dependent patterns hold within topic domains.

Mitigation status. The paper does not address this limitation beyond stating the belief that PaLM 2-S* is "representative." No experiments with other models (GPT, Claude, LLaMA, Gemini), other benchmarks (GSM8K, HumanEval, MMLU, ARC), or other task types (code generation, logical reasoning, factuality) are reported, nor are any planned. The paper's title and framing position it as a general study of test-time compute scaling, but the evidence is entirely from a single narrow domain and model. This is the most significant barrier to practitioners adopting the specific recommendations (use beam search on medium problems, revisions on easy problems, best-of-N weighted on hard problems) — they have no way of knowing whether their model and task distribution will produce the same difficulty-dependent patterns that make the compute-optimal approach beneficial.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood not as a new architecture or a new objective function, but as a diagnostic correction to the JEPA family of models — and by extension, to the broader class of mask-denoising SSL approaches — that demonstrates that dense spatial features and global semantic understanding are not fundamentally opposed forces but can coexist when the training signal is distributed correctly across the model hierarchy and across spatial positions.

The magnitude of this shift is medium-to-large for video SSL, incremental for image SSL. For the V-JEPA lineage specifically, this is a transformative correction: V-JEPA 2 was essentially unusable for dense prediction tasks (22.2 mIoU on ADE20K, 0.642 RMSE on NYUv2), and V-JEPA 2.1 makes it competitive or state-of-the-art on those same benchmarks (47.9 mIoU, 0.307 RMSE) while preserving its video understanding strengths (77.7% on SSv2). This transforms V-JEPA from a specialist model into a generalist one, which directly impacts its deployment potential. For image SSL, the shift is more incremental: DINOv3 still leads on semantic segmentation and video object segmentation, and the paper does not claim to surpass it on those tasks. The contribution there is demonstrating that video-pretrained models can now compete with image-pretrained models on dense tasks — something no prior work had shown.

The conceptual reframing the paper provides is a unified explanation for a previously contradictory pattern of results. Prior to this work, the field had two sets of findings that seemed at odds: (1) JEPA-based video models achieved excellent action recognition and temporal prediction but poor spatial localization; (2) DINO-based image models achieved excellent spatial localization but poor temporal understanding. The implicit assumption was that these reflected an inherent trade-off — that objectives which learn temporal dynamics must sacrifice spatial precision, and vice versa. The paper's diagnosis (that the absence of spatial structure in JEPA features is a consequence of the asymmetric mask-only loss, not an inherent limitation of predictive objectives) and its resolution (that deep self-supervision + distance-weighted context loss can resolve the local-global gradient conflict) reframes this as a training-signal design problem rather than a fundamental representational tension. An objective that applies loss to all tokens, at multiple representational depths, with spatially-aware weighting, can produce features that are simultaneously spatially structured and temporally predictive.

This reframing has two immediate consequences for the research landscape:

Directions that become more attractive:

• Applying context-token supervision to other masked-prediction models. The diagnosis that mask-only losses cause context tokens to collapse into global aggregators is not specific to JEPA — it applies to any masked autoencoder that computes loss only on masked positions. VideoMAE (Tong et al., 2022), MaskFeat (Wei et al., 2021), and related approaches likely suffer from the same pathology. A direct replication of the context-loss experiment in those frameworks would test the generality of the diagnosis.
• Hierarchical loss design for multi-task SSL. The paper's finding that deep self-supervision resolves gradient conflicts by letting different encoder layers specialize toward different aspects of the loss (local spatial structure at early layers, global semantics at later layers) suggests a general principle: when objectives conflict, apply supervision at multiple representational depths rather than forcing a single representation to serve all objectives. This is portable to other SSL objectives (e.g., combining contrastive and predictive losses, combining image and text alignment with video prediction).
• Leveraging video models for robotic perception. The robot manipulation results (80% grasping with V-JEPA 2.1 vs. 60% with V-JEPA 2) provide concrete evidence that better dense features from video pretraining directly improve embodied task performance. This makes the case for investing in video SSL as a pretraining strategy for robotics, where spatial precision matters as much as semantic understanding. The navigation results (10× faster planning with V-JEPA 2.1 features) provide a second data point: better representations reduce downstream computation, not just improve accuracy.

Directions that become less pressing:

• Specialized dense-feature losses for video models. Prior to this work, one might have assumed that getting dense features from video models required adding explicit dense-prediction objectives (e.g., depth estimation, optical flow, segmentation) during pretraining — essentially turning video SSL into multi-task supervised learning. The paper shows that a relatively simple modification to the standard JEPA loss (context supervision + deep self-supervision) is sufficient, making specialized losses less necessary.
• Pure image-pretraining for dense tasks when video data is available. The paper demonstrates that a video-pretrained model can match or surpass image-pretrained models on depth estimation (0.307 vs. 0.309 RMSE for DINOv3 ViT-7B) while dramatically outperforming them on temporal tasks. For practitioners who have access to both images and videos, jointly pretraining on both modalities now offers a clearer win than pretraining on images alone, even if the downstream task is purely image-based.

---

Follow-Up Research This Work Enables

Does the context-loss diagnosis generalize to other masked autoencoder architectures? The paper's central hypothesis — that mask-only losses cause context tokens to collapse into global aggregators, suppressing spatial structure — is demonstrated only for the V-JEPA family (ViT, JEPA objective, soft-target L1 loss). VideoMAE (Tong et al., 2022) uses a different objective (pixel reconstruction with MSE loss), a different architecture (encoder-decoder rather than encoder-predictor), and different masking (higher masking ratios, typically 90%). Would adding a context reconstruction loss to VideoMAE similarly improve its dense features? The diagnostic framework predicts yes — the asymmetry exists in VideoMAE too, since the decoder's output at visible positions is not supervised — but the pixel-space reconstruction target might change the dynamics, because reconstructing pixel values at visible positions is trivially easy (copy the input), potentially leading to a different failure mode (the model ignores context reconstruction because it provides no useful learning signal) rather than the gradient conflict observed in JEPA. A clean experiment would train VideoMAE ViT-L with and without context-token reconstruction loss on Something-Something-v2, then evaluate on NYUv2 depth and ADE20K segmentation with the same linear-probe protocol used in V-JEPA 2.1. If context supervision helps VideoMAE too, the diagnosis would be validated as a general property of masked autoencoders; if it doesn't, the diagnosis would be refined as specific to latent-space prediction, where avoiding trivial solutions requires more careful gradient management.

Can you learn difficulty without 2,048 samples per question? The paper's most significant practical limitation is the cost of difficulty estimation — generating 2,048 samples per question, scoring them with the PRM, and binning into quintiles costs more compute than the problem-solving budget itself. This is flagged as future work in Section 3.2 and Section 8, but the paper provides no data on how much the estimation budget can be reduced. A natural follow-up would be to measure the correlation between difficulty bin assignments using 2,048 samples and assignments using 64, 128, 256, and 512 samples. If the correlation is high at 64–128 samples, the estimation overhead drops from 8×–128× the problem-solving budget to roughly 1×–2×, making the compute-optimal approach genuinely practical. A second approach is to train a lightweight difficulty predictor: use the PRM's average final-answer scores from the 2,048-sample oracle as training labels, and train a small classifier (e.g., a linear layer or lightweight transformer) on top of the base LLM's [CLS] token to predict the difficulty bin directly from the question text without any sampling. The paper's cross-validation setup (two-fold, 500 questions) provides a natural evaluation framework: train the difficulty predictor on one fold, evaluate compute-optimal strategy selection on the other, and compare to oracle-bin performance from Figure 4. A third approach is adaptive estimation: start with 8 parallel samples, compute the PRM score distribution, and use it as a coarse difficulty signal to decide whether to allocate more budget to estimation or problem-solving. This connects to the multi-armed bandit literature and could be evaluated by measuring the accuracy-vs-total-cost curve (including estimation cost) compared to the fixed-2048-sample approach.

What is the marginal value of each deep-supervision level, and can the MLP fusion be improved? The paper uses four equally-spaced encoder levels with a concatenation+MLP fusion but provides no ablation on either the number of levels or the fusion architecture. A systematic study would train ViT-L models with 1, 2, 4, 8, and all levels (48 for ViT-G), measuring NYUv2 depth and SSv2 accuracy at each configuration. The hypothesis is diminishing returns beyond some optimal number — early levels provide mostly spatial detail, later levels provide semantic content, and very many levels may cause the MLP to become a bottleneck or introduce optimization difficulties. Additionally, comparing the current concatenation+MLP fusion to alternatives — simple averaging, learned per-level weights (attention over levels), or a small transformer that cross-attends across levels — would establish whether the fusion mechanism matters or whether any reasonable combination works. The ablations in Appendix D.1 (Table 12) provide a starting point by showing that deep supervision reduces the performance gap between single-layer and multi-layer evaluation, but they don't tell us which specific levels contribute most or how many are sufficient.

Can the correct-to-incorrect reversion problem be solved by using the PRM as a stopping criterion? The revision model exhibits a 38% rate of converting correct answers back to incorrect ones during sequential revisions (Section 6.1), which the paper mitigates via within-chain majority voting or verifier-based selection. But a more principled approach would be to use the PRM's step-level or final-answer scores to detect when the current answer is already correct and stop the revision chain. This would require evaluating the PRM's discrimination on revision model outputs — can it distinguish correct revisions from incorrect ones, even though there's a distribution shift from base-model outputs that the PRM was trained on? The paper reports in Appendix J (Figure 15a) that the base-LM PRM underperforms a revision-specific ORM when scoring revision model outputs, but the gap at 64 generations is modest (~40% vs. ~42% accuracy). A direct experiment would train the PRM on a mix of base-model and revision-model outputs (to reduce distribution shift), then measure the precision and recall of PRM-based stopping: what fraction of correct answers are correctly identified as correct (so the chain stops) vs. incorrectly revised further? What fraction of incorrect answers are correctly identified as still needing revision? The 38% reversion rate represents an upper bound on potential improvement — if perfect stopping could be achieved, sequential revision accuracy might rise by several percentage points (since 38% of correct answers would no longer be lost).

What happens when you combine PRM tree-search with the revision model as the proposal distribution? Section 8 acknowledges this as a major missing experiment. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates via iterative refinement, best on easy problems), while PRM search improves candidate selection (finding the best among generated candidates via beam search, best on medium problems). A combined system would use the revision model as the generator within beam search: at each step of the search tree, the model conditions on its own previous rejected branches as context, using the PRM to score and prune branches. This could yield three benefits: (1) beam search over revisions could explore branching revision strategies rather than linear chains, potentially finding solutions that sequential revisions miss; (2) the PRM could identify promising partial revisions early and allocate more search budget to them, analogous to how beam search prunes unpromising solution prefixes; (3) the PRM could serve as the revision-chain stopping criterion discussed above. A strong follow-up would evaluate this on the MATH benchmark with the same difficulty-binned analysis as the paper, testing whether the combined system exceeds the 4× efficiency gain over best-of-N that each mechanism achieves individually (Figures 4 and 8). The baseline would be the compute-optimal policy that selects between search and revisions per difficulty bin; the hypothesis is that combining them per-problem yields additional gains, particularly on medium-difficulty problems where both mechanisms individually show benefits.

Does the approach scale to domains without clean binary correctness signals? The paper's entire pipeline — PRM training via Monte Carlo rollouts, difficulty estimation via pass@1 or PRM score distributions, verifier-based answer selection — depends on exact-answer matching to determine whether a solution is correct. MATH is an ideal case: answers are strings that can be checked against a ground-truth string via an automated grading function. Many important reasoning domains lack this property: code generation (where correctness is functional but not answer-matched), open-ended QA (where multiple answers can be correct), factual recall (where correctness is binary but evaluating it requires external knowledge), and creative generation (where there are no objective correctness criteria). Extending the compute-optimal framework to these domains would require either: (1) verifiers trained on human preference judgments rather than binary correctness, which changes the PRM training pipeline and the difficulty estimation approach; or (2) automatic evaluation proxies (e.g., unit test pass rate for code, entailment scores for QA, factuality classifiers for recall) that approximate binary correctness. A strong follow-up would test the simplest extension first: on a code generation benchmark like HumanEval or MBPP, train an outcome reward model to predict whether a generated program passes its unit tests, then evaluate whether the over-optimization pattern (beam search degrading on easy problems at high budgets) replicates, and whether the compute-optimal difficulty-conditioned allocation yields efficiency gains comparable to the 4× observed on MATH. If the approach transfers, it would demonstrate robustness to the verifier training signal; if it doesn't, it would identify the binary-correctness assumption as a fundamental constraint on the framework.

---

Practical Applications and Downstream Use Cases

Robot perception and manipulation with frozen video-pretrained backbones. The paper demonstrates that V-JEPA 2.1 ViT-G features, used as a frozen encoder for a learned world model, achieve 80% success rate on a real-robot grasping task — a 20 percentage point improvement over V-JEPA 2. The features are used without fine-tuning; the world model (a 300M-parameter transformer predictor) is trained on the Droid dataset and then deployed zero-shot on a Franka Panda arm. For robotics practitioners, this means that a single video-pretrained encoder can serve as the perception backbone for multiple manipulation tasks, reducing the need for task-specific visual training. The navigation results reinforce this: replacing NWM's SD-VAE with V-JEPA 2.1 features enables 10× faster planning (10.6 seconds vs. 103.2 seconds) with better trajectory accuracy. The practical upshot is that video SSL pretraining — which requires no robot-specific data — can substitute for expensive in-domain visual feature engineering in robot perception pipelines, with the frozen-feature approach providing computational efficiency benefits (smaller downstream models, faster inference) that task-specific training may not match.

Efficient video understanding systems that don't sacrifice spatial precision. Prior to this work, building a video understanding pipeline that required both action recognition and spatial localization (e.g., detecting what action is happening and where in the frame) meant running two separate models: a video model for temporal understanding and an image model for spatial precision. V-JEPA 2.1 shows that a single encoder can achieve state-of-the-art on both: 77.7% on SSv2 (action recognition) and 0.307 RMSE on NYUv2 (depth estimation) with the same frozen backbone. For applications like egocentric activity analysis (e.g., Ego4D), where systems must simultaneously recognize actions and localize interacted objects, deploying V-JEPA 2.1 as a single encoder reduces model count, memory footprint, and inference cost. The Ego4D STA results (7.71 mAP All, +35% over prior best) directly validate this: the task requires simultaneous bounding-box localization, noun/verb classification, and temporal anticipation, and V-JEPA 2.1's unified backbone outperforms methods that use task-specific architectures or separate image and video encoders.

Cost-efficient data generation for self-improvement loops. The compute-optimal allocation framework, while developed and tested on MATH evaluation, has a natural application to data generation for self-improvement: when using an LLM to generate training solutions that will be used to fine-tune itself (as in STaR, ReST^EM, or rejection-sampling fine-tuning), the budget for generating high-quality solutions matters. The paper's difficulty-conditioned allocation suggests that compute should be spent differently depending on problem difficulty: sequential revisions for easy problems (where initial attempts are mostly correct and need refinement), beam search for medium problems (where exploration of different strategies helps), and no extra compute for hard problems (where the base model has negligible chance of producing a correct solution regardless of budget). This targeted allocation could be integrated into existing self-improvement pipelines by: (1) running the PRM-based difficulty estimation on the training set, (2) assigning per-problem generation budgets based on the difficulty bin, and (3) comparing the quality and diversity of generated solutions against uniform best-of-N allocation at the same total budget. The paper's FLOPs-matched results (Section 7) suggest that on easy-to-medium problems, test-time compute with a smaller model can outperform a 14× larger model with greedy decoding, implying that self-improvement data generated with compute-optimal allocation might be higher-quality at the same total cost than data generated by a larger model with simple sampling.

---

When to Prefer This Method

The paper positions V-JEPA 2.1 as a general-purpose visual encoder that excels at both dense spatial tasks and global video understanding tasks. The decision rule for practitioners is guided by which capability matters more for their downstream application and whether they have access to both image and video data for pretraining:

Use V-JEPA 2.1 as your encoder when:

• You need strong performance on both dense spatial tasks and temporal video understanding tasks from a single model, and the engineering simplicity of one backbone outweighs small per-task gaps to specialized alternatives. V-JEPA 2.1 is the only model that achieves state-of-the-art on depth estimation (0.307 RMSE on NYUv2), action anticipation (40.8 Recall@5 on EK100), and action recognition (77.7% on SSv2) simultaneously — no prior model appears in the top tier on all three tasks.
• Your downstream application is robot manipulation or navigation, where spatial precision directly impacts task success. The 80% grasping success rate (vs. 60% for V-JEPA 2) and 10× faster navigation planning (vs. NWM's SD-VAE baseline) provide direct evidence that V-JEPA 2.1's improved dense features translate to better embodied task performance.
• You have access to both large-scale image and video datasets for pretraining. The VisionMix163M recipe (142M images from LVD-142M + motion-heavy video sources) is designed to be replicated, and the paper demonstrates that joint image-video training with this data composition outperforms video-only or image-only pretraining on both spatial and temporal tasks.
• You prioritize frozen-feature evaluation over fine-tuning. All of V-JEPA 2.1's reported results use linear probes or lightweight attentive probes on frozen encoder features, meaning you can deploy the encoder as-is without task-specific adaptation. This simplifies deployment and reduces the risk of overfitting to small downstream datasets.

Use DINOv3 or specialized image encoders instead when:

• Your primary task is semantic segmentation on heavily cluttered scenes (ADE20K, Cityscapes), where DINOv3 ViT-7B maintains a clear lead (55.9 vs. 47.9 mIoU on ADE20K, 81.1 vs. 73.5 on Cityscapes). The paper hypothesizes that VisionMix163M may lack sufficient cluttered-scene examples, and until this data gap is addressed, DINOv3 remains stronger on these benchmarks.
• You need absolute maximum video object segmentation accuracy and the 2.1 J &F gap on DAVIS (71.1 for DINOv3 ViT-7B vs. 69.0 for V-JEPA 2.1 ViT-G) is significant for your application. DINOv3's specialized dense-feature training still provides an edge on temporal correspondence.
• Your deployment is image-only with no need for temporal understanding, and you care primarily about ImageNet classification accuracy. V-JEPA 2.1 ViT-G achieves 85.5% on IN1K, trailing DINOv3 ViT-7B (88.4%), SigLIP2 (88.0%), and PEcore G (87.6%), suggesting that joint image-video training incurs a modest cost on pure image classification relative to image-specialized models.
• You lack the compute budget to pretrain a 2B-parameter model from scratch (135K iterations at batch sizes of 2,304 images + 128 video clips). In this case, the distilled ViT-L (300M) or ViT-B (80M) variants provide a practical path to using V-JEPA 2.1 features, though with a moderate performance drop (distilled ViT-L achieves ~46.7 mIoU on ADE20K vs. 47.9 for ViT-G, and 76.5% on SSv2 vs. 77.7%).