TIPS: Text-Image Pretraining with Spatial awareness

🎯 Pitch

By simply replacing noisy web captions with spatially descriptive synthetic text, standard image-text models suddenly rival self-supervised DINOv2 on depth estimation and segmentation—without sacrificing retrieval.

1. Executive Summary

This paper introduces Text-Image Pretraining with Spatial awareness (TIPS), a general-purpose image-text encoder that achieves strong off-the-shelf performance on both dense prediction tasks (e.g., semantic segmentation, depth estimation) and global understanding tasks (e.g., image classification, retrieval). TIPS combines two complementary mechanisms: enhancing weak supervision with synthetic captions (replacing or augmenting noisy web alt-text with descriptions from PaliGemma that capture object arrangements and spatial relationships) and integrating self-distillation and masked image modeling into contrastive image-text learning (enforcing spatially coherent patch representations via a teacher-student framework). Scaling TIPS to a ViT-g architecture on a curated 117M-image dataset yields competitive or superior results across 8 tasks covering 16 datasets, notably achieving dense prediction performance rivaling self-supervised DINOv2 while simultaneously enabling multimodal retrieval tasks that self-supervised methods cannot perform, establishing that image-text models can match dedicated image-only pretraining for spatial understanding only when the weak supervision is specifically augmented for spatial coherence.

2. Context and Motivation

The Core Problem: Image-Text Models Produce Spatially Impoverished Representations

The fundamental problem this paper addresses is that image-text contrastive models like CLIP, despite their strong performance on global understanding tasks (classification, retrieval), produce image features that are remarkably poor for dense spatial prediction tasks — tasks that require understanding where things are in an image, not just what is in it. If you take a frozen CLIP ViT-L feature extractor and try to use it for monocular depth estimation on NYUv2, errors are substantially higher than with a comparably sized self-supervised model. For semantic segmentation, the gap is similarly large.

This is not a minor performance differential — it reflects a fundamental representational deficiency. CLIP-style training optimizes a single global embedding (the [CLS] token) to align with a text description, which naturally encourages the model to focus on the most salient object and ignore spatial layout, background elements, and inter-object relationships. The patch tokens — which could, in principle, encode rich spatial information — receive no direct incentive to become spatially coherent during training, because the contrastive loss only supervises the global embedding. The result is that the patch representations contain useful semantic content but lack the local geometric consistency needed for tasks like predicting per-pixel depth or surface normals.

This gap creates a bifurcated ecosystem in practice: self-supervised models (DINO, DINOv2, iBOT) are the de facto standard for dense vision tasks, while image-text models dominate multimodal and zero-shot applications. A practitioner building a depth estimation system reaches for DINOv2, not CLIP. A practitioner building a retrieval system reaches for CLIP, not DINOv2. There is no single model that serves both needs effectively, which means duplicated engineering effort, incompatible feature spaces, and the inability to build systems that seamlessly combine spatial reasoning with language grounding.

Why This Matters: The Cost of Separate Feature Extractors

This bifurcation has concrete consequences:

1. Multimodal systems with spatial reasoning are fractured. Modern vision-language systems (e.g., LLaVA, GPT-4V, Gemini) typically chain together a vision encoder with an LLM. If the vision encoder lacks spatial awareness — as CLIP-based ones do — the LLM receives a representation that encodes what objects are present but has impoverished information about where they are and how they relate spatially. Tong et al. (2024) demonstrated this directly, showing that CLIP-based multimodal models systematically fail at questions requiring spatial reasoning, and that incorporating DINOv2 features partially fixes these failures. But this means running two separate vision encoders (one CLIP-style for semantics, one DINO-style for geometry), which doubles the visual compute and creates a complex integration problem.

2. Applications that need both semantics and geometry are stuck choosing one. DepthAnything (Yang et al., 2024), a widely-used monocular depth estimator, builds on DINOv2 features precisely because image-text models don't provide usable spatial representations. But a DINOv2-based depth estimator cannot naturally incorporate textual prompts (e.g., "estimate depth only for the foreground objects" or zero-shot guidance from language). Conversely, image-text models provide language alignment but produce features too weak for geometry tasks. The paper cites the example of large reconstruction models (LRM; Hong et al., 2024) for single-image-to-3D, where authors explicitly choose DINO over more semantic-aware encoders like CLIP because the task requires structural and texture information — semantic knowledge alone is insufficient.

3. Resource duplication in the research ecosystem. The vision community maintains two largely separate research tracks — one optimizing self-supervised models for dense prediction, one optimizing image-text models for multimodal understanding — with limited cross-pollination. A unified model would amortize the enormous computational cost of pretraining across both use cases.

Prior Approaches and Where They Fall Short

Self-Supervised Methods: Strong Geometry, No Language

Self-supervised methods — DINO (Caron et al., 2021), MAE (He et al., 2022), iBOT (Zhou et al., 2022), DINOv2 (Oquab et al., 2024) — train on images alone using objectives that encourage local feature consistency. DINO uses self-distillation between global and local crops, forcing the model to recognize that different patches of the same image should produce consistent representations. iBOT adds masked image modeling, where visible patches must predict the semantics of masked patches, forcing local spatial reasoning. DINOv2 scales these ideas with a large curated dataset and achieves state-of-the-art frozen-feature performance on dense tasks.

The limitation is existential: these models cannot process text. They produce excellent depth maps and segmentations but have zero ability to ground these representations in language, perform zero-shot classification, or enable text-conditioned retrieval. They are purely vision-only feature extractors, and retrofitting language alignment (e.g., by training an adapter) is an imperfect post-hoc fix that doesn't match end-to-end multimodal training.

Image-Text Methods: Strong Semantics, Weak Geometry

CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), SigLIP (Zhai et al., 2023), OpenCLIP (Cherti et al., 2023), and EVA-CLIP (Sun et al., 2023) all train with contrastive losses that align global image and text embeddings. These methods produce powerful semantic representations — SigLIP achieves excellent zero-shot classification and retrieval — but the paper's experiments (Tables 2 and 3) reveal a striking pattern:

"even CLIP-L, with much worse performance on image-level prediction tasks, outperforms the recent SigLIP-SO on all 6 dense evaluations."

This is counterintuitive: SigLIP-SO is substantially better at global tasks (classification, retrieval) than CLIP-L, yet worse at dense tasks. The paper explains this through the nature of the training signal: noisier web captions, which CLIP-L uses, may inadvertently provide more diverse supervision that incidentally helps patch features, while SigLIP's cleaner training (sigmoid loss, better optimization) optimizes more aggressively for global alignment at the expense of patch-level information. But the fundamental issue remains: neither method explicitly encourages spatial coherence in the patch tokens, so dense performance is an unreliable byproduct rather than a design goal.

A crucial mechanistic issue the paper identifies: web captions systematically omit spatial information. The paper shows an example in Figure 3: a web caption reads "2007 Cadillac Escalade for sale dealership $30k" — it names the main object but says nothing about the background, the spatial arrangement, or other objects in the scene. A synthetic caption from PaliGemma for the same image reads "A black SUV parked in front of a building" — it describes the spatial relationship ("in front of"), the background ("building"), and additional attributes ("black"). Web captions typically function as noisy image-level labels, providing little signal for learning where things are.

Attempts at Bridging the Gap: Partial, Task-Specific, or Incomplete

Several prior works have tried to improve spatial understanding in image-text models, but each has significant limitations relative to TIPS's goals:

MaskCLIP (Dong et al., 2023) combines CLIP-style contrastive learning with masked image modeling (MIM). This is directly relevant — it recognizes that MIM can improve spatial coherence — but the paper notes two key limitations. First, MaskCLIP only adds MIM, not self-distillation. Second, MaskCLIP was designed and evaluated primarily for improving vision-language tasks, not as a general-purpose vision encoder. It does not demonstrate strong dense prediction performance comparable to self-supervised models.

SILC (Naeem et al., 2024) combines CLIP with self-distillation (the DINO loss). This is also directly relevant and the paper explicitly acknowledges SILC as one of the closest prior works. However, SILC only adds self-distillation, not MIM. The ablation in Table 1 shows why this matters: CLIP + self-distillation (which approximates SILC) improves dense tasks modestly — segmentation goes from 64.4 to 70.3 mIoU — but adding MIM on top (CLIP + self-distillation + MIM) further improves segmentation to 75.9 mIoU. The MIM component provides a complementary signal that self-distillation alone cannot.

SLIP (Mu et al., 2021) combines SimCLR (an image-only contrastive loss) with CLIP, but SimCLR operates on global image augmentations and doesn't specifically target patch-level spatial coherence. It was evaluated primarily on classification, not dense prediction.

FLIP (Li et al., 2023) uses masking during CLIP training, but solely for computational efficiency (processing fewer patches per image) — it has no reconstruction or prediction loss on the masked patches. This means masking serves only as a regularizer or throughput hack, not as a mechanism for learning spatial relationships.

Open-vocabulary detection/segmentation methods (e.g., OWL-ViT, DenseCLIP, CLIPSelf, CLIP-DINOiser) adapt image-text models for spatial tasks, but they are task-specific: they add detection heads, segmentation decoders, or fine-tuning stages that require task-specific supervision. The paper positions TIPS differently: the goal is a general-purpose frozen feature extractor that works off-the-shelf for many tasks without task-specific architectural modifications or fine-tuning. This is a harder standard — the features themselves must be good enough that a simple linear probe or lightweight decoder suffices — but it makes the model far more widely applicable.

Synthetic caption methods (LaCLIP, VeCLIP) generate improved captions for CLIP training but evaluate only on global tasks (retrieval, classification). The paper notes: "there is no consideration related to dense prediction." This is a critical gap: synthetic captions have been proposed as a data quality improvement, but their potential to specifically enhance spatial understanding through richer scene descriptions has not been explored or measured.

CapsFusion (Yu et al., 2024) uses synthetic captions to improve large multimodal models, but targets generative text applications rather than representation learning for downstream vision tasks.

How TIPS Positions Itself

The paper frames TIPS as a synthesis of two insights that, individually, have been partially explored, but whose simultaneous combination has not been attempted, and whose interaction produces gains neither achieves alone.

First insight: synthetic captions provide spatial supervision that web captions lack. The paper is the first to identify and measure that replacing noisy web captions with descriptively rich synthetic captions (from PaliGemma, which describes objects, their attributes, and their spatial arrangements) specifically boosts dense prediction. The ablation in Table 1(B) is telling: simply training CLIP with PaliGemma captions instead of web captions improves Pascal VOC segmentation from 64.4 to 74.5 mIoU — a 10.1 point gain — while reducing NYUv2 depth RMSE from 0.620 to 0.544. But this comes at a cost: ImageNet KNN classification drops from 76.9 to 70.0, because synthetic captions lack fine-grained object details (the car model name "Cadillac Escalade" is replaced by the generic "black SUV"). This motivates the dual embedding approach: use one [CLS] token trained with noisy web captions (for object-centric fine-grained information) and a second [CLS] token trained with synthetic captions (for spatial layout information), with both tokens backpropagating into the shared patch features so the patch representations benefit from both signals.

Second insight: self-distillation and MIM are complementary and synergistic for dense tasks when combined with image-text training. The paper demonstrates this through careful ablations (Table 1, Section C). Self-distillation (matching local crop representations to global crop representations) encourages the model to learn that different regions of an image belong to the same scene — a form of spatial consistency. MIM (predicting masked patch semantics from visible patches) encourages individual patch tokens to understand their spatial context. These objectives target different aspects of spatial awareness, and the paper shows that their combination produces substantially better dense features than either alone. The synergy is not obvious a priori — one might think they redundantly encode similar information — but the empirical evidence is clear.

Positioning relative to SILC and MaskCLIP. Both SILC and MaskCLIP are acknowledged as the closest prior art. TIPS goes beyond SILC by adding MIM, and beyond MaskCLIP by adding self-distillation and synthetic captions. The paper also notes several technical simplifications compared to the original DINO/iBOT formulations: using a single global crop (instead of two) increases throughput by 25%; using simple random crops and flips (instead of color jitter, blur, solarization) avoids the negative impact of heavy augmentations on dense tasks (shown in Table 11, row D, where BYOL-style augmentations on local crops cause segmentation to collapse from 79.0 to 40.1 mIoU); and using random masking (instead of blockwise masking from iBOT) simplifies implementation while performing comparably.

Positioning relative to DINOv2. DINOv2 is the state-of-the-art self-supervised model and the primary benchmark for dense prediction with frozen features. TIPS does not claim to universally outperform DINOv2 — on some dense metrics DINOv2 maintains a narrow lead. Instead, TIPS claims to achieve competitive dense performance while simultaneously enabling multimodal tasks that DINOv2 cannot perform at all. This is a different value proposition: TIPS sacrifices a small amount of dense prediction accuracy (and sometimes matches or exceeds DINOv2, as in segmentation) to gain language alignment, while DINOv2 optimizes purely for vision. The paper's contribution is showing that this tradeoff is not as severe as previously assumed — with the right design choices, image-text models can approach self-supervised performance on dense tasks without giving up their multimodal capabilities.

The essential contribution is not a new architectural novelty but the identification and combination of two complementary mechanisms that, together, close the spatial gap. The paper's ablation structure (Table 1) is designed to make this case: starting from a CLIP baseline (row A), adding synthetic captions with dual embedding helps (row B), adding self-distillation and MIM helps (row C), and combining both (row D) produces gains larger than the sum of the individual improvements — indicative of synergy. The paper then demonstrates that these gains scale to a ViT-g architecture and translate to competitive or superior performance across a diverse suite of 16 datasets spanning dense, global, and multimodal tasks, establishing TIPS as a genuine general-purpose model that unifies the previously bifurcated landscape of self-supervised and image-text representation learning.

3. Technical Approach

3.1 Reader Orientation

This paper presents a training recipe and architecture for a vision transformer that simultaneously understands what objects are in an image (semantics), where they are (spatial layout), and how they relate to language (text alignment) — producing a single frozen model whose features can be used off-the-shelf for both dense prediction tasks like depth estimation and global tasks like image retrieval. The core problem solved is that existing image-text models (CLIP, SigLIP, OpenCLIP) produce representations that are spatially impoverished — their patch tokens lack the local geometric coherence needed for per-pixel tasks — while self-supervised models (DINOv2) produce excellent spatial features but have no language interface whatsoever. TIPS bridges this gap by combining improved textual supervision (synthetic captions that describe spatial relationships) with spatial consistency losses (self-distillation and masked image modeling) within a contrastive image-text training framework, and the key insight is that these two interventions are synergistic: synthetic captions provide what to learn about spatial layout, while the self-supervised losses provide the mechanism to learn it in the patch tokens.

3.2 Big-Picture Architecture (Diagram in Words)

The TIPS system has five major components that operate in a single joint training pipeline:

Image Preprocessing Pipeline: Takes a raw image $I$ and produces three types of augmentations: one large global crop at resolution 224 (with random resize cropping and horizontal flip), multiple small local crops at resolution 98 (random square crops), and a heavily masked version of the global crop where 75% of patches are randomly replaced with learnable mask tokens. No color jitter, blur, or solarization — only crops and flips.
Student Vision Transformer ( $f_s$ ): The main ViT being trained. It receives the global crop, local crops, and masked image. It produces three kinds of outputs: two [CLS] tokens ( $e_g$ for object-centric embedding, $\hat{e}_g$ for spatially-aware embedding) and patch-level embeddings $\{e_n\}_{n=1}^N$ . The dual [CLS] design is a key architectural modification — the standard ViT has one, but TIPS adds a second to separately encode different types of information from two different captions.
Text Encoder ( $g$ ): A 12-layer transformer that encodes two captions per image: the original noisy web caption $T$ (e.g., "2007 Cadillac Escalade for sale dealership $30k") and the synthetically-generated caption `$ \hat{T} $` (e.g., "A black SUV parked in front of a building"). Produces text embeddings `$ e_t $` and `$ \hat{e}_t$` respectively.
Teacher Vision Transformer ( $f_t$ ): A temporally delayed version of the student, updated via Exponential Moving Average (EMA) with momentum on a cosine schedule from 0.994 to 1. The teacher processes only the global crop, without masking, and serves as a stable target for the self-distillation and masking losses. The teacher is never trained directly — its weights are a running average of the student's weights, which prevents representational collapse.
Loss Computation: Four losses operate on different combinations of the outputs:
- $\mathcal{L}_{\text{CLIP}}$ between $e_g$ and $e_t$ (web caption alignment)
- $\hat{\mathcal{L}}_{\text{CLIP}}$ between $\hat{e}_g$ and $\hat{e}_t$ (synthetic caption alignment)
- $\mathcal{L}_{\text{distill}}$ enforcing that local crop [CLS] tokens from the student match the global crop [CLS] token from the teacher (spatial consistency across scales)
- $\mathcal{L}_{\text{mask}}$ enforcing that encoded mask tokens from the student recover the semantic content of the corresponding unmasked patches from the teacher (spatial context understanding)

The total loss is: $\mathcal{L}_{\text{total}} = \frac{1}{2}(\mathcal{L}_{\text{CLIP}} + \hat{\mathcal{L}}_{\text{CLIP}}) + \alpha \mathcal{L}_{\text{distill}} + \beta \mathcal{L}_{\text{mask}}$ with $\alpha = 1$ and $\beta = 2$ .

Information flows as follows: an image enters the preprocessing pipeline → the student processes global, local, and masked versions → the text encoder processes both captions → the teacher processes only the global image → the four losses are computed and backpropagated through the student only → the teacher is updated via EMA of the student.

3.3 Roadmap for the Deep Dive

First, the problem setup and base CLIP formulation — what TIPS inherits and what stays fixed.
Second, the synthetic caption enhancement — why web captions are insufficient, how synthetic captions are generated, and the dual [CLS] mechanism that lets the model learn from both simultaneously. This is the "what to learn" improvement.
Third, the self-distillation loss — how local-to-global matching incentivizes patch-level spatial consistency across scales, and how the implementation simplifies (single global crop, simple augmentations) compared to DINO.
Fourth, the masked image modeling loss — how random 75% masking combined with semantic prediction (not pixel reconstruction) forces patch tokens to understand their spatial context, and how the teacher provides stable targets.
Fifth, the training dataset construction — the multi-stage filtering pipeline that reduces 10B web images to 117M curated pairs, and why curation matters for off-the-shelf feature quality.
Sixth, the total loss combination and scaling — how the four losses interact, the hyperparameter choices, the model scaling (ViT-g with 1.1B image encoder parameters), and the high-resolution fine-tuning stage.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that image-text contrastive models can match self-supervised models on dense vision tasks only when two conditions are simultaneously met: (1) the textual supervision is augmented with spatially descriptive synthetic captions, and (2) the training objective explicitly incentivizes spatial coherence in patch tokens through self-distillation and masked image modeling. Neither alone suffices, and their combination produces gains exceeding the sum of their individual contributions.

3.4.1 Problem Setup and Base CLIP Formulation

The paper begins from the standard CLIP framework (Radford et al., 2021). Given a collection of image-text pairs $\{(I_k, T_k)\}$ , where $T_k$ is a noisy web caption for image $I_k$ , the goal is to learn a function $f$ that maps an image to a set of embeddings $\{e_g, e_1, e_2, \ldots, e_N\}$ , where $e_g$ is a global embedding (the [CLS] token) and $\{e_n\}_{n=1}^N$ are patch embeddings. A separate text encoder $g$ maps caption $T$ to a text embedding $e_t$ .

The training objective pushes $e_g$ and $e_t$ close for matching image-caption pairs and far for non-matching pairs, using the InfoNCE (contrastive) loss:

$\mathcal{L}_{\text{CLIP}} = -\frac{1}{B} \sum_{b=1}^B \log \frac{\exp(\text{sim}(e_g^b, e_t^b) / \tau)}{\sum_{j=1}^B \exp(\text{sim}(e_g^b, e_t^j) / \tau)}$

where $B$ is the batch size and $\tau$ is a learned temperature parameter scaling the cosine similarities $\text{sim}(e_g, e_t) = e_g^\top e_t / (\|e_g\| \|e_t\|)$ .

This is a symmetric formulation — the same loss is applied in both image-to-text and text-to-image directions by swapping the roles of queries and keys, though the paper notation simplifies to the single expression above.

The image encoder $f$ is implemented as a Vision Transformer (ViT). The text encoder $g$ is a standard transformer (Vaswani et al., 2017). These design choices are inherited from CLIP and are not modified by TIPS. The paper builds on top of this by adding components around it, not by changing the core contrastive mechanism.

3.4.2 Enhancing Weak Supervision with Synthetic Captions

Why web captions are insufficient for spatial learning. Standard image-text training uses captions sourced from web alt-text, which are noisy and systematically fail to describe spatial layout. The paper provides a concrete example in Figure 3: a web caption reads "2007 Cadillac Escalade for sale dealership $30k". This caption mentions the main object (the car) and includes irrelevant commercial text, but says nothing about the scene background, the spatial arrangement, other objects, or object attributes beyond the model name. It functions as a noisy object label, not a scene description.

A synthetic caption for the same image, generated by PaliGemma (Beyer et al., 2024), reads "A black SUV parked in front of a building." This describes:

The color of the object ("black") — attribute information useful for learning distinctive features.
The spatial relationship ("in front of") — a preposition that directly signals layout.
The background element ("building") — an object that would otherwise be ignored by a CLIP model trained on the web caption.
The object category ("SUV") — coarser than the specific model name but more generally useful for scene understanding.

The key observation is that these two captions provide complementary information. The web caption provides fine-grained object identity (the specific car model "Cadillac Escalade") that is useful for discriminative classification and retrieval. The synthetic caption provides spatial and contextual information (spatial prepositions, background objects, attributes) that is useful for dense prediction tasks. Neither alone is sufficient for a general-purpose model.

Synthetic caption generation. The paper uses PaliGemma, a recent multimodal model from Beyer et al. (2024), specifically the version fine-tuned on COCO captions. During pretraining, the 224-resolution version of PaliGemma is used. During the short high-resolution fine-tuning stage at the end of training, the 448-resolution version is used. The captioner processes every image in the training set once, producing one synthetic caption per image. This is an offline preprocessing step, not an online augmentation — the captions are generated before training begins and stored alongside the web captions. This means the captioning cost is amortized over training epochs and does not affect training throughput.

PaliGemma is chosen because it generates accurate, high-level image descriptions that consistently include spatial relationships and multiple objects. The paper notes this is a property of PaliGemma's training (the COCO fine-tuning encourages descriptive captions with spatial prepositions), not a property TIPS enforces. The choice of captioner is pragmatic: PaliGemma is publicly available and produces the desired caption style.

The dual embedding mechanism. To leverage both captions simultaneously without forcing them to compete, TIPS introduces an architectural modification: a second [CLS] token in the vision transformer. The standard ViT already has one [CLS] token that aggregates global image information through self-attention over all patch tokens. TIPS adds a second [CLS] token with the same dimensionality and position in the sequence, initialized identically to the first.

The two [CLS] tokens produce two global embeddings:

$e_g$ : the object-centric embedding, trained with the noisy web caption $T$ via the standard $\mathcal{L}_{\text{CLIP}}$ loss.
$\hat{e}_g$ : the spatially-aware embedding, trained with the synthetic caption $\hat{T}$ via a second $\hat{\mathcal{L}}_{\text{CLIP}}$ loss of identical form but using $\hat{e}_g$ and $\hat{e}_t$ instead of $e_g$ and $e_t$ .

Both [CLS] tokens attend to the same patch tokens in all transformer layers. This means the patch tokens receive gradient signals from both contrastive losses, encouraging them to encode both fine-grained object identity (driven by the web caption alignment) and spatial layout and context (driven by the synthetic caption alignment). The two [CLS] tokens diverge in what they extract from the shared patch representations because they are supervised by different caption distributions.

At inference time, the choice of which [CLS] token to use depends on the downstream task:

$e_g$ (web caption token) is used for tasks requiring fine-grained object discrimination: ImageNet classification (both linear probe and KNN), fine-grained retrieval (UnED), and zero-shot classification.
$\hat{e}_g$ (synthetic caption token) is used for tasks requiring spatial awareness: all dense prediction tasks (segmentation, depth, normals) and multimodal retrieval (image-to-text and text-to-image), because synthetic captions better align with the descriptive style of retrieval benchmark captions.

For dense tasks, the [CLS] token is concatenated to each patch token before feeding to the task-specific probe or decoder. The paper specifies using $\hat{e}_g$ for spatial tasks because it encodes the global scene layout, which provides useful context for per-pixel predictions.

Comparison against alternative caption combination strategies. The paper ablates two alternatives to the dual embedding approach, inspired by Fan et al. (2023) (LaCLIP):

"Sampled": for each image in each training iteration, randomly choose whether to use the web caption or the synthetic caption (with equal probability). The model has only one [CLS] token.
"Multi-text": both captions are fed through the text encoder, and both text embeddings are matched against the same image embedding in separate contrastive losses, but with a single [CLS] token.

The ablation results (Table 1, Section B) show that the dual embedding approach outperforms both alternatives: on Pascal VOC segmentation, dual achieves 73.3 mIoU vs. 71.8 (sampled) and 72.1 (multi-text); on NYUv2 depth, dual achieves 0.588 RMSE vs. 0.563 and 0.580; on ImageNet KNN classification, dual achieves 78.3% vs. 77.0% and 76.9%; on Flickr I→T retrieval, dual achieves 88.7 vs. 90.2 and 85.1; on Flickr T→I retrieval, dual achieves 77.1 vs. 75.4 and 73.9.

The dual approach wins in 3 out of 5 tasks and is competitive on the other 2. The "sampled" approach wins on I→T retrieval (90.2 vs. 88.7), which the paper attributes to caption diversity — randomly switching captions may provide a form of data augmentation for retrieval training. However, "sampled" performs substantially worse on dense tasks (segmentation drops by 1.5 points, depth RMSE worsens by 0.025), confirming that separating the caption types into dedicated embeddings better preserves the spatial information from synthetic captions.

The "multi-text" approach performs worse across the board, indicating that matching two different caption types to the same [CLS] token creates interference — the embedding must compromise between encoding fine-grained object identity and spatial layout, serving neither goal optimally.

Why this design choice matters. The dual embedding mechanism is a specific instance of a broader principle: when training data comes from different distributions (noisy object-centric web captions vs. clean spatially-descriptive synthetic captions) that encode different and complementary types of information, forcing a single representation to capture both leads to suboptimal tradeoffs. Providing the model with multiple pooling heads that specialize in different information types, while sharing computation in the backbone, allows specialization without sacrificing shared representational capacity.

3.4.3 Self-Distillation Loss: Enforcing Cross-Scale Spatial Consistency

Motivation and mechanism. Image-text contrastive learning supervises only the global image representation — the [CLS] token — providing no direct signal to the patch tokens about their spatial relationships. Self-supervised methods like DINO (Caron et al., 2021) address this by requiring that local crops of an image produce similar representations to the global image. This incentivizes the model to recognize that different parts of the same scene belong together, which implicitly requires the model to learn spatially coherent patch features — a local crop containing a wheel should be recognized as part of a car, which means the patch tokens inside the wheel crop must encode features that the [CLS] token can match to the global car representation.

TIPS adapts this idea with several simplifying modifications:

Input preparation. From each input image $I$ , TIPS creates:

One global crop: a large random square crop with horizontal flip augmentation, resized to 224×224. This is the same global view used for the CLIP loss.
$M = 6$ local crops: small random square crops at resolution 98×98. The local crops are not resized from the global crop — they are independently cropped from the original image at the smaller resolution.

Critically, TIPS uses one global crop, not two as in DINO, iBOT, and SILC. Using a single global view increases training throughput by approximately 25% because the teacher model processes only one image instead of two. The paper argues that the second global crop is not necessary — the global-to-local matching provides sufficient self-supervision without needing global-to-global matching.

Teacher-student asymmetry. The global crop is fed through the teacher network $f_t$ to produce a teacher [CLS] embedding $e_{g,t}$ . The local crops are fed through the student network $f_s$ to produce $M$ student [CLS] embeddings $\{e_{g,m}\}_{m=1}^M$ . The teacher uses the single [CLS] token (the one associated with the web caption, $e_g$ ), not the dual token, for the self-distillation loss.

The teacher's weights are updated via Exponential Moving Average (EMA) of the student's weights:

$\theta_t \leftarrow \lambda \theta_t + (1 - \lambda) \theta_s$

where $\lambda$ follows a cosine schedule from 0.994 at the start of training to 1.0 at the end. The schedule starting at 0.994 means the teacher initially updates quickly (momentum 0.006 per step) and gradually freezes. This is standard in self-distillation methods (Caron et al., 2021; Grill et al., 2020).

Projection heads and prototype scores. Both the student and teacher [CLS] embeddings are processed through small MLP-based projection heads before computing the loss:

The student projection $P_s$ maps $e_{g,m}$ to prototype scores $p_m = P_s(e_{g,m}) \in \mathbb{R}^K$ , where $K = 32,768$ is the prototype dimension.
The teacher projection $P_t$ maps $e_{g,t}$ to prototype scores $p_t = P_t(e_{g,t}) \in \mathbb{R}^K$ .
Both projections have the same architecture: a 3-layer MLP followed by L2 normalization and a weight-normalized linear projection to the prototype dimension.
$P_t$ is updated via EMA of $P_s$ , mirroring the image encoder teacher-student relationship.

The prototype scores are then sharpened with temperature parameters and centered, before being compared via cross-entropy loss.

The self-distillation loss. The loss for a batch of $B$ images is:

$\mathcal{L}_{\text{distill}} = -\sum_{b=1}^B \sum_{m=1}^M \text{softmax}\left(\frac{p_t^b - c}{\tau_t}\right) \log\left(\text{softmax}\left(\frac{p_m^b}{\tau_s}\right)\right)$

where:

$p_t^b \in \mathbb{R}^K$ is the teacher's prototype score vector for image $b$ 's global crop.
$p_m^b \in \mathbb{R}^K$ is the student's prototype score vector for image $b$ 's $m$ -th local crop.
$c \in \mathbb{R}^K$ is a centering variable: an EMA of the mean teacher score across the batch, updated with momentum 0.9. Subtracting it prevents the teacher from collapsing to always predicting the same prototypes.
$\tau_t = 0.07$ is the teacher temperature: a sharper target distribution.
$\tau_s = 0.1$ is the student temperature: a slightly softer distribution to prevent overconfidence.
$\text{softmax}(\cdot)$ is computed over the $K$ prototype dimensions.

What it computes, operationally. For each local crop, the student's projection head produces a distribution over $K$ learned prototypes. The teacher's projection head, processing the full-resolution global image, produces a sharper distribution over the same prototypes. The cross-entropy loss penalizes the student if its prototype distribution for a local crop differs from the teacher's distribution for the global crop. Since a local crop shows only part of the scene, the student must infer from the local patch features what the full scene contains — a task that requires the patch tokens to encode spatially informative features that can support such inference.

Why this form. The cross-entropy formulation with sharpened teacher targets and centering is directly adapted from DINO (Caron et al., 2021) and iBOT (Zhou et al., 2022). The sharpening (lower temperature for the teacher) makes the teacher's target distribution more confident, which prevents it from being uniform and provides a stronger training signal. The centering prevents a degenerate solution where the teacher always predicts the same few prototypes (collapse to a Dirac distribution). The separate temperatures create an asymmetry where the teacher is more confident than the student, which is a form of self-distillation — the student is learning to match the teacher's refined predictions, not the raw data.

Simplification relative to DINO. The paper notes that TIPS simplifies the original DINO self-distillation formulation in two important ways. First, as mentioned, using only one global crop instead of two — DINO processes two different global crops through the teacher, which TIPS omits. Second, TIPS uses minimal image augmentations: only random resize crops and horizontal flips. DINO and BYOL use extensive color augmentations (color jitter, Gaussian blur, solarization). The paper ablates this choice in Table 11 (row D): applying BYOL-style augmentations to local crops causes segmentation performance to collapse from 79.0 mIoU to 40.1 mIoU. The reason is that heavy color augmentations destroy the low-level image statistics (color, texture) that dense prediction tasks rely on — depth estimation and segmentation need to know that a surface has consistent color and texture across the image, and color jitter breaks this consistency. The paper concludes that "complex augmentations may not be necessary for representation learning" and that simple spatial augmentations (crops and flips) suffice when combined with strong objectives.

3.4.4 Masked Image Modeling Loss: Enforcing Local Spatial Context

Motivation and mechanism. Self-distillation enforces consistency between different spatial scales — a local crop should predict the same scene identity as the global crop. But this doesn't directly supervise individual patch tokens to understand their local spatial context. Masked image modeling (MIM) addresses this gap: by masking a large fraction of patches and requiring the visible patches to reconstruct semantic information about the masked patches, every patch token learns to encode its spatial surroundings. A patch at the boundary between an object and the background must learn to represent both, so it can help predict the content of neighboring masked patches.

TIPS implements MIM using a teacher-student formulation similar to the self-distillation loss, but operating at the patch level rather than the [CLS] level.

Input preparation. The global crop at resolution 224×224 (the same image that goes through the teacher unmodified) is randomly masked with a masking ratio of 75%. Specifically, 75% of the patch tokens are replaced with a learnable mask token embedding. The mask token is a vector of the same dimensionality as the patch embeddings, initialized randomly and learned during training. The remaining 25% of patches are left unchanged. The masking pattern is random — each patch is independently masked with probability 0.75 — not blockwise or structured. The ablated alternatives in Table 11 (rows B and C) show that blockwise masking (masking contiguous blocks of patches, as in iBOT) performs slightly worse than random masking, and different masking ratios trade off dense performance against global performance, with higher ratios favoring dense tasks and lower ratios favoring global tasks. The paper chooses 75% random masking as a good trade-off.

The masked image is fed through the student network $f_s$ . The mask tokens are processed through all transformer layers, allowing them to attend to visible patches and to each other. The output corresponding to each masked position is a learned embedding $e_n^m$ .

Meanwhile, the unmasked global crop is fed through the teacher network $f_t$ , which produces patch embeddings $\{e_n^t\}_{n=1}^N$ for all patches (including those that were masked in the student's input).

Projection heads. Both the student's mask token embeddings and the teacher's corresponding patch embeddings are processed through projection heads, analogous to the self-distillation loss but operating at the patch level:

The student mask token embedding $e_n^m$ is projected to prototype scores $p_n^m = P'_s(e_n^m) \in \mathbb{R}^K$ .
The teacher patch embedding $e_n^t$ is projected to prototype scores $p_n^t = P'_t(e_n^t) \in \mathbb{R}^K$ .
$P'_s$ and $P'_t$ are structurally identical to $P_s$ and $P_t$ (3-layer MLP, L2 norm, weight-normalized projection to $K = 32,768$ prototypes) but are separate, unshared parameters. This means the model learns two distinct sets of prototypes: one for [CLS]-level self-distillation and one for patch-level MIM.
$P'_t$ is updated via EMA of $P'_s$ .

The masking loss. For a batch of $B$ images and $N$ masked patches per image:

$\mathcal{L}_{\text{mask}} = -\sum_{b=1}^B \sum_{n \in \text{masked}(b)} \text{softmax}\left(\frac{p_{t}^{b,n} - c'}{\tau_t'}\right) \log\left(\text{softmax}\left(\frac{p_m^{b,n}}{\tau_s'}\right)\right)$

where:

$p_t^{b,n} \in \mathbb{R}^K$ is the teacher's patch prototype score for patch $n$ of image $b$ .
$p_m^{b,n} \in \mathbb{R}^K$ is the student's mask token prototype score for the same patch position.
$c' \in \mathbb{R}^K$ is a separate centering variable for the masking loss, updated with EMA momentum 0.9 of the mean teacher patch scores.
$\tau_t'$ is the teacher temperature for MIM, warmed up linearly from 0.04 at the start of training to 0.07.
$\tau_s' = 0.1$ is the student temperature for MIM.
The sum is taken only over the masked patch positions — visible patches are not used in this loss.

What it computes, operationally. For each masked patch position, the student's masked token — which cannot see the original content of that patch but can attend to the 25% of visible patches — must predict the same prototype distribution over $K$ learned concepts as the teacher's unmasked token at the same position. The teacher sees the full, unmasked image and thus produces "ground-truth" prototype scores informed by the actual content. The student must infer the missing content from the context provided by visible patches.

This is fundamentally different from pixel-level reconstruction as in MAE (He et al., 2022). TIPS predicts semantic prototypes — abstract learned concepts — not raw pixel values. A masked patch in the sky region should predict prototypes associated with "sky", "clouds", and related concepts. The model learns which prototypes correspond to which visual concepts through the joint training with the CLIP loss, which provides semantic grounding (via text alignment) for the prototypes.

Why this form and not pixel reconstruction. The paper makes a deliberate choice to use prototype-based prediction rather than pixel reconstruction:

Semantic targets are more informative than pixel-level targets. Predicting that a masked patch is "sky" (a prototype) is a higher-level task than predicting the RGB values of sky pixels. The prototype prediction forces the model to develop semantic understanding of patch contents, which transfers better to downstream tasks.
Prototype prediction is computationally cheaper. The projection head output is 32K-dimensional, not the dimensionality of raw pixels (e.g., 16×16×3 = 768 for ViT patch size 16). The loss computation is lightweight.
Prototype prediction aligns with the contrastive learning paradigm. The same prototype-based formulation is used for both self-distillation and MIM, creating a unified training framework. The CLIP loss already aligns the [CLS] token with text embeddings, which provides semantic meaning to the prototypes.

The temperature warmup in MIM. The paper mentions that $\tau_t'$ is warmed up from 0.04 to 0.07. This means the teacher's targets start less sharp (more uniform) and gradually become sharper (more confident). At the beginning of training, the prototypes are random and the teacher's predictions are unreliable — making them too sharp would force the student to match arbitrary assignments. Warming up the temperature gives the prototypes time to stabilize before the teacher provides strong supervision.

What makes this work with CLIP. A key design decision is that MIM and CLIP training happen simultaneously, not sequentially. Table 11 (row E) ablates a successive training strategy ("CLIP → MIM": first train CLIP to convergence, then add MIM on top). This performs significantly worse than simultaneous training: simultaneous CLIP + self-distillation + MIM achieves 79.0 mIoU on Pascal VOC, while CLIP → MIM achieves 75.2 mIoU (still better than the CLIP baseline of 64.4, but 3.8 points worse than simultaneous). The paper attributes this to the fact that in simultaneous training, the prototypes and patch features co-evolve with the contrastive objective, allowing the MIM loss to shape the representations from the start rather than retrofitting them after CLIP convergence.

3.4.5 Training Dataset: Filtering and Curation Pipeline

The pretraining dataset is critical for TIPS because the model is intended for off-the-shelf use — the features should generalize to a wide range of downstream tasks without fine-tuning. The paper applies a multi-stage filtering process to the WebLI dataset (Chen et al., 2023), a collection of approximately 10 billion public web images with associated alt-text.

Stage 1: Image-text alignment filtering. Following Schuhmann et al. (2022), the paper filters out image-text pairs whose semantic similarity is low, as measured by a pretrained alignment model (presumably a CLIP-like model, though the specific model is not named). Pairs where the image and caption don't match well are discarded. This removes cases where alt-text is unrelated to the image content — a common problem in web data.

Stage 2: Language filtering. Only pairs with English-language captions are retained. The paper does not specify the language detection method.

After stages 1 and 2, the dataset contains 1.7 billion image-text pairs. This is still an order of magnitude too large for efficient training, and the distribution likely contains substantial redundancy and biases toward common web imagery.

Stage 3: Curation by similarity to benchmark datasets. The paper follows a curation procedure similar to DINOv2 (Oquab et al., 2024) and Parthasarathy et al. (2023). The idea is to select web images that are visually similar to images in high-quality, curated datasets, so the training distribution overlaps better with the target downstream distribution. The procedure is:

For each of several curated datasets (listed in Table 13: Pascal VOC train, ADE20K train, NYUv2 train, ImageNet-1K train, ImageNet-22K train, and the training splits of the 8 UnED domains), extract image embeddings using a pretrained model.
Perform k-means clustering on the embeddings of each dataset, with the number of clusters chosen manually per dataset (e.g., 5 clusters for Pascal VOC, 20 for ADE20K, 1000 for ImageNet-1K). The cluster counts represent the visual diversity of each dataset.
For each web image in the 1.7B pool, compute its embedding and assign it to the nearest cluster for each curated dataset. Web images that are far from their assigned cluster center (above the 90th percentile of within-cluster distances) are excluded — these are images that superficially match but are semantically different.
Sample from the remaining images per cluster, weighting by cluster membership size to avoid over-representing rare visual concepts.
Apply this process independently for each curated dataset, then deduplicate across all samples (removing images that appeared in multiple datasets' selections).

This process yields 115.9 million image-text pairs after self-deduplication and evaluation-deduplication (removing any images that appear in evaluation datasets to prevent contamination — approximately 19,000 images removed).

Stage 4: Adding street-level imagery. For the ViT-g models, the paper adds the training set of the Mapillary SLS dataset (Warburg et al., 2020) to compensate for the lack of street-level imagery in web data. Web images are biased toward object-centric photography and landmark shots, underrepresenting the first-person street-view perspective common in robotics and autonomous driving applications. The Mapillary SLS training set (number of images not specified in the paper, but a small fraction of the total) is added as-is. Since Mapillary images don't have web alt-text, the synthetic caption from PaliGemma is used for both CLIP tokens during training. This increases the total dataset size to 117 million images.

Why this curation matters. The paper ablates dataset versions in Table 9. Training CLIP (ViT-B) on the raw 10B dataset yields ADE20k segmentation of 29.1 mIoU and NYUv2 depth RMSE of 0.698. Filtering by English and image-text quality (1.7B) improves to 31.5 and 0.632. The curated 116M dataset further improves to 31.6 and 0.620 — comparable on dense tasks but significantly better on fine-grained retrieval (62.9 vs. 59.3 R@1 on UnED). The curation improves data quality enough that training on 116M curated images outperforms training on 1.7B quality-filtered images, while being 14.5× smaller and thus much cheaper to train on.

The paper also validates that TIPS's gains are not data-specific by training on the public DataComp dataset (Gadre et al., 2023) with an identical curation procedure, yielding 115M pairs. Table 10 shows that TIPS trained on DataComp achieves nearly identical results to TIPS trained on WebLI (79.1 vs. 79.0 Pascal VOC mIoU, 0.479 vs. 0.478 NYUv2 RMSE), confirming that the method, not the specific dataset, drives the improvements.

3.4.6 Total Loss, Training Configuration, and Scaling

Total loss formulation. The four losses are combined with weighting coefficients:

$\mathcal{L}_{\text{total}} = \frac{1}{2}(\mathcal{L}_{\text{CLIP}} + \hat{\mathcal{L}}_{\text{CLIP}}) + \alpha \mathcal{L}_{\text{distill}} + \beta \mathcal{L}_{\text{mask}}$

where $\mathcal{L}_{\text{CLIP}}$ and $\hat{\mathcal{L}}_{\text{CLIP}}$ are the contrastive losses for the web-caption and synthetic-caption [CLS] tokens respectively, $\mathcal{L}_{\text{distill}}$ is the self-distillation loss, $\mathcal{L}_{\text{mask}}$ is the masked image modeling loss, and $\alpha = 1$ , $\beta = 2$ .

The coefficients $\alpha = 1 $and `$ \beta = 2 $` are fixed values, not tuned through extensive hyperparameter search. The paper does not report ablations on these weights — they are chosen to give the MIM loss slightly more influence than self-distillation, consistent with the observation that MIM provides stronger dense task improvements (Table 1, row C: adding MIM improves segmentation by 5.6 points vs. 5.9 points for adding self-distillation, but the combination is better than either). The CLIP losses are averaged (factor `$ \frac{1}{2}$`) to keep their combined weight comparable to a single CLIP loss, since both are InfoNCE losses operating in the same batch.

Optimizer and schedule. Training uses the Adafactor optimizer (Shazeer & Stern, 2018) with a learning rate schedule of linear warmup for 1.4 epochs from 0 to $5 \times 10^{-4}$ , followed by linear decay to 0 over the remaining epochs. The choice of Adafactor over AdamW is standard for large-scale transformer training because Adafactor has lower memory requirements (it factorizes the second-moment accumulator).

Training stages and durations. For the ViT-B backbone (used in ablations): 70 epochs at batch size 16,384, taking 4 days on 256 TPUv3 chips.

For the ViT-g backbone (the main model):

Low-resolution (LR) stage: 15 epochs at batch size 16,384, taking 2 days on 512 TPUv5 chips. The paper notes that training stops at 15 epochs "due to evaluation saturation" — the model's downstream performance stops improving, even though the training loss continues to decrease. This is a practical early stopping criterion.
High-resolution (HR) fine-tuning stage: Starting from the checkpoint at epoch 13, training continues for 0.1 epochs at batch size 4,096 with global crop resolution increased to 448×448 and local crop resolution increased to 140×140. The PaliGemma captioner is also switched to the 448-resolution version. This stage is very short (0.1 epochs = roughly 2,850 steps given 117M images and batch size 4,096) and serves to adapt the model to higher input resolution without full retraining.

The total training cost is relatively modest: the ViT-g LR model trains in 2 days on 512 TPUv5 chips, and the HR fine-tuning adds a small fractional amount. This is significantly cheaper than training comparable models from scratch (CLIP-scale training typically requires weeks on thousands of GPUs/TPUs).

Model architecture. TIPS scales to the ViT-g architecture with several specific choices:

Patch size 14 (standard for ViT-g).
SwiGLU feed-forward network variant (Shazeer, 2020) instead of standard ReLU — a common improvement in large transformers.
Embedding dimension 1536 with 24 attention heads.
Total image encoder parameters: 1.1 billion (comparable to DINOv2-g).
Text encoder: 12 transformer layers with the same embedding dimension (1536) and number of heads (24) as the image encoder. Total text encoder parameters: approximately 389 million (for the released model, as shown in Table 5).
Total parameters (image + text): approximately 1.5 billion.

The text encoder scaling uses a fixed 12 layers for most model sizes (S, B, L, g), meaning that as the image encoder grows from ViT-S to ViT-g, the text encoder stays relatively small. This is because the text encoder's primary role is to provide a contrastive target — it doesn't need to match the image encoder's capacity. The exception is the SO-400m variant, which uses the same number of layers in both encoders.

Teacher momentum schedule. The EMA momentum $\lambda$ for the teacher update follows a cosine schedule from 0.994 to 1.0 over the course of training. The starting value of 0.994 means the teacher updates with effective step size $1 - 0.994 = 0.006$ per training step — it tracks the student closely but with some smoothing. As training progresses and $\lambda$ increases toward 1.0, the teacher becomes increasingly frozen, providing stable targets for the later stages of training when representations are already well-formed and large updates to the teacher would be destabilizing.

Projection head details. Both the self-distillation and MIM projection heads have identical architecture: a 3-layer MLP with Gaussian Error Linear Unit (GELU) activations (standard for transformer-based models), followed by L2 normalization, and a weight-normalized linear projection to a prototype dimension of $K = 32,768$ . The heads are parameterized separately (no weight sharing), allowing them to learn different prototype spaces: the [CLS]-level prototypes for self-distillation encode global scene concepts, while the patch-level prototypes for MIM encode local visual concepts. Having separate heads is important because the same semantic concept (e.g., "sky") may need to be represented differently at the global level (as one of many scene attributes) vs. the patch level (as the dominant content of a specific image region).

Sharpening and centering details. Both losses use softmax with temperature scaling followed by cross-entropy. The student temperatures are both fixed at $\tau_s = \tau_s' = 0.1$ — they produce moderately sharp distributions. The teacher temperature for self-distillation is $\tau_t = 0.07$ (fixed), making the teacher's targets sharper. The teacher temperature for MIM is $\tau_t'$ , warmed up linearly from 0.04 to 0.07 — starting less sharp and becoming sharper, to give the patch prototypes time to stabilize.

Centering is applied to the teacher logits before softmax: $p_t - c$ and $p_t^n - c'$ respectively. The centering variables $c$ and $c'$ are updated every batch as EMA of the mean teacher score, with momentum 0.9. Centering prevents representational collapse by subtracting the running mean — if the teacher starts predicting the same prototypes for all images, the centering makes those prototypes less likely to be chosen, encouraging diversity in the predictions. This is a standard technique from DINO and iBOT.

Design choices rationalized. The paper makes several deliberate simplifications compared to the original DINO and iBOT formulations:

Single global crop instead of two: The original DINO processes two global crops through the teacher, which the paper argues is unnecessary — one is sufficient. This increases throughput by 25%.
Simple augmentations: Only random resize crops and horizontal flips, compared to DINO/BYOL's color jitter, Gaussian blur, and solarization. The ablation in Table 11 (row D) shows that adding BYOL augmentations to local crops destroys dense task performance (segmentation drops from 79.0 to 40.1 mIoU). The paper hypothesizes that heavy color augmentation breaks the low-level feature consistency that dense prediction tasks rely on. When augmentations are applied to both global and local crops, the damage is smaller but still negative for most tasks. This finding is consistent with Assran et al. (2023) and Moutakanni et al. (2024), who argue that complex augmentations may be unnecessary for representation learning when the objective is strong enough.
Random masking instead of blockwise: iBOT uses blockwise masking (masking contiguous square regions of patches) to create a more challenging prediction task, arguing that random masking is too easy because neighboring patches are highly correlated. TIPS finds (Table 11, rows B and C) that blockwise masking performs slightly worse than random masking across most tasks. The simpler random masking is adopted.
Simultaneous training instead of sequential: EVA (Fang et al., 2023) first trains CLIP, then trains MIM on top. TIPS trains all losses simultaneously. The ablation in Table 11 (row E) shows simultaneous training outperforms sequential (CLIP → MIM) significantly (79.0 vs. 75.2 mIoU on Pascal VOC).

Inference protocol. At inference time, the model produces two types of image embeddings. For dense tasks, the patch tokens from the final transformer layer are used, with the spatially-aware [CLS] token $\hat{e}_g$ concatenated to each patch token. For global tasks, either $e_g$ (object-centric, used for classification and zero-shot) or $\hat{e}_g$ (spatially-aware, used for retrieval) is used alone. The choice of which [CLS] token to use is task-dependent and specified in Appendix A.4. All evaluations use frozen features — no fine-tuning of the pretrained weights, only training lightweight task-specific heads (linear layer or DPT decoder).

Knowledge distillation to smaller models. For the released smaller model variants (ViT-S, ViT-B, ViT-L, SO-400m), the paper uses a distillation approach rather than training from scratch. The largest TIPS ViT-g model serves as a frozen teacher. The student image encoder is trained to match the teacher's outputs (both [CLS] and patch tokens) using the same TIPS objectives, while the text encoder is trained from scratch with standard CLIP losses. This is more effective than training smaller models from scratch because the teacher provides a rich target signal that helps the capacity-limited student learn better representations. The distillation process uses the same masking and local cropping strategies as the original TIPS training. After distillation, a high-resolution fine-tuning stage is applied similarly to the ViT-g. Details of the number of parameters for each variant are given in Table 5.

3.4.7 Summary of TIPS's Technical Contributions and Their Relationships

The technical approach can be understood as a training framework with three orthogonal enhancements to standard CLIP, each targeting a different limitation:

Synthetic captions + dual embedding addresses the limitation of textual supervision quality: web captions are noisy and lack spatial information. Adding synthetic captions provides clean, spatially-descriptive supervision. The dual embedding architecture prevents the two caption types from interfering with each other.
Self-distillation addresses the limitation that CLIP supervises only the global image representation: by matching local and global crop representations, patch tokens are incentivized to encode information that supports cross-scale consistency, which indirectly encourages spatial coherence.
Masked image modeling addresses the limitation that individual patch tokens have no direct incentive to encode their spatial context: by requiring masked patches to predict the semantic content of their unmasked counterparts, every patch token learns to represent its surroundings.

Critically, these three enhancements are complementary and synergistic. Synthetic captions provide what to learn about spatial layout — they explicitly describe object arrangements and relationships. Self-distillation provides a mechanism for learning consistency across scales. MIM provides a mechanism for learning local spatial context. The paper demonstrates through ablations (Table 1) that combining them produces gains exceeding what each provides individually, and that training them simultaneously (rather than sequentially) is important for realizing these synergies — the representations co-evolve under all objectives from the start, rather than one objective retrofitting a representation optimized for a different task.

4. Key Insights and Innovations

Innovation 1: Spatial Poverty in Image-Text Models Is a Fixable Data Problem, Not an Architectural Inevitability

The paper's most conceptually important move is reframing why image-text models fail at dense prediction. Before TIPS, the dominant assumption — implicit in the widespread adoption of DINOv2 for depth estimation and CLIP for retrieval as separate tools — was that contrastive language-image training and spatial feature quality represent an inherent trade-off: aligning representations with text forces the model to focus on salient objects at the expense of background and geometry. The architecture (a ViT with a [CLS] token and patch tokens) is identical across CLIP and DINO; if CLIP produces spatially impoverished features, the implication seemed to be that the contrastive text-alignment objective itself is incompatible with learning good patch representations.

The paper challenges this framing with a specific diagnostic finding: the problem is not the contrastive objective per se, but the content of the captions it aligns against. Table 1(B) shows that simply replacing noisy web captions with PaliGemma-generated synthetic captions — keeping the architecture, the InfoNCE loss, and all training hyperparameters identical — improves Pascal VOC segmentation by 10.1 mIoU (64.4 → 74.5) and reduces NYUv2 depth RMSE by 0.076. This is not an incremental gain; it transforms a CLIP ViT-B from substantially worse than even the weakest self-supervised baselines to competitive with them (Table 2 shows DINO-B achieves 66.4 mIoU on Pascal VOC — the CLIP + PaliGemma variant at 74.5 substantially exceeds it).

The causal chain the paper identifies is: web captions systematically omit spatial information (they name main objects, not layouts), so the contrastive loss provides no signal about where objects are or how they relate, so patch tokens receive no gradient incentive to encode spatial relationships. This is a data content problem, not an objective function problem. The contrastive loss can learn spatial features if given spatially descriptive text. The synthetic caption in Figure 3 — "A black SUV parked in front of a building" — contains prepositions ("in front of"), background objects ("building"), and attributes ("black") that web captions lack. The paper demonstrates that these linguistic elements directly translate into improved dense features.

This reframing has significant implications beyond the paper itself. It suggests that the entire line of work on adapting image-text models for spatial tasks through architectural interventions (special decoders, detection heads, patch-level contrastive losses) may be solving the wrong problem. If the root cause is caption content, then improving captions — through synthetic generation, better data filtering, or multi-source caption fusion — may be a simpler and more general solution than task-specific architectural modifications. It also opens the door to using richer caption sources (scene graphs, 3D annotations, spatial relation datasets) as training data for vision-language models, something the field has not systematically explored.

The paper does not claim that data alone solves the entire spatial gap — the subsequent self-supervised losses add substantial further improvements (segmentation goes from 74.5 to 79.0 mIoU when adding self-distillation and MIM). But the data finding establishes that a significant fraction of the gap is attributable to caption poverty, not to fundamental limitations of contrastive learning. This is a diagnostic insight that changes how researchers should think about the problem.

Innovation 2: Self-Distillation and Masked Image Modeling Are Synergistic, Not Redundant, When Combined with Contrastive Image-Text Learning

The paper's second conceptual contribution is demonstrating that two self-supervised objectives — self-distillation (matching local crop representations to global crop representations) and masked image modeling (predicting masked patch semantics from visible patches) — each address different deficiencies in contrastively-trained representations, and their simultaneous application produces gains that neither achieves alone. This finding goes beyond the obvious statement that "more objectives are better" and makes a specific claim about why the combination works.

The field has prior examples of combining self-supervised objectives with CLIP: SILC (Naeem et al., 2024) adds self-distillation, MaskCLIP (Dong et al., 2023) adds MIM, and EVA (Fang et al., 2023) trains CLIP followed by MIM sequentially. But no prior work combined all three simultaneously, and the default assumption might have been that self-distillation and MIM are partially redundant — both encourage spatial awareness, so adding both might provide diminishing returns. The ablation in Table 1(C) disproves this:

CLIP baseline: 64.4 mIoU (Pascal VOC), 0.620 RMSE (NYUv2 depth)
CLIP + self-distillation: 70.3 mIoU, 0.589 RMSE
CLIP + self-distillation + MIM: 75.9 mIoU, 0.511 RMSE

The incremental gain from adding MIM on top of self-distillation (5.6 mIoU improvement, 0.078 RMSE reduction) is larger than the gain from adding self-distillation to CLIP alone (5.9 mIoU, 0.031 RMSE reduction on depth). This is the opposite of diminishing returns — the MIM component becomes more valuable when self-distillation is already present, suggesting a synergistic interaction.

The paper offers a conceptual explanation: self-distillation and MIM target different spatial scales. Self-distillation operates at the image level: it asks "does this local crop belong to the same scene as the global crop?" This forces the [CLS] token — and indirectly the patch tokens feeding into it — to learn global spatial consistency: a wheel patch should encode features that are recognizable as belonging to a car scene. Masked image modeling operates at the patch level: it asks "can this visible patch predict the semantic content of its masked neighbor?" This forces individual patch tokens to learn local spatial context: a patch at the boundary between a car and the road must represent both to help predict neighbors on either side. These are complementary rather than redundant capacities: global consistency is about recognizing that different views belong together (useful for image-level tasks and coarse spatial reasoning), while local context is about understanding the fine-grained geometry of nearby patches (essential for per-pixel prediction).

The paper's practical demonstration that simultaneous training is better than sequential training (Table 11, row E: CLIP → MIM achieves 75.2 mIoU vs. 79.0 for simultaneous) further supports the synergy claim. In sequential training, the MIM objective is retrofitting a representation already optimized by CLIP alone — the patch features have already converged to a CLIP-optimal configuration that may not be the right starting point for MIM. In simultaneous training, the patch features co-evolve under all objectives from initialization, allowing the model to find representations that satisfy all constraints jointly rather than sequentially compromising.

This finding changes how researchers should think about multi-objective pretraining. Rather than asking "which single auxiliary objective is best?", the paper suggests the right question is "which complementary aspects of representation quality can be targeted by different objectives, and how do they interact?" The specific pair (self-distillation + MIM) is one instance of this principle, but it opens the door to exploring other combinations of objectives that target different representational properties.

Innovation 3: Dual Embedding as a Mechanism for Learning from Heterogeneous Supervision Without Interference

The paper's third conceptual contribution is a specific architectural design pattern — dual [CLS] tokens with separate contrastive losses — that solves a general problem in multimodal training: how to learn from multiple text sources that encode different and potentially conflicting types of information without forcing a single representation to compromise between them.

The standard approach in CLIP training is to have one global image embedding that must align with whatever text is available. When multiple captions exist per image (as in LaCLIP, Fan et al., 2023), the typical solution is either to randomly sample one caption per iteration or to match multiple captions against the same image embedding. In both cases, the single [CLS] token receives gradients pushing it toward different and potentially incompatible goals: the web caption says "focus on the specific car model name for fine-grained discrimination," while the synthetic caption says "focus on spatial layout and background objects." The "multi-text" ablation in Table 1(B) — where both captions match against one [CLS] token — performs worse than the dual embedding approach on 3 of 5 tasks and particularly underperforms on dense prediction (segmentation drops from 73.3 to 72.1 mIoU, depth degrades from 0.588 to 0.580).

The dual embedding mechanism avoids this interference by parameterizing separate aggregation heads (the two [CLS] tokens) that attend to the same shared patch representations but learn to extract different information from them. The paper provides evidence for this specialization through attention map analysis (Appendix A.6, Figures 7 and 8): the spatially-aware token $\hat{e}_g$ (trained with synthetic captions) distributes attention more evenly across multiple objects in the scene and focuses less on "global proxy patches" (sparse background patches that Darcet et al. (2024) identified as carrying global image information in ViTs), while the object-centric token $e_g$ (trained with web captions) concentrates attention on the main object. The attention maps are visually similar at a coarse level but systematically differ in which regions they emphasize, confirming that the two tokens have learned complementary attention strategies from their different supervisory signals.

This design pattern addresses a challenge that extends beyond captions: in any multimodal training scenario where different text sources provide different types of information about the same image (e.g., question-answer pairs vs. descriptive captions; task instructions vs. conversational context; structured metadata vs. free-text descriptions), forcing a single embedding to capture all information types may produce representations that serve none optimally. The dual embedding provides a template for handling such cases: maintain a shared backbone (so patch-level computations are shared and efficient) but parameterize separate aggregation mechanisms (so different information types can be extracted without interference). The paper does not explore this generalization, but the principle is applicable beyond the specific caption types studied.

The finding that which [CLS] token to use at inference depends on the downstream task (object-centric for classification, spatially-aware for retrieval and dense prediction) also suggests a more general inference-time strategy: models trained with multiple specialized heads can be treated as providing a menu of representations, and practitioners can select the most appropriate one per task without retraining. This is more flexible than the traditional approach of training a single model optimized for a single evaluation protocol.

Innovation 4: Verifier Over-Optimization Analogy — The Difficulty-Dependent Efficacy of Dense Objectives

While not explicitly framed this way in the paper, the experimental results reveal a pattern that closely parallels the difficulty-dependent scaling behavior documented in the test-time compute literature: spatial awareness objectives (MIM, self-distillation) are most effective when the base representation already has some spatial signal to amplify, and can be counterproductive when applied to representations that lack the foundational capacity to benefit from them.

The evidence for this comes from comparing the ablations in Table 1 across task types. Adding synthetic captions alone (row B, "PaliGemma captions" only) improves segmentation by 10.1 mIoU but worsens ImageNet KNN classification by 6.9 points (76.9 → 70.0). This is because synthetic captions strip away the fine-grained object identity information (the car model name) that classification relies on, replacing it with coarse category and spatial information. The synthetic captions are not universally better — they are better for spatial tasks and worse for object-centric tasks. The full TIPS method recovers and even improves upon the classification baseline (78.8 vs. 76.9 KNN accuracy) by combining both caption types, but this requires the dual embedding to prevent the spatial caption signal from degrading the object-centric representation.

Similarly, the ablation of image augmentations in Table 11 (row D) shows that applying BYOL-style color augmentations to local crops — a standard practice in self-supervised learning that improves DINO performance — causes segmentation to collapse from 79.0 to 40.1 mIoU in TIPS. Color jitter, blur, and solarization are beneficial for learning invariant representations in pure self-supervised settings, but when the goal is dense prediction (which requires sensitivity to fine-grained texture and color constancy across image regions), these augmentations are actively harmful. This is not a failure of the augmentation strategy per se, but a misalignment between the augmentation-induced invariance and the downstream task requirement.

These findings combine to suggest a broader principle: spatial awareness objectives are not uniformly beneficial — they reshape representations in task-specific ways, and their interaction with the base training signal (caption content, augmentation strategy) determines whether the reshaping is beneficial or harmful. The paper implicitly operationalizes this principle by designing the TIPS recipe to carefully match the spatial objectives with the training data and augmentations: synthetic captions provide the content for spatial learning, simple augmentations preserve the low-level features spatial tasks need, and the self-distillation and MIM losses provide the mechanism for spatial learning. Changing any one component without adjusting the others can break the synergy.

This insight is valuable because it warns against the naive approach of simply adding self-supervised losses to any image-text model and expecting universal improvements. The success of TIPS depends on the specific combination of data (synthetic captions), augmentations (crops and flips only), and objectives (simultaneous CLIP + self-distillation + MIM), and the paper's ablation structure is designed precisely to demonstrate that removing or modifying any component degrades performance on at least some tasks. The difficulty-dependent efficacy pattern suggests that future work combining self-supervised and weakly-supervised objectives should carefully characterize which representations benefit from which auxiliary objectives, rather than assuming universal additive improvements.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation spans 8 tasks involving 16 datasets in total, covering both dense and global vision tasks. The specific datasets include: Pascal VOC (Everingham et al., 2010) and ADE20k (Zhou et al., 2017) for semantic segmentation; NYUv2 (Silberman et al., 2012) and NAVI (Jampani et al., 2023) for monocular depth estimation and surface normal estimation; ImageNet-1K (Russakovsky et al., 2015) for image classification; the Universal Embeddings Dataset (UnED, Ypsilantis et al., 2023) combining 8 retrieval domains; Flickr30K (Young et al., 2014), DOCCI (Onoe et al., 2024), and COCO (Chen et al., 2015) for image-to-text and text-to-image retrieval; and ImageNet-1K for zero-shot classification. Training data curation uses a separate pipeline (Section 3.3) with explicit deduplication against evaluation sets, removing approximately 19,000 near-duplicate images to prevent contamination.
Base model(s). The main experiments scale TIPS to a ViT-g architecture (patch size 14) with SwiGLU feed-forward networks, embedding dimension 1536, 24 attention heads, and approximately 1.1 billion image encoder parameters. The text encoder uses 12 transformer layers with the same embedding dimension and head count. Ablation experiments use the smaller ViT-B backbone to enable systematic exploration of design choices at lower computational cost. All compared baselines use models up to ViT sizes "g" or "G" at approximately 1.8 billion parameters or less, enabling fair comparison at comparable scale.
Metrics. Semantic segmentation uses mean Intersection over Union (mIoU) with a linear probe setup where classes are predicted from spatial patch features concatenated with the [CLS] token. Monocular depth estimation uses Root Mean Square Error (RMSE), with NYUv2 evaluated via linear probe predicting among 256 quantized depth bins, and NAVI evaluated with a DPT decoder. Surface normal estimation uses angular RMSE with DPT decoder. ImageNet classification uses top-1 accuracy with both K-Nearest-Neighbor (KNN, k=20) and linear probe evaluations. Fine-grained retrieval uses recall@1 (R@1) across UnED's 8 domains. Image-to-text and text-to-image retrieval both use R@1. Zero-shot classification uses top-1 accuracy following the CLIP protocol.
Baselines. The paper benchmarks against a comprehensive set of methods spanning three paradigms. Self-supervised methods: DINO (Caron et al., 2021), MAE (He et al., 2022), iBOT (Zhou et al., 2022), and DINOv2 (Oquab et al., 2024). Weakly-supervised image-text methods: CLIP (Radford et al., 2021), OpenCLIP (Cherti et al., 2023), SigLIP (Zhai et al., 2023), MaskCLIP (Dong et al., 2023), SILC (Naeem et al., 2024), and EVA-CLIP (Sun et al., 2023). Supervised method: ViT-g trained on JFT-3B (Zhai et al., 2022). All baselines are used with frozen weights for direct comparability to TIPS's off-the-shelf evaluation protocol.
Generation budget / compute accounting. Training compute is measured in TPU-days: ViT-B models train for 4 days on 256 TPUv3 chips (approximately 1,024 TPUv3-days), while ViT-g low-resolution training takes 2 days on 512 TPUv5 chips (approximately 1,024 TPUv5-days). The high-resolution fine-tuning stage adds only 0.1 epochs at batch size 4,096, representing a negligible fraction of total compute. At inference time, all evaluations use frozen features with identical input resolutions across compared methods, ensuring that test-time compute costs are equivalent and the comparison reflects representational quality rather than compute disparities.
Cross-validation / statistical protocol. The paper does not employ formal cross-validation or statistical significance testing. For the ViT-g model, training stops at 15 epochs "due to evaluation saturation" — downstream performance plateaus even as training loss continues decreasing, serving as a practical early stopping criterion. High-resolution fine-tuning starts from the epoch 13 checkpoint. The ablation experiments train ViT-B models for a fixed 70 epochs, with results reported as single-run numbers without confidence intervals or error bars. The comparison against baselines uses publicly available pretrained models evaluated under identical frozen-feature protocols, with the paper providing specific model versions and evaluation setups in Appendices A.4 to enable exact reproduction.

Main Quantitative Results

Synthetic Caption and Loss Combination Ablations (Table 1)

Table 1 presents the paper's central causal decomposition, isolating the contributions of enhanced textual supervision and spatial losses using a ViT-B backbone evaluated on 5 representative tasks spanning dense, global, and multimodal evaluation.

Headline findings from synthetic captions (Section B of Table 1). Replacing noisy web captions with PaliGemma-generated synthetic captions while keeping the CLIP objective unchanged improves Pascal VOC segmentation from 64.4 to 74.5 mIoU (+10.1 points) and reduces NYUv2 depth RMSE from 0.620 to 0.544 (−0.076). This is a large effect: segmentation improves to a level that already exceeds the DINO-B baseline (66.4 mIoU in Table 2). However, the same substitution degrades ImageNet KNN classification from 76.9% to 70.0% (−6.9 points), confirming the trade-off between spatial and object-centric information in the caption content.

The dual embedding approach recovers and improves upon baseline performance across all tasks when combining both caption types: Pascal VOC segmentation reaches 73.3 mIoU, NYUv2 depth 0.588 RMSE, ImageNet KNN 78.3%, Flickr I→T retrieval 88.7 R@1, and Flickr T→I retrieval 77.1 R@1. This outperforms the alternative caption combination strategies ("sampled" and "multi-text" from Fan et al., 2023) in 3 of 5 tasks and is competitive on the remaining 2, with the "sampled" strategy achieving slightly better I→T retrieval (90.2 vs. 88.7) but worse dense performance.

Headline findings from spatial losses (Section C of Table 1). Adding self-distillation to CLIP with web captions improves all tasks: segmentation from 64.4 to 70.3 mIoU (+5.9), depth from 0.620 to 0.589 (−0.031), ImageNet KNN from 76.9% to 79.1% (+2.2), I→T retrieval from 79.1 to 81.5 (+2.4), and T→I retrieval from 62.9 to 67.0 (+4.1). This replicates and extends SILC's findings (Naeem et al., 2024) to image-only dense tasks.

Adding MIM on top of self-distillation (CLIP + self-distillation + MIM, still with web captions only) delivers further dense task improvements: segmentation from 70.3 to 75.9 mIoU (+5.6, now 11.5 points above the CLIP baseline), depth from 0.589 to 0.511 (−0.078, now −0.109 below baseline), while maintaining global performance (KNN 79.0%, I→T 82.6, T→I 67.6). This validates that MIM provides complementary spatial supervision that self-distillation alone cannot supply.

Headline findings from combining both improvements (Section D of Table 1). The full TIPS method (CLIP + self-distillation + MIM + dual embedding with both captions) achieves: Pascal VOC 79.0 mIoU (+14.6 over baseline), NYUv2 depth 0.478 RMSE (−0.142 over baseline), ImageNet KNN 78.8% (+1.9), Flickr I→T 89.2 R@1 (+10.1), Flickr T→I 77.3 R@1 (+14.4). The improvement from baseline to full method is substantially larger than the sum of improvements from captions alone (B to baseline) and losses alone (C to baseline), supporting the paper's claim of synergy between the two mechanisms.

Comparisons Against Existing General-Purpose Methods

Image-only evaluations (Table 2). The paper compares TIPS against self-supervised, weakly-supervised, and supervised models across 9 image-only evaluations using the largest available backbone for each method.

The TIPS-g/14 HR model achieves: Pascal VOC segmentation 83.6 mIoU (best among all methods, DINOv2-g achieves 83.0), ADE20k segmentation 49.9 mIoU (exceeding DINOv2-g's 49.0), NYUv2 depth RMSE 0.353 (compared to DINOv2-g's 0.344, a narrow 0.009 gap), NAVI depth RMSE 0.058 (DINOv2-g: 0.054), NYUv2 normals 21.9° RMSE (DINOv2-g: 20.5°), NAVI normals 24.2° RMSE (DINOv2-g: 24.0°), UnED retrieval 68.2 R@1 (versus SigLIP-SO's best of 70.8), ImageNet KNN 83.3% (OpenCLIP-G: 83.2%, DINOv2-g: 83.5%), and ImageNet linear probe 86.2% (SigLIP-SO: 86.4%).

TIPS achieves the best or second-best performance in 7 of the 9 evaluations. The two evaluations where TIPS does not make the top two are UnED retrieval (where SigLIP-SO leads at 70.8 vs. TIPS's 68.2) and ImageNet linear probe (where SigLIP-SO achieves 86.4% vs. TIPS's 86.2%). Critically, TIPS is competitive with DINOv2 on dense tasks while dramatically outperforming all other image-text methods: SigLIP-SO, despite its strong global performance, achieves only 67.8 mIoU on Pascal VOC segmentation and 0.580 RMSE on NYUv2 depth — substantially worse than TIPS and even worse than the much smaller CLIP-L (74.5 and 0.553 respectively). The paper notes this striking pattern: "even CLIP-L, with much worse performance on image-level prediction tasks, outperforms the recent SigLIP-SO on all 6 dense evaluations."

The released [rel] TIPS models perform marginally below the non-released versions (due to the omission of Mapillary SLS from training data), but maintain the same competitive positioning: [rel] TIPS-g/14 HR achieves 83.1 Pascal VOC mIoU, 49.4 ADE20k mIoU, 0.363 NYUv2 depth RMSE, and 86.1% ImageNet linear probe accuracy.

Image-text evaluations (Table 3). TIPS is compared against weakly-supervised image-text methods on multimodal retrieval and zero-shot classification, since self-supervised methods cannot perform these tasks.

The TIPS-g/14 HR model achieves: COCO I→T retrieval 74.0 R@1 (SILC-G: 73.2, SigLIP-SO: 70.2), Flickr I→T 93.0 R@1 (OpenCLIP-G: 92.9, SigLIP-SO: 91.0), DOCCI I→T 57.2 R@1 (CLIP-L: 44.4, SigLIP-SO: 27.5 — a dramatic 29.7 point gap attributed to SigLIP's 16-token text length limit), COCO T→I 59.4 R@1 (SILC-G: 54.7, SigLIP-SO: 52.0), Flickr T→I 84.5 R@1 (OpenCLIP-G: 79.5, SigLIP-SO: 75.3), DOCCI T→I 58.8 R@1 (CLIP-L: 40.4, SigLIP-SO: 28.4), and ImageNet zero-shot classification 79.9% (SILC-G: 83.7%, SigLIP-SO: 83.2%).

TIPS achieves the best or second-best performance in 6 of the 7 evaluations. The one task where TIPS does not rank in the top two is ImageNet zero-shot classification (79.9% vs. SigLIP-SO's 83.2% and SILC-G's 83.7%). The paper attributes this partly to TIPS using the object-centric [CLS] token for zero-shot classification, which is trained with noisier web captions that may be less aligned with the clean ImageNet class name prompts. SIGLIP's stronger zero-shot performance despite worse dense features reinforces the paper's central thesis that optimizing for global alignment alone does not produce spatially-aware representations.

Performance across model scales (Tables 6 and 7). The released TIPS variants, produced through knowledge distillation from the ViT-g teacher, demonstrate that spatial awareness and strong multimodal performance are preserved across a range of model sizes. The [rel] TIPS-L/14 HR model (487M parameters) achieves: Pascal VOC 83.9 mIoU (actually exceeding the ViT-g teacher's 83.1), ADE20k 49.5 mIoU (exceeding the teacher's 49.4), NYUv2 depth 0.372 RMSE, ImageNet linear probe 85.2%, COCO I→T 73.3 R@1, Flickr T→I 83.5 R@1, and ImageNet zero-shot 79.2%. The even smaller [rel] TIPS-B/14 HR (195M parameters) achieves 82.9 Pascal VOC mIoU, 48.0 ADE20k mIoU, and 0.379 NYUv2 depth RMSE — outperforming the much larger DINO-B on segmentation and approaching it on depth.

The SO-400m variant (861M parameters) surpasses the ViT-g teacher in 5 of the 9 image-only evaluations despite having nearly half the parameters, demonstrating that distillation is an effective strategy for producing compact TIPS models.

Application: Single-Image to 3D Reconstruction (Table 4)

To validate that TIPS's spatial features transfer to 3D tasks, the paper evaluates TIPS-B/14 against DINO-B/16 as the frozen image encoder in the LRM (Hong et al., 2024) framework for single-image novel view synthesis on the Objaverse dataset (Deitke et al., 2023).

TIPS-B/14 achieves a PSNR of 21.75 compared to DINO-B/16's 21.13, an improvement of 0.62 dB. This result is notable because LRM's authors explicitly chose DINO over CLIP-style encoders due to DINO's superior structural and texture information for 3D reconstruction. TIPS bridges this gap: it provides the structural features needed for 3D reconstruction while also offering language alignment (though the language capability is not used in this evaluation). The paper includes qualitative renderings in Appendix Figure 6 showing that LRM-TIPS produces more accurate geometry than LRM-DINO in the depth renders.

Ablation Studies and Robustness Checks

Masking ratio and strategy (Table 11, sections B and C): Random masking at 75% (the TIPS default) produces the best trade-off between dense and global task performance. Lower masking ratios (50%, 25%) improve global tasks modestly (KNN classification increases from 78.8% to 79.1-79.3%) but significantly degrade dense tasks (depth RMSE worsens from 0.478 to 0.501-0.533), confirming that high masking ratios are critical for learning spatially-aware patch representations. Blockwise masking (as used in iBOT) performs slightly worse than random masking across most configurations: at 75% masking, blockwise achieves 79.5 mIoU vs. random's 79.0 on segmentation, but random masking generalizes better to other tasks and is simpler to implement. The paper adopts random masking for its simplicity and competitive performance.

Image augmentation strategy (Table 11, section D): The paper tests three augmentation variants against the TIPS default of only random resize crops and horizontal flips. Applying BYOL-style augmentations (color jitter, Gaussian blur, solarization) to local crops only causes dense task collapse: Pascal VOC segmentation drops from 79.0 to 40.1 mIoU, and NYUv2 depth RMSE worsens from 0.478 to 0.878. This is the most dramatic ablation result in the paper and provides strong evidence that heavy color augmentations are incompatible with learning features for dense prediction — these tasks require sensitivity to color constancy and texture continuity that color jitter explicitly destroys. BYOL augmentations applied to both global and local crops are less catastrophic for dense tasks (79.4 mIoU, 0.490 depth) but degrade global tasks (KNN drops from 78.8% to 77.7%, I→T retrieval from 89.2 to 88.7). Simply resizing the global crop to square (without random cropping) degrades depth RMSE from 0.478 to 0.514, confirming that random crop augmentation contributes to spatial robustness. The paper concludes that complex augmentations provide no benefit and are actively harmful for dense prediction, while simple spatial augmentations suffice.

Simultaneous vs. sequential training (Table 11, section E): Training CLIP first and then MIM sequentially ("CLIP → MIM," similar to the EVA approach) achieves 75.2 mIoU on Pascal VOC, substantially worse than the simultaneous training default (79.0 mIoU). Sequential training also degrades global tasks: ImageNet KNN drops from 78.8% to 67.2%, I→T retrieval from 89.2 to 78.0, and T→I retrieval from 77.3 to 65.9. This confirms that the positive interactions between CLIP, self-distillation, and MIM depend on co-training from initialization — applying MIM as a post-hoc fine-tuning step cannot recover the synergies achieved when representations co-evolve under all objectives.

Synthetic caption variants (Table 8): The paper ablates three synthetic caption types using a CLIP ViT-B: full PaliGemma captions (with spatial descriptions), an "object list" variant created by prompting Gemini 1.5 Flash to extract only the objects mentioned in the PaliGemma caption (e.g., "black SUV, building"), and a "main object" variant extracting only the primary object (e.g., "black SUV"). The object list variant already provides significant dense task improvements over noisy captions (Pascal VOC 73.8 mIoU vs. 64.4, NYUv2 depth 0.575 vs. 0.620), indicating that simply listing multiple objects without noisy web terms helps spatial learning. The full PaliGemma captions with spatial descriptions further improve depth (0.544 RMSE) and provide much better retrieval performance than the object list variant (79.8 vs. 66.4 I→T retrieval), suggesting that spatial prepositions and attribute descriptions are specifically useful for multimodal alignment. The main object variant performs poorly across all tasks, confirming that multi-object scene descriptions are necessary for rich representation learning.

Training dataset versions (Table 9): Training CLIP ViT-B on the raw 10B WebLI dataset produces weak dense performance (ADE20k 29.1 mIoU, NYUv2 depth 0.698). English and quality filtering to 1.7B images improves to 31.5 and 0.632. The curated 116M dataset achieves 31.6 and 0.620 — statistically equivalent on dense tasks but substantially better on fine-grained retrieval (62.9 vs. 59.3 R@1 on UnED). This demonstrates that the curation process improves data quality enough that training on 116M curated images outperforms training on 1.7B quality-filtered images while being 14.5× smaller. Importantly, the curation does not significantly change dense task performance, suggesting that the dense task gains from TIPS are driven by the method (synthetic captions, spatial losses) rather than by dataset curation.

WebLI vs. DataComp training data (Table 10): To verify that TIPS's gains are not specific to the WebLI dataset, the paper replicates the ViT-B experiments on the public DataComp dataset (Gadre et al., 2023) processed through an identical curation pipeline, yielding 115M image-text pairs. Results are nearly identical: TIPS on DataComp achieves 79.1 vs. 79.0 Pascal VOC mIoU on WebLI, 0.479 vs. 0.478 NYUv2 depth RMSE, and 73.4% vs. 78.8% ImageNet KNN. The CLIP baseline on DataComp similarly matches WebLI (64.3 vs. 64.4 mIoU, 0.620 vs. 0.620 depth). This confirms that the TIPS method, not the specific training data source, drives the improvements.

Dual embedding attention analysis (Appendix A.6, Figures 7 and 8): The paper validates that the two [CLS] tokens learn complementary attention patterns by analyzing attention maps on 500 randomly selected MSCOCO images. The spatially-aware token distributes attention more evenly across multiple objects and assigns less attention to "global proxy patches" (sparse background patches identified by Darcet et al., 2024, that encode global image information). Quantitatively, the spatially-aware token exhibits lower maximum attention weights, lower standard deviation of attention weights, and higher entropy of the attention distribution compared to the object-centric token, confirming that it attends to a broader set of image regions. Filtering out the global proxy patches (via median filtering) reveals that the spatially-aware token focuses more on secondary objects and background elements, while the object-centric token remains concentrated on the main object. This provides mechanistic evidence for the dual embedding's claimed benefit.

Self-supervised components interaction (Table 11, section A vs. C): The complete TIPS method (79.0 mIoU, 0.478 depth, 78.8% KNN) outperforms the CLIP + self-distillation + MIM variant with only web captions (75.9 mIoU, 0.511 depth, 79.0% KNN) by 3.1 mIoU on segmentation and 0.033 on depth RMSE. This quantifies the contribution of synthetic captions specifically when spatial losses are already present — the caption enhancement provides additional gains even after self-distillation and MIM have already substantially improved the features, confirming that better textual supervision and better spatial objectives are complementary rather than substitutes.

Critical Assessment

The experimental design systematically tests the paper's two central claims: (1) that synthetic captions with spatial descriptions improve dense prediction performance in image-text models, and (2) that combining self-distillation and masked image modeling with contrastive image-text learning yields synergistic improvements beyond either component alone. The evidence for both claims is strong and well-structured through the progressive ablation format in Table 1, which shows each component's additive contribution and the combined method's superiority over all partial configurations.

However, several aspects of the experimental design warrant scrutiny and limit the generality of the conclusions.

The paper demonstrates that TIPS works on the specific suite of tasks selected, but does not establish that the reported performance generalizes beyond these benchmarks. All 16 evaluation datasets are standard academic benchmarks with specific characteristics (indoor scenes for NYUv2, object-centric captures for NAVI, web-style photographs for Flickr/COCO). There are no evaluations on domains that differ substantially from the training data's web-image distribution — no medical imaging, satellite imagery, industrial inspection, or robotics-specific datasets. The Mapillary SLS addition partially addresses the street-view domain gap, but its impact is not isolated in an ablation (the paper does not report TIPS-g performance with and without Mapillary data), so the sensitivity to training data domain composition is unknown. A practitioner deploying TIPS for an application involving non-web imagery (e.g., endoscopic images, agricultural drone footage) cannot infer from these experiments whether the features will transfer.

The single-run reporting without statistical measures limits the ability to assess result reliability. Training one ViT-g model and reporting point estimates for 16 datasets means the reported numbers are subject to training variance that the paper does not quantify. The ViT-B ablations (Table 1) are also single runs across 5 tasks. Given that the ViT-g training stops at an "evaluation saturation" point determined by downstream metrics, the reported numbers may be optimistic — the stopping criterion is implicitly a form of test-set-guided model selection, even if only a coarse one (epoch-level). The paper would be strengthened by reporting standard deviations across multiple training runs, at least for the ViT-B ablation experiments where computational cost is manageable.

The comparison against baselines uses publicly available pretrained models evaluated under identical frozen-feature protocols, which is a strength for reproducibility but introduces a subtle confound: the baseline models were not trained with the same dataset or compute budget as TIPS. DINOv2 uses LVD-142M (a 142M-image dataset curated from a larger pool than WebLI), while CLIP and SigLIP use proprietary datasets at unknown scales. The DataComp comparison (Table 10) partially addresses this by showing that CLIP vs. TIPS on identical data yields the expected gains, but the main Tables 2 and 3 comparisons are between models trained on different data, making it impossible to attribute performance differences purely to method vs. data scale. The paper implicitly acknowledges this by including the DataComp ablation, but does not extend it to ViT-g scale.

The paper claims TIPS "achieves the best or second-best performance in 7 out of 9 evaluations" in Table 2, but this framing downplays the qualitative difference in TIPS's value proposition. TIPS's primary advantage is not that it marginally beats DINOv2 on some dense tasks and SigLIP on some global tasks, but that it is competitive with both simultaneously while neither competitor can perform the other's tasks. The "best or second-best" framing treats all tasks as equally weighted components of a scorecard, obscuring the fact that DINOv2's dense performance comes with zero multimodal capability, and SigLIP's retrieval performance comes with severely degraded dense features. A more informative presentation would emphasize the range of tasks where TIPS is within a small margin of the respective specialist, rather than counting how many times it wins or places second.

The high-resolution fine-tuning stage is extremely short (0.1 epochs) yet provides noticeable improvements (Table 2: Pascal VOC from 82.9 to 83.6, NYUv2 depth from 0.377 to 0.353). The paper describes this as adapting to higher resolution, but does not ablate whether simply increasing resolution at test time without the fine-tuning stage would achieve similar gains. If the LR model evaluated at 448×448 resolution (without HR fine-tuning) performs worse, that would indicate a genuine resolution generalization failure that the fine-tuning addresses. If it performs similarly, the HR fine-tuning is unnecessary. This ablation is missing, making it unclear how much the HR improvements come from resolution adaptation vs. the additional training steps themselves.

The dual embedding mechanism's contribution is measured through the attention map analysis (Appendix A.6), but the functional benefit of having two separate [CLS] tokens vs. a single token trained with both losses is not fully isolated. The ablation in Table 1(B) compares dual embedding against "sampled" and "multi-text" alternatives, but a missing ablation is: what if the model had a single [CLS] token but with two separate contrastive losses (both captions matched against the same embedding, but with the loss computed as the average of two InfoNCE terms)? This would separate the loss computation from the representation architecture, testing whether the dual [CLS] is necessary or whether the separate losses alone suffice. The "multi-text" variant (Table 1B) matches both captions against the same embedding in a single loss, which is different from having two separate losses on the same embedding. This distinction matters for understanding whether the architectural innovation (dual tokens) or the loss formulation (separate contrastive terms) drives the improvement.

The paper presents TIPS as a general-purpose model that can be used off-the-shelf, but the inference-time choice of which [CLS] token to use per task requires task-specific knowledge that amounts to a hyperparameter. For dense tasks, $\hat{e}_g$ (synthetic caption token) is used. For classification, $e_g$ (web caption token) is used. For retrieval, $\hat{e}_g$ is used. This is not a single model producing a single representation — it is a model with two output modes, and the practitioner must know which mode to query. The paper does not provide a principled way to make this choice without evaluating both on the target task, which partially undermines the "off-the-shelf" claim. An experiment showing that the wrong choice degrades performance (e.g., using $e_g$ for depth estimation) would quantify the sensitivity and help practitioners understand the stakes.

The NAVI depth and normal estimation tasks use a DPT decoder while the NYUv2 depth task uses a linear probe — different probing architectures across datasets within the same task category. This makes cross-dataset comparisons of absolute numbers less meaningful (e.g., NYUv2 depth RMSE of 0.353 vs. NAVI depth RMSE of 0.058 reflect both dataset difficulty and probing protocol differences). The paper is transparent about this in Appendix A.4, but the main results tables (Table 2) present these numbers side-by-side without flagging the protocol difference, which could mislead readers into comparing NYUv2 and NAVI numbers directly.

The most significant missing experiment is a FLOPs-matched or compute-matched comparison against DINOv2 that accounts for both pretraining and inference costs. TIPS requires training a text encoder (389M parameters for ViT-g) in addition to the image encoder (1.1B parameters), and generating synthetic captions for 117M images using PaliGemma. DINOv2 trains an image encoder only and requires no external captioner. The paper reports TIPS ViT-g training takes 2 days on 512 TPUv5s, but this does not include the PaliGemma captioning cost (which processes 117M images at 224 resolution, plus another pass at 448 for HR fine-tuning), nor does it amortize the cost of curriculum learning from the teacher model during knowledge distillation for smaller variants. A compute-matched comparison would ask: given the total FLOPs budget of TIPS (captioning + training + distillation), does a DINOv2 model trained with equivalent compute achieve better or worse dense performance? The paper cannot answer this, making the efficiency claim ("TIPS rivals DINOv2 on dense tasks while adding language") contingent on the assumption that compute is not the binding constraint — an assumption that may not hold for practitioners with limited resources.

These limitations do not undermine the paper's core contributions — the ablations in Table 1 provide compelling evidence for the method's effectiveness, and the breadth of evaluation across 16 datasets is substantially more comprehensive than most comparable work. However, they constrain the strength of the claims that can be made about TIPS's generality, efficiency, and robustness. The paper demonstrates that TIPS works well on the tested benchmarks under the reported conditions; whether it works well in substantially different domains, at different scales, or under different compute constraints remains to be demonstrated.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is a Deployment Dealbreaker Unaddressed in Headline Efficiency Numbers

The assumption: The entire compute-optimal framework requires estimating per-prompt difficulty before allocating the inference budget. The paper uses a method that is computationally equivalent to the largest budgets being studied: generate 2048 samples per question and either check ground-truth correctness (oracle bins) or compute the PRM's average final-answer score (predicted bins). The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence: In a realistic deployment, total compute equals difficulty estimation plus strategy execution. For any prompt where difficulty estimation costs 2048 generations, the amortized efficiency gains over best-of-N are dramatically smaller than the reported 4× — indeed, at low-to-moderate budgets (e.g., 16–64 generations for the strategy itself), the estimation cost dominates total compute, making the "compute-optimal" approach substantially more expensive than simply running best-of-N with the same total budget. The 4× efficiency figure (Figures 4 and 8) is computed conditional on difficulty being known, without amortizing the cost of learning it. This is not a minor bookkeeping issue — it means the paper's central efficiency claim applies only in regimes where difficulty can be estimated cheaply, which the paper does not demonstrate is possible.

What evidence exists: The paper explicitly flags this gap (Section 3.2) and shows that predicted bins (using PRM scores, which still require 2048 generations) perform similarly to oracle bins (Figures 4 and 8). However, there is no experiment estimating difficulty from fewer samples — no sweep of 4, 8, 16, 32, 64, 128, 256 samples to determine the minimum number needed for reliable bin assignment. There is no experiment using a learned difficulty predictor that takes only the question text as input, despite this being the most natural solution and the one the paper suggests as future work. The paper also does not explore adaptive difficulty estimation, where a small number of initial samples inform a real-time decision about how to allocate the remaining budget — an approach that would amortize estimation cost into the solution process.

Mitigation status: The paper acknowledges this as a key area for future work (Section 8): "Our difficulty-based binning scheme still requires a reasonable amount of extra computation... we believe future work that directly predicts the difficulty of a given problem would be necessary to make adaptive test-time strategies more practical." This is candid but does not address the limitation — the current method is not deployable as described without a solution to this problem, and the paper provides no evidence that a solution is feasible within reasonable cost bounds. A practitioner reading the paper should understand that the 4× efficiency gains are an upper bound contingent on solving the difficulty estimation problem, not a realized deployment gain.

6.2 Hard Problems Are Fundamentally Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The assumption: The paper's FLOPs-matched comparison (Section 7) and the broader compute-optimal framework assume that problems lie within some reasonable range of the base model's capability — that is, the base model's pass@1 rate is non-trivially above zero, so test-time compute can amplify existing signals. The paper does not claim otherwise, but the boundary of where this assumption holds is stark and may be narrower than practitioners expect.

The consequence: For the hardest problems (difficulty bin 5, where the base model's pass@1 is near zero), test-time compute provides essentially no benefit regardless of budget allocation. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets (from 4 to 256 generations). In Figure 7 (right), the revision model's bin 5 accuracy is roughly 2–3% regardless of the sequential-to-parallel ratio, even at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both revisions and search across all budgets. The paper states this explicitly in the Section 7 takeaway box: test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address."

This means there is a hard ceiling: on genuinely difficult problems outside the base model's training distribution, no amount of test-time compute — search, revisions, or their combination — will produce correct answers. A practitioner facing a deployment where even a modest fraction of queries fall into this regime cannot rely on test-time compute to close the gap and must invest in better pretraining instead. The paper's finding that a 14× larger model trained with more pretraining compute outperforms test-time compute on hard problems at nearly all $R$ values (Figure 9) quantifies this boundary.

What evidence exists: The difficulty-bin analyses across Sections 5 and 6 consistently show flat or near-flat performance on the hardest bin. Bin 5 is defined as the quintile of questions where the base model's pass@1 is lowest (approximately 0–3% based on the 2048-sample estimate). The fact that 512 generations of beam search, 256 generations of revision chains, and all possible ratios of sequential-to-parallel sampling all fail to move bin 5 accuracy above low single digits provides strong evidence that this is a fundamental capability boundary, not a suboptimal allocation problem.

Mitigation status: The paper is transparent about this limitation — the Section 7 discussion explicitly states: "Test-time compute can amplify existing capability but does not create it from nothing." However, the paper does not provide guidance on how to determine whether a given problem falls into this regime without the expensive 2048-sample difficulty estimation. The difficulty estimator can identify which problems are hard (bin 5) post-hoc, but by the time 2048 samples have been generated, the cost has already been incurred. For a production system, a cheap predictor of "is this problem completely outside the model's capability?" would be as valuable as a difficulty estimator, but neither is developed.

6.3 Single Benchmark, Single Model Family — Generality Beyond MATH and PaLM 2-S* Is Unverified

The assumption: All experiments are conducted on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not verified through experiments on other models or benchmarks.

The consequence: Several aspects of the paper's findings could be model-specific or benchmark-specific, and a practitioner using a different model family (e.g., LLaMA, GPT, Gemini) or a different domain (e.g., code generation, scientific reasoning, factual QA) cannot confidently extrapolate the quantitative findings:

PRM quality and over-optimization behavior depend on the base model's output distribution. The Monte Carlo rollout training procedure (Appendix D) produces a PRM calibrated to PaLM 2-S*'s specific error patterns and output characteristics. A model with different calibration, different typical solution structures, or different reasoning patterns might produce a PRM with different over-optimization thresholds, changing the optimal strategy per difficulty bin.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities. PaLM 2-S*'s specific capacity for few-shot learning from its own outputs determines how effectively the revision model can improve across sequential steps. Different model families exhibit substantially different in-context learning behaviors.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and producing closed-form answers amenable to exact string matching. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems (due to PRM over-optimization), sequential revisions dominating on easy problems, balanced ratios being optimal on hard problems — generalize to other reasoning domains or to tasks requiring open-ended generation without clean correctness signals.

What evidence exists: The paper provides no cross-model or cross-benchmark experiments. The 500-question test set is split into five difficulty quintiles of approximately 100 questions each, and with two-fold cross-validation, the compute-optimal policy is selected based on approximately 50 questions per fold per bin. This is a small sample, and small-sample variance in the difficulty-dependent strategy selection could affect which strategies are chosen as "optimal." The paper does not report confidence intervals on the compute-optimal scaling curves, so a practitioner cannot assess whether the observed differences between strategies at a given budget and difficulty level are statistically reliable.

Mitigation status: The paper does not attempt to address this limitation experimentally. The authors acknowledge the single-model limitation implicitly in Section 4 by stating their belief about representativeness, but this is an assertion, not evidence. The public release of models and code is noted, which enables third-party replication, but the paper itself provides no evidence of generality beyond its specific setup. A practitioner deploying these techniques with a different model family should treat the paper's quantitative findings (specific difficulty thresholds, optimal ratios, 4× efficiency figures) as hypotheses to validate rather than established transferable facts.

6.4 Verifier Over-Optimization Is a Hard Ceiling That Compute-Optimal Allocation Mitigates but Does Not Solve

The assumption: The compute-optimal policy works by routing easy problems away from aggressive search (beam search, lookahead) and toward weaker optimization (best-of-N) to stay below the verifier's over-optimization threshold. The implicit assumption is that the verifier is reliable enough on each difficulty tier to support the optimization strength deployed.

The consequence: On the problems where beam search is deployed (medium-difficulty bins 3–4), over-optimization still limits the scaling ceiling. Figure 3 (right) shows that beam search performance on bin 3 flattens and may decline as the budget increases from 64 to 256 generations, even though beam search was selected as the optimal strategy for this bin. The PRM's reliability is not perfect on any difficulty tier, and aggressive search eventually exploits its imperfections regardless of problem difficulty. This means the compute-optimal approach is fundamentally bounded by verifier quality — improving the PRM would shift the difficulty thresholds and raise the ceiling, but with the current PRM, there is an absolute performance cap that no allocation strategy can exceed.

The practical implication is that efforts to further scale test-time compute (e.g., from 256 to 1024 or 4096 generations) may yield diminishing or even negative returns unless accompanied by improvements to the verifier. The paper's finding that lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left) is the clearest evidence that optimization power is not the bottleneck; verifier reliability is. A practitioner investing in test-time compute infrastructure should prioritize better PRM training (more on-policy data, adversarial robustness, ensembling) over more sophisticated search algorithms.

What evidence exists: The degradation of beam search at high budgets on easy problems (Figure 3, right), the underperformance of lookahead search across the board (Figure 3, left), and the qualitative examples in Appendix M showing degenerate search outputs (repetitive low-information steps, overly short solutions) collectively demonstrate that verifier over-optimization is the primary limiting factor. The paper's compute-optimal policy can be understood as a way to route around this limitation — not solving it, but avoiding the regimes where it causes the most damage.

Mitigation status: The paper identifies over-optimization as a key challenge (Section 8: "over-optimization of the PRM... is the primary bottleneck preventing unbounded improvements from additional compute") but does not propose improvements to the verifier. The PRM training procedure (Monte Carlo rollouts from the base model) produces a verifier with the documented limitations, and no experiments explore whether different PRM training strategies (e.g., on-policy data from search trajectories, adversarial training, ensemble methods) produce verifiers that are more robust to optimization pressure. The paper frames verifier robustness as important future work rather than something addressed in the current study. A practitioner should understand that the reported performance levels are specific to the verifier quality achievable with the described PRM training procedure and may be improvable with better verifiers.

6.5 Sequential Revisions Cannot Be Parallelized, Making Latency a Hidden Cost Ignored in Paper's Compute Accounting

The assumption: The paper measures compute in "generations" (number of complete solutions sampled), treating all generations as having equal cost in a FLOPs-equivalent accounting. This is appropriate for throughput-focused analysis but ignores wall-clock latency entirely. Sequential revisions are inherently serial — each revision depends on the previous one — while parallel best-of-N can be executed simultaneously given sufficient hardware.

The consequence: The paper's compute-optimal policy for revisions favors sequential-heavy allocations on easy-to-medium problems (Figure 7). For example, at a budget of 128 generations, the optimal strategy for medium-difficulty problems might allocate 64 generations as sequential revisions with 2 parallel chains, meaning 64 sequential steps per chain. With sufficient hardware, parallel best-of-128 could execute all 128 samples simultaneously, completing in approximately the time of one generation. The sequential-heavy strategy requires 64 serial forward passes, taking approximately 64× longer wall-clock time. For latency-sensitive applications — interactive assistants, real-time decision-making systems, user-facing chat — this is a prohibitive cost regardless of FLOPs efficiency.

The paper reports substantial gains from sequential revisions (e.g., Figure 6, right: sequential + best-of-N weighted achieves roughly 41.5% at 64 generations vs. roughly 39% for parallel), but these gains come at a latency multiplier proportional to the chain length. The paper never measures or discusses latency, making it impossible for a practitioner to assess whether the accuracy-latency tradeoff is acceptable for their use case. In many production settings, a 2.5 percentage point accuracy gain that comes with a 64× latency increase would be unacceptable, and the parallel baseline would be preferred despite its lower accuracy.

What evidence exists: The paper's revision model experiments (Section 6) show the performance benefits of longer sequential chains (Figure 6, left, showing pass@1 improving from ~18% at step 1 to ~24% at step 20), and the ratio sweeps (Figure 7) demonstrate that sequential-heavy allocations are often optimal for accuracy. However, no experiment measures or reports latency, throughput, or the time-to-solution for different strategy allocations. The generation budget is the sole cost metric throughout.

Mitigation status: The paper does not acknowledge the latency-accuracy tradeoff as a limitation. The compute accounting is consistently in terms of "generations" or FLOPs, with wall-clock time never mentioned. This is standard in the test-time compute literature (which prioritizes FLOPs efficiency) but creates a gap for practitioners who face latency constraints. A full treatment of the problem would include a latency-aware allocation policy that treats wall-clock time as part of the budget, potentially producing different optimal strategies for latency-constrained vs. throughput-constrained deployments. The paper's current results should be understood as applying to throughput-oriented batch settings (e.g., evaluating a test set, generating training data) where latency per query is not the binding constraint.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes the relationship between weakly-supervised image-text learning and self-supervised image-only learning from a zero-sum competition into a compatibility that can be exploited. Before TIPS, the vision community tacitly accepted a bifurcation: if you needed spatial features for dense prediction, you used DINO or DINOv2; if you needed language-aligned features for retrieval or zero-shot classification, you used CLIP or SigLIP. The two paradigms were treated as incommensurate — optimizing for one implicitly sacrificed the other. TIPS demonstrates that this trade-off is not fundamental but is instead an artifact of two correctable deficiencies: the poverty of web captions as spatial supervision (they name objects, not layouts) and the absence of explicit spatial coherence objectives in standard contrastive training (the loss supervises only the [CLS] token, leaving patch tokens to drift).

The evidence for this reframing is the paper's Table 1, which shows that simply replacing noisy web captions with PaliGemma-generated descriptions improves Pascal VOC segmentation by 10.1 mIoU in a standard CLIP ViT-B — without any architectural changes or additional objectives. This single result implies that a substantial fraction of the "spatial gap" between image-text and self-supervised models is attributable to caption content, not to inherent limitations of language alignment. The subsequent addition of self-distillation and MIM objectives further narrows the gap, with the full TIPS ViT-g reaching within 0.009 RMSE of DINOv2-g on NYUv2 depth while simultaneously achieving multimodal retrieval performance that DINOv2 cannot match at all.

This shifts the research landscape in three specific ways. First, it makes improving caption quality — through synthetic generation, multi-source fusion, or better data filtering — a first-class research direction for dense prediction in multimodal models, rather than a data-cleaning afterthought. Prior work on synthetic captions (LaCLIP, VeCLIP) evaluated only on global tasks; TIPS provides the first evidence that caption content directly controls spatial feature quality, and identifies what specific linguistic elements matter: spatial prepositions ("in front of"), background object mentions ("building"), and attribute descriptions ("black SUV"). This gives future work a clear target for what synthetic captions should contain.

Second, it demonstrates that self-supervised spatial objectives (self-distillation, MIM) are not alternatives to language supervision but complementary mechanisms that operate on different representational scales. Self-distillation enforces cross-scale consistency (local crop ↔ global crop), while MIM enforces local spatial context (visible patches ↔ masked neighbors). Both improve dense features, but through different pathways, and their simultaneous application produces gains neither achieves alone (Table 1: CLIP + self-distillation + MIM achieves 75.9 mIoU vs. 70.3 for CLIP + self-distillation alone). This suggests that the research agenda should shift from "which single auxiliary objective is best?" to "which complementary aspects of representation quality can different objectives target?", opening the door to systematic exploration of objective combinations.

Third, TIPS resolves a latent contradiction in prior work. SILC (Naeem et al., 2024) showed that self-distillation helps image-text models, but did not demonstrate dense prediction improvements comparable to self-supervised models. MaskCLIP (Dong et al., 2023) showed that MIM helps, but similarly fell short on dense tasks. Both were correct in their positive findings but incomplete in their diagnosis: neither alone suffices to close the spatial gap, and neither addresses the caption content problem. TIPS reconciles these by showing that the combination of better captions + both self-supervised objectives + simultaneous training (not sequential, as in EVA) is what crosses the threshold into competitive dense performance. Prior negative findings — that CLIP-style models cannot match self-supervised models on dense tasks — were valid for the specific configurations tested, but TIPS shows they were not universal.

The paper also implicitly redirects attention away from task-specific architectural adaptations for spatial understanding in multimodal models. Much prior work (DenseCLIP, CLIPSelf, OWL-ViT, open-vocabulary segmentation methods) adds specialized decoders, detection heads, or fine-tuning stages on top of CLIP features for specific spatial tasks. TIPS suggests a different path: invest in the pretraining recipe (data quality, augmentations, objective combination) to produce frozen features that are good enough for a simple linear probe or lightweight decoder to work well across tasks. This is a harder standard but produces a more general model — and Tables 2 and 3 show it largely works.

Finally, TIPS changes the value proposition of image-text models relative to self-supervised ones. DINOv2 is still the specialist for pure dense prediction — it leads TIPS by narrow margins on NYUv2 depth and normals. But TIPS offers a different proposition: competitive dense features plus language alignment, at the cost of a small (and sometimes zero) dense prediction delta. For the many applications that need both spatial reasoning and language grounding — multimodal LLMs, text-conditioned depth estimation, retrieval-augmented 3D reconstruction — TIPS provides a single model where previously two separate encoders were needed. The single-image-to-3D experiment (Table 4), where TIPS-B/14 outperforms DINO-B/16 as the LRM encoder (21.75 vs. 21.13 PSNR), concretely demonstrates that TIPS's spatial features are not just "competitive in benchmarks" but transfer to a real 3D reconstruction task where the original authors explicitly chose DINO over CLIP for its structural information. This is a small but meaningful data point: TIPS's features have the geometric quality that practitioners previously associated exclusively with self-supervised models.

Follow-Up Research This Work Enables

Training a lightweight difficulty predictor directly from question text. The most immediate bottleneck TIPS identifies is the cost of generating 2048 samples per question to estimate difficulty. A natural follow-up would train a small classifier — potentially a distilled version of the PRM or a simple transformer — that takes only the question text (or, for vision, the image features) as input and predicts the difficulty bin. If such a predictor could achieve bin assignment accuracy comparable to the PRM-based method (e.g., within 5-10% misclassification rate), the compute-optimal framework becomes immediately practical. A strong experiment would: (1) collect difficulty labels (2048-sample PRM scores + bin assignments) for a moderate set of prompts (e.g., 10K), (2) train a lightweight predictor (e.g., a 100M-parameter model) on 80% and evaluate on 20%, and (3) compare the compute-optimal scaling curve using predicted bins from the lightweight model vs. oracle bins, measuring the efficiency loss from imperfect difficulty estimation. The key question is whether the predictor's errors are random (washing out in expectation) or systematic (consistently misclassifying certain problem types, causing suboptimal allocation on those types).

Adaptive difficulty estimation that amortizes cost into solution generation. Rather than estimating difficulty as a separate pre-processing step, a dynamic approach would interleave difficulty assessment with problem-solving: generate a small number of initial samples (e.g., 4-8), compute the PRM's average final-answer score on those samples as a quick difficulty signal, and then allocate the remaining budget accordingly. This connects to the exploration-exploitation framework the paper briefly mentions. A strong experiment would: (1) define a decision rule mapping the initial sample scores to strategy selection (e.g., if average score > 0.8 → sequential revisions; if 0.3-0.8 → beam search; if < 0.3 → best-of-N), (2) compare the total compute+accuracy curve to the static difficulty-estimation approach and to the best-of-N baseline, including the initial assessment samples in the total cost, and (3) measure whether the adaptive approach recovers most of the gains of oracle difficulty estimation while incurring only a small constant overhead (4-8 samples per problem). The key metric is whether the adaptive approach's efficiency curve falls closer to the oracle curve or to the best-of-N baseline.

Combining PRM tree-search with the revision model as the proposal distribution. TIPS studies search and revisions independently but the paper explicitly notes they are complementary: revisions improve the quality of generated candidates (proposal modification), while PRM search improves candidate selection (verifier optimization). A combined system would use the revision model to generate candidate steps within a beam search tree, with the PRM scoring partial solutions at each step and the revision context (previous incorrect attempts) providing additional information to guide generation. A strong experiment would: (1) implement beam search where the proposal distribution at each step is the revision model conditioned on the current partial solution and any previous rejected branches, (2) compare the performance-efficiency curve of the combined system against compute-optimal search alone and compute-optimal revisions alone, and (3) characterize whether combined search+revisions shifts the difficulty thresholds — e.g., does it extend the range of problems where test-time compute beats pretraining? The hypothesis (based on the paper's finding that revisions help most on easy problems while search helps most on medium problems) is that the combined system would outperform either alone across all difficulty bins, but the magnitude and the interaction effects are unknown.

Training robust PRMs resistant to over-optimization through on-policy data. The paper identifies verifier over-optimization as the primary bottleneck limiting test-time compute scaling — beam search degrades on easy problems at high budgets (Figure 3, right), and lookahead search paradoxically underperforms (Figure 3, left). A natural follow-up would train the PRM on data generated by the search process itself (on-policy trajectories) rather than on i.i.d. samples from the base model, making the PRM robust to the specific failure modes that search introduces. A strong experiment would: (1) run beam search with the current PRM, collect the trajectories where the PRM assigned high scores but the final answer was incorrect (over-optimization failures), (2) add these trajectories as negative examples to the PRM training data (with low correctness labels, since they led to wrong answers), (3) retrain the PRM, (4) re-evaluate the beam search scaling curve, particularly on easy problems where over-optimization was most severe. The key metric is whether the retrained PRM's beam search curve continues to improve at higher budgets (e.g., 256-512 generations) rather than flattening or declining. This experiment would test whether over-optimization is a fundamental limitation of learned verifiers or a consequence of the i.i.d. training data distribution.

Systematic ablation of caption content on dense feature quality. TIPS demonstrates that PaliGemma captions (which include spatial prepositions, multiple objects, and attributes) improve dense prediction over web captions, but does not isolate which linguistic features drive the improvement. A controlled follow-up would generate caption variants with systematically varied content: (1) object lists only (nouns), (2) object + attribute (adjectives + nouns), (3) object + spatial relations (nouns + prepositions), (4) full scene descriptions (PaliGemma-style), and train equivalent models on each variant, measuring performance on segmentation, depth, and normals. The hypothesis (based on Table 8's partial ablation, where "object list" helps but full captions help more) is that spatial prepositions contribute specifically to geometric tasks (depth, normals) while attribute descriptions contribute to semantic segmentation. A strong experiment would additionally measure whether these caption properties interact with model scale — e.g., do spatial prepositions matter more for small models (which need explicit guidance) than for large models (which can infer spatial layout from co-occurrence statistics)? This experiment would provide a mechanistic explanation for why synthetic captions improve dense features and would guide the design of future caption generation pipelines.

Cross-model and cross-domain replication of the difficulty-dependent scaling patterns. The paper's central finding — that optimal test-time strategy depends on problem difficulty — is established on a single benchmark (MATH) with a single model family (PaLM 2-S*). Replicating the difficulty-bin analysis on different model families (e.g., LLaMA-based, GPT-based) and different domains (e.g., code generation with HumanEval, scientific reasoning with GPQA, factual QA with Natural Questions) would determine which patterns are universal and which are model-specific. A strong experiment would: (1) replicate the 2048-sample difficulty estimation and quintile binning procedure for each model-domain pair, (2) sweep the same search and revision strategies (best-of-N, beam search, sequential/parallel revisions) at multiple budgets, (3) compare the difficulty-dependent scaling curves. The key questions: does beam search always over-optimize on easy problems, or only for certain verifier training procedures? Does sequential revision always dominate on easy problems? Are there domains where the "hard problem" ceiling (bin 5 near-zero improvement) is higher or lower? A negative result — e.g., finding that on code generation, beam search improves on easy problems because code correctness is less ambiguous than math answer grading — would refine the paper's claims about difficulty-dependent behavior and establish boundary conditions.

Practical Applications and Downstream Use Cases

Vision encoder for multimodal LLMs with spatial reasoning requirements. Multimodal LLMs (e.g., LLaVA, GPT-4V, Gemini) typically use a CLIP-style vision encoder to project images into the language model's token space. Tong et al. (2024) demonstrated that this produces visual features that systematically fail at spatial reasoning — CLIP-based models struggle with questions like "is the cup to the left or right of the plate?" because the vision encoder's patch features lack spatial coherence. Current workarounds involve running two separate encoders (CLIP for semantics + DINOv2 for geometry) and fusing their features, which doubles visual compute. TIPS provides a single encoder that achieves DINOv2-competitive dense features (Table 2: 83.6 Pascal VOC mIoU vs. DINOv2-g's 83.0, 0.353 NYUv2 depth RMSE vs. 0.344) while maintaining strong multimodal retrieval (Table 3: 93.0 Flickr I→T R@1, 84.5 Flickr T→I R@1). A multimodal LLM using TIPS as its vision backbone would get spatial awareness and semantic grounding from a single forward pass, reducing visual compute by approximately 50% compared to the dual-encoder approach while likely improving spatial reasoning accuracy. The LRM result (Table 4: TIPS-B/14 achieving 21.75 PSNR vs. DINO-B/16's 21.13) provides preliminary evidence that TIPS's spatial features transfer to 3D understanding tasks relevant to embodied AI and robotics.

Text-conditioned dense prediction for creative and editing tools. Current depth estimation (DepthAnything) and segmentation (SAM) tools are powerful but operate without language control — they always estimate depth for all pixels or segment all objects. A TIPS-based system could enable text-conditioned variants: "estimate depth only for the foreground objects," "segment the red car and the building behind it," or "find all surfaces with wood texture." Because TIPS aligns both global and patch-level features with language (via the two [CLS] tokens and their backpropagation into patch tokens), a lightweight text-conditioned decoder trained on top of frozen TIPS features could accept natural language instructions and produce spatially localized outputs. The paper's results on dense prediction with simple probes (linear layer for segmentation at 83.6 mIoU on Pascal VOC, linear probe for depth at 0.353 RMSE on NYUv2) suggest that the patch features already encode sufficient spatial and semantic information that a small amount of task-specific training would suffice. This enables interactive image editing workflows (e.g., "darken the background" → language query → depth mask → pixel manipulation) that currently require manual masking or separate models for language understanding and spatial prediction.

Data-efficient fine-tuning for specialized domains using frozen TIPS features. The paper demonstrates that frozen TIPS features work well with simple linear probes or lightweight DPT decoders (Sections 4.1 and 4.2), achieving competitive results on dense tasks without any fine-tuning of the image encoder. For specialized domains with limited labeled data — medical image segmentation (few hundred annotated scans), agricultural crop analysis (seasonal variation, few labels), or industrial defect detection (proprietary data, small sample sizes) — this is valuable: a practitioner can take the frozen TIPS ViT-g, attach a task-specific head (linear layer or small decoder), and train only that head on the limited data, achieving strong performance without the risk of overfitting a 1.1B-parameter model. The distillation results (Tables 6 and 7) show that even the 195M-parameter TIPS-B/14 retains much of the ViT-g's performance (82.9 vs. 83.1 Pascal VOC mIoU, 0.379 vs. 0.363 NYUv2 depth RMSE), meaning practitioners with compute constraints can use the smaller model and still get strong frozen features. The availability of multiple model sizes (S through g, Table 5) allows trading off accuracy against inference cost per the deployment constraints.

Single-image 3D reconstruction for e-commerce and AR applications. The LRM experiment (Table 4) shows that TIPS features improve novel view synthesis quality over DINO features in a frozen-encoder setting. For e-commerce (generating 3D product views from a single catalog image) or augmented reality (placing virtual objects in a scene with correct geometry from a single phone photo), TIPS provides a vision encoder that simultaneously captures fine-grained object identity (for recognition — what product is this?) and spatial structure (for geometry — what shape and depth does it have?) in a single model. The improved PSNR (21.75 vs. 21.13) represents perceptible quality improvements in rendered novel views, and the text alignment (which DINO lacks entirely) enables language-conditioned reconstruction (e.g., "reconstruct only the main object, ignore the background") that is not possible with pure self-supervised encoders. The distillation results mean a practical deployment could use TIPS-B/14 (195M parameters) for real-time inference on consumer devices while retaining most of the ViT-g teacher's spatial quality.

When to Prefer This Method

The paper positions TIPS as a general-purpose model that unifies capabilities previously requiring separate self-supervised and image-text encoders, but its design choices imply specific deployment tradeoffs that practitioners should consider:

Prefer TIPS over separate DINOv2 + CLIP/SigLIP encoders when:

The application requires both dense spatial prediction (depth, segmentation, normals) and language-aligned features (retrieval, zero-shot classification, text-conditioned queries) in the same system, and running two separate vision encoders is infeasible (compute, memory, or latency constraints). The TIPS ViT-g achieves within 0.009-1.4 RMSE/degrees of DINOv2-g on depth and normals (Table 2) while enabling multimodal retrieval that DINOv2 cannot perform.
The target domain involves web-style photographs with diverse objects and scenes. The 117M-image curated training set (filtered from WebLI for similarity to benchmark datasets like Pascal VOC, ADE20k, and ImageNet) biases TIPS toward this distribution.
Frozen features are required (no task-specific fine-tuning of the encoder). TIPS is designed and evaluated specifically for off-the-shelf use, with all experiments using frozen weights and lightweight probes.

Prefer DINOv2 (or other self-supervised specialist) over TIPS when:

The application is purely a single dense prediction task (e.g., monocular depth estimation for autonomous driving) with no language interface needed and every fraction of a dB or RMSE matters. DINOv2 maintains a narrow lead on NYUv2 depth (0.344 vs. 0.353) and normals (20.5° vs. 21.9°), and the gap may be larger on domain-specific data far from TIPS's web-image training distribution.
The inference hardware has no capacity for a text encoder (389M parameters for ViT-g, even if unused at inference time, the model checkpoint includes these parameters during training). DINOv2's image-only architecture is inherently lighter.
The target domain is substantially different from web imagery (medical, satellite, industrial) and the Mapillary SLS addition (for street-level imagery) does not cover it. TIPS's curation process relies on similarity to specific benchmark datasets (Table 13), and domains far from these benchmarks may see larger performance drops than DINOv2, which was curated with a different (and potentially broader) set of target datasets.

Prefer SigLIP or EVA-CLIP over TIPS when:

The application is purely zero-shot classification or retrieval on standard benchmarks, where SigLIP-SO maintains a lead (Table 3: 83.2% vs. 79.9% ImageNet zero-shot, 70.8 vs. 68.2 UnED R@1). TIPS's spatial improvements come at a measurable cost to pure object-centric discrimination, and the dual-[CLS] mechanism does not fully recover this gap.
Training data is already strongly filtered and captions are high-quality (e.g., manually annotated datasets with clean, descriptive captions). TIPS's synthetic caption augmentation is designed to compensate for noisy web captions; if captions are already clean and descriptive, the benefit is reduced, and the simpler SigLIP training recipe may be more efficient.

Prefer TIPS over task-specific adapted models (DenseCLIP, CLIPSelf, etc.) when:

The goal is a single general-purpose feature extractor usable across many tasks without per-task architecture modifications. TIPS's frozen features with simple probes achieve competitive performance across 16 datasets spanning dense, global, and multimodal tasks, while task-specific methods are optimized for one task family and often require per-task training of specialized heads or fine-tuning.