A Mixed Diet Makes DINO An Omnivorous Vision Encoder

ArXiv: 2602.24181

🎯 Pitch

DINOv2 representations for paired RGB and depth images of the same scene are barely more similar than two random images — they occupy entirely disjoint subspaces. By simply fine-tuning the last few layers with a dual loss that anchors to the frozen teacher, this method not only collapses those distances by over an order of magnitude but also, counterintuitively, boosts RGB-only ImageNet accuracy. The resulting encoder enables zero-shot cross-modal transfer, where an RGB-trained depth head can run directly on segmentation maps without retraining.

---

1. Executive Summary

This paper introduces the Omnivorous Vision Encoder, a parameter-efficient teacher-student framework that learns a modality-agnostic feature space by fine-tuning only the final blocks of a frozen DINOv2 backbone. The core mechanism is a dual objective—a symmetric cross-modal alignment loss (InfoNCE over paired RGB, depth, and segmentation views of the same scene) combined with an anchoring loss (cosine-distance distillation from the frozen teacher's output)—which prevents representational collapse while pulling disparate modalities into a shared embedding. The method achieves a 46.1% R@1 on inter-modal retrieval on ScanNet (vs. DINOv2's 4.6%), improves ImageNet linear-probe accuracy from 80.4% to 83.8%, and enables zero-shot cross-modal transfer—deploying an RGB-trained depth head on segmentation or NOCS inputs without retraining—establishing that post-hoc alignment of visual modalities can simultaneously improve cross-modal understanding and enrich single-modality semantics, even when the training data mixes modalities at an extreme imbalance.

2. Context and Motivation

The Core Problem: Vision Encoders Are Modal, Not Omnivorous

The fundamental gap this paper identifies is deceptively simple: state-of-the-art vision foundation models like DINOv2 produce feature representations that are surprisingly misaligned across different visual modalities of the same scene. If you take an RGB image and its corresponding depth map—two views of the identical physical environment—and pass them through DINOv2, the cosine similarity between their resulting feature embeddings is often statistically indistinguishable from the similarity between two completely unrelated images of different scenes (Section 1, Figure 1). Specifically, the paper observes that:

"cos(f(x_r), f(x_d)) ≈ cos(f(x_{r,1}), f(x_{r,2}))"

where $f$ is the pretrained encoder, $x_r$ is an RGB image, $x_d$ is its paired depth map, and $x_{r,1}$, $x_{r,2}$ are two unrelated RGB images.

This is not a minor calibration issue—it reveals that DINOv2 has learned modality-specific feature subspaces that do not meaningfully intersect. A depth map of a kitchen and an RGB photo of that same kitchen occupy fundamentally different regions of the representation space. The consequence is that knowledge learned from one modality (e.g., a depth prediction head trained on RGB images) cannot transfer to another modality (e.g., deploying that same head on segmentation maps) because the feature basis is different.

Why This Problem Matters

The significance of this gap extends across multiple dimensions:

Practical deployment fragility. Any system that might encounter multiple sensor modalities—a robot with RGB-D cameras, an autonomous vehicle with LiDAR and visible-spectrum inputs, an AR headset producing segmentation masks—currently needs to either maintain separate encoders per modality or accept that its RGB-trained task heads will catastrophically fail when the input modality changes. Table 5 in the paper demonstrates exactly this: a depth prediction head trained on RGB images from NYUv2 achieves an RMSE of 0.842 meters when tested on RGB inputs from PACE, but that same head degrades to RMSE = 1.536 meters (effectively random guessing) when fed segmentation maps through the frozen DINOv2 backbone. In any safety-critical application where modality inputs might shift (e.g., a robot switching from daylight RGB to thermal at night), this brittleness is unacceptable.

Wasted training signal. Depth maps, segmentation masks, surface normal maps, and other structural modalities contain rich geometric and semantic information about scenes. These modalities are routinely available as byproducts of simulation engines, depth sensors, or human annotation pipelines. Yet current unimodal encoders like DINOv2 treat each RGB image as an isolated training sample, ignoring the complementary structural views of the same scene that could serve as a powerful form of self-supervision. The paper's central insight is that this is analogous to the early days of NLP, where models were trained separately per language until multilingual alignment techniques unlocked cross-lingual transfer and improved representation quality for all languages—including high-resource ones (Section 1). The authors explicitly draw this parallel:

"We argue that vision models face a similar inflection point. By aligning abundant modalities (RGB) with structure-rich but scarcer signals (depth, segmentation), we can create a more robust, shared visual language."

Theoretical understanding of representation invariance. From a representation-learning perspective, the observed misalignment reveals that DINOv2's invariance is largely photometric (invariant to color jitter, blur, cropping) but not structural or geometric. Two physically identical scenes captured through different sensing mechanisms should, ideally, map to the same point in feature space—this is the invariance that human perception exhibits (the authors cite work on color constancy and cortical representations; Zeki, 1983; Land, 1977). The gap between DINOv2's behavior and this ideal highlights a fundamental limitation in how current self-supervised objectives (which rely on data augmentations applied to RGB images) construct invariance: they learn to ignore pixel-level photometric variations but never encounter the kind of structured, modality-level transformation that would teach them to recognize geometric content independent of its sensory expression.

Where Prior Approaches Fall Short

The paper situates its contribution against four established research threads, each of which approaches the cross-modal alignment problem but—the authors argue—leaves a specific, practically important regime underserved.

Unified co-training (Omnivore, ImageBind, Unified-IO). These methods train a single backbone from scratch (or from random initialization) on multiple modalities simultaneously, learning shared parameters that natively handle RGB, depth, video, and other inputs (Girdhar et al., 2022, 2023; Lu et al., 2023). The key limitation is that co-training requires full-model retraining, which discards the enormous investment in pre-trained unimodal encoders. DINOv2, for instance, was trained on a carefully curated dataset of 142 million images with a sophisticated self-supervised objective and high-resolution fine-tuning—replicating that quality from scratch for a multimodal encoder is computationally prohibitive. The authors' approach is fundamentally different: rather than retraining, they perform post-hoc alignment on top of a frozen, pre-existing foundation model. As they state in Section 2:

"Compared to unified co-training... our method targets a pragmatic regime: post-hoc alignment of heterogeneous modalities by learning a single lightweight projector g on top of a fixed foundational backbone $f^*$."

RGB-D pretraining schemes (CLIP2Point, CoMAE, Mask3D). These methods explicitly align RGB and depth representations, often through contrastive objectives (CLIP2Point uses image-depth contrastive pretraining to transfer CLIP knowledge to 3D; Huang et al., 2023) or masked autoencoding objectives that reconstruct depth from RGB features (Mask3D; Hou et al., 2023). CoMAE (Yang et al., 2023) combines both: contrastive alignment followed by masked autoencoding on paired RGB-D data from datasets like SUN RGB-D and NYUv2. The limitation of these approaches is their narrow scope: they focus on a specific pair of modalities (typically RGB and depth) and train on relatively small, domain-restricted datasets (indoor scenes in SUN RGB-D and NYUv2). They do not produce a general-purpose encoder that aligns arbitrary visual modalities (RGB, depth, segmentation, and—as the paper's zero-shot experiments show—even out-of-distribution modalities like NOCS maps). Furthermore, many of these methods are not parameter-efficient: they either modify the full backbone or train large adapter modules, making them heavy to deploy alongside multiple unimodal baselines.

Adapter and parameter-efficient alignment methods (ViT-Adapter, MA-AVT). These inject small trainable modules into frozen backbones to adapt them for new tasks or modalities. ViT-Adapter (Chen et al., 2023) adds task-specific priors for dense prediction while keeping the ViT largely frozen. MA-AVT (Mahmud et al., 2024) performs blockwise contrastive alignment across audio-visual tokens in a parameter-efficient manner. The key limitation is that these methods are designed for task adaptation, not for constructing a unified modality-agnostic embedding. They produce features that work well for a specific downstream task (e.g., segmentation) but do not enforce that the same scene, regardless of input modality, maps to the same point in embedding space. The Omnivorous encoder, by contrast, directly optimizes for this cross-modal alignment property via the symmetric InfoNCE loss.

Cross-modal knowledge distillation and source-free transfer (SOCKET, CMKD variants). These methods transfer supervision from a source modality (e.g., an RGB teacher) to a target modality (e.g., a depth student) without requiring labeled data in the target domain. SOCKET (Ahmed et al., 2022) performs source-free cross-modal transfer by matching batch normalization statistics between modalities. Newer variants (Ferrod et al., 2025) incorporate disentanglement and contrastive terms for RGB-D semantic segmentation. The shortcoming is that these methods are downstream-task-specific: they train a new student model for a particular task (e.g., semantic segmentation) in a particular target modality. They do not produce a single encoder that can serve as a drop-in replacement for DINOv2 across arbitrary tasks and modalities. The Omnivorous encoder's goal is precisely this: one shared encoder (the fine-tuned final blocks) that emits modality-agnostic embeddings usable for classification, depth prediction, segmentation, 3D correspondence, and retrieval without task-specific architectural modifications.

The Trivial Solution Problem: Why Alignment Requires Careful Regularization

A crucial subtlety that the paper identifies—and that shapes its entire loss design—is that aligning cross-modal representations is not just about pulling features together. If you naively maximize cosine similarity between paired RGB, depth, and segmentation embeddings (e.g., with a simple contrastive loss), the network can learn a trivial, degenerate solution: collapse all features to a constant vector, which trivially satisfies the alignment objective but destroys all semantic information. This is the classic representational collapse problem in contrastive learning, and it is especially acute when aligning modalities because:

1. Extreme data imbalance. RGB images are vastly more abundant than depth or segmentation maps. Contrastive methods like CMC (Contrastive Multiview Coding; Tian et al., 2020) prevent collapse by pushing apart representations from different scenes, but this requires large sets of diverse "negative" examples. When one modality is rare (e.g., depth maps are only available for a fraction of scenes), the pool of negatives is small, making it difficult for the model to learn robust discrimination. The paper's modality mixup strategy (§3.2, §6.1.4) is specifically designed to address this: by blending modalities (e.g., producing images that are 50% depth and 50% RGB), the training procedure creates a continuous spectrum of inputs, effectively increasing the density of negative examples spanned by the contrastive loss.

2. Shortcut learning from colormaps. A depth map is typically visualized as a grayscale image; a segmentation map is visualized with a fixed colormap (e.g., jet). If the encoder sees raw grayscale depth maps and RGB photos, it can learn to distinguish modalities based on trivial low-level statistics—the depth map has only one channel (replicated to three) and a narrow intensity distribution, while the RGB photo has full color variation. The network can then "align" features by learning a modality-specific projection that collapses the depth-specific channel statistics to match the RGB statistics, without ever learning about shared geometric content. The paper's natural colorization strategy (§6.1.2, Algorithm 1) eliminates this shortcut by re-rendering depth and segmentation maps using a color palette derived from the paired RGB image. A depth map of a kitchen is colored with the kitchen's actual colors—the blue of the walls, the brown of the cabinets—forcing the encoder to attend to structural content (edges, shapes, spatial layout) rather than low-level color distributions to solve the alignment task. This creates what the authors call "hard positives" for the contrastive objective.

3. Catastrophic forgetting of pretrained semantics. Even if collapse is prevented, pulling modalities together in feature space can cause the encoder's representations to "drift" away from the rich semantic structure learned during DINOv2 pretraining. The anchoring loss ($\mathcal{L}_{\text{anchor}}$, Equation 3) directly addresses this by penalizing cosine distance between the student's output $h_m$ and the frozen teacher's output $h_m^*$ for the same input. This acts as a distillation term that preserves the original feature space's discriminative power. The $\lambda_{\text{anchor}}$ hyperparameter becomes the central control knob (Figure 4a): at $\lambda_{\text{anchor}} = 0$, alignment is excellent but discriminability collapses; at $\lambda_{\text{anchor}} = 100$, discriminability is preserved but alignment is weak; the default $\lambda_{\text{anchor}} = 10$ sits at a carefully chosen operating point on this Pareto frontier.

How This Paper Positions Itself

The authors explicitly frame their contribution as a pragmatic compromise that targets a specific, underexplored regime in the design space of cross-modal vision encoders. This positioning is articulated through several design choices:

Post-hoc rather than from-scratch. Unlike unified co-training approaches (Omnivore, ImageBind), the method does not require retraining a foundation model. This is an engineering decision with practical implications: it allows practitioners to leverage the enormous existing investment in DINOv2 (or potentially other unimodal encoders) while adding cross-modal alignment as a lightweight fine-tuning step. The entire training process uses a ViT-B/14 model (173M parameters), fine-tunes only 4 out of 12 blocks, runs on a $4 \times 4 \times 4$ TPU v4 configuration for just 20,000 steps (1 hour 14 minutes total; Table 7), and uses a global batch size of 512. This is orders of magnitude cheaper than pretraining a multimodal encoder from scratch.

Parameter-efficient rather than full-model fine-tuning. The method freezes the first 8 blocks of the ViT backbone (the "frozen backbone" $f^*$) and only updates the final 4 blocks (the "adapter" $g$). Table 12 ablates this choice, showing that freezing 8 blocks provides the best trade-off across depth estimation and segmentation tasks compared to freezing 4, 6, or 10 blocks. The frozen backbone preserves the low-level feature extractors that DINOv2 learned during pretraining, while the fine-tuned upper blocks learn to reorganize these features into a modality-agnostic space. This is conceptualized as a teacher-student distillation framework where the teacher $f_T = g^* \circ f^*$ (fully frozen) provides stability and the student $f_S = g \circ f^*$ (frozen backbone, trainable head) learns alignment.

Single shared adapter rather than modality-specific modules. A crucial architectural choice: the adapter $g$ is shared across all modalities. There are no separate branches for RGB, depth, and segmentation. The same learned function maps all three input types to the shared embedding space. This is what makes the encoder truly "omnivorous"—it is modality-agnostic, not just modality-invariant. Modality-invariant representations could be achieved by training separate encoders per modality that are constrained to produce similar outputs for the same scene; the Omnivorous encoder's stronger claim is that a single set of parameters handles all inputs, which enables the zero-shot cross-modal transfer demonstrated in Section 4.3.

Data-centric contributions integrated with the training objective. The paper couples the architectural and loss design with two data processing innovations—natural colorization and modality mixup—that are not merely preprocessing steps but are integral to making the alignment problem solvable. Natural colorization eliminates the colormap shortcut; modality mixup constructs a continuous modality spectrum that provides dense negative examples for contrastive learning and teaches the encoder invariance to the texture-to-structure blending ratio. The ablation in Table 6 shows that increasing the mixup blending parameter $\alpha_{\text{max}}$ from 0 (no mixup) to 1.0 (full-spectrum blending) monotonically improves classification, segmentation, and 3D correspondence performance, with the only regression being a slight decrease in depth prediction $\delta_1$ (from 0.899 at $\alpha_{\text{max}} = 0$ to 0.891 at $\alpha_{\text{max}} = 1.0$). The default $\alpha_{\text{max}} = 0.5$ is chosen to balance across all tasks.

Evaluation designed to test both alignment and retention. The paper's experimental protocol directly measures the two competing desiderata: cross-modal alignment (retrieval, zero-shot transfer) and semantic discriminability (classification, depth prediction, segmentation). This dual evaluation is necessary because the trivial solution (collapse) would excel at alignment metrics but fail catastrophically on task performance. The results in Tables 1–5 demonstrate that the Omnivorous encoder achieves the desired outcome: dramatically better alignment ($4.6\% \rightarrow 46.1\%$ R@1 on ScanNet retrieval) while maintaining or improving downstream task performance ($80.4\% \rightarrow 83.8\%$ ImageNet linear-probe accuracy). This is the empirical validation that the anchoring loss successfully navigates the alignment-discriminability tradeoff identified in Figure 4a.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper proposes a training procedure, not a new architecture. The system is a Vision Transformer (ViT) that has been fine-tuned in a specific, carefully constrained way so that a single shared encoder maps RGB images, depth maps, and segmentation masks of the same physical scene to nearly identical points in a feature space — something the original DINOv2 ViT catastrophically fails to do. The problem it solves is that off-the-shelf foundation models like DINOv2 produce modality-specific embeddings (the feature for a depth map of a kitchen is as different from the feature for an RGB photo of that kitchen as it is from a feature of an unrelated bedroom photo), which makes it impossible to transfer task-specific heads across modalities. The "shape" of the solution is a parameter-efficient teacher-student framework where a frozen backbone (the first 8 of 12 ViT blocks from DINOv2) is shared with a frozen teacher, a trainable head (the final 4 blocks) is updated to pull multimodal embeddings together via a contrastive loss, and an anchoring loss (cosine-distance distillation from the frozen teacher's output) prevents the student from drifting into a degenerate collapsed representation that would lose all semantic information.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that interact in a teacher-student framework:

1. The frozen backbone $f^*$: The first 8 blocks of a pretrained DINOv2 ViT-B/14 (173M parameters total). This component never receives gradient updates during Omnivorous training. It processes raw pixel inputs of any modality (RGB, colorized depth, or colorized segmentation) and produces intermediate feature tokens. These are the shared, stable representations that both the teacher and student inherit.

2. The frozen teacher head $g^*$: The final 4 blocks of the original pretrained DINOv2 ViT-B/14, kept frozen. When combined with $f^*$, it forms the full teacher model $f_T = g^* \circ f^*$ that produces "anchor" embeddings $h^*_m$ for each modality $m$. These embeddings preserve DINOv2's original semantic structure (excellent for classification and dense prediction) but are modality-misaligned (the color distributions in Figure 5 show disjoint subspaces for RGB, depth, and segmentation).

3. The trainable student head $g$: The final 4 blocks of the ViT, initialized from the pretrained DINOv2 weights but updated during Omnivorous training. When combined with $f^*$, it forms the student model $f_S = g \circ f^*$ that produces "adapted" embeddings $h_m$. This is the only component that receives gradient updates. The same $g$ processes features from all modalities — there are no modality-specific branches or parameters. This is what makes the encoder "omnivorous" (modality-agnostic) rather than merely "modality-invariant."

4. The training data pipeline: Raw images (RGB, depth maps, segmentation masks) from six datasets undergo three sequential transformations: (1) photometric augmentation of the RGB image (brightness, contrast, hue, saturation jitter), (2) natural colorization of depth and segmentation maps using a color palette derived from the paired RGB image (Algorithm 1), and (3) modality mixup that blends the colorized structural maps with the RGB image at a randomly sampled mixing ratio $\alpha \in [0, 0.5]$. The pipeline produces three augmented views per scene: an augmented RGB image $x^{\text{aug}}_r$, a mixed-up depth image $x^{\text{mixup}}_d$, and a mixed-up segmentation image $x^{\text{mixup}}_s$.

5. The dual-objective loss function: Two losses are computed and summed. The symmetric cross-modal alignment loss ($\mathcal{L}_{\text{align}}$, Equations 1–2) is an InfoNCE contrastive loss computed over all three modality pairs (RGB-Depth, Depth-Seg, Seg-RGB) that pulls student embeddings of the same scene together and pushes embeddings of different scenes apart. The anchoring loss ($\mathcal{L}_{\text{anchor}}$, Equation 3) is a cosine-distance penalty between the student's output $h_m$ and the teacher's output $h^*_m$ for each modality, weighted by $\lambda_{\text{anchor}} = 10$.

Information flow during training: A scene is sampled from one of six datasets → the paired (RGB, depth, segmentation) images undergo data augmentation and colorization → the three augmented views are passed through $f^*$ to produce three sets of intermediate tokens → both $g^*$ (frozen) and $g$ (trainable) process these tokens to produce teacher embeddings ($h^*_r, h^*_d, h^*_s$) and student embeddings ($h_r, h_d, h_s$) → $\mathcal{L}_{\text{align}}$ is computed between student embeddings across all modality pairs → $\mathcal{L}_{\text{anchor}}$ is computed between student and teacher embeddings within each modality → gradients from $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{align}} + 10 \cdot \mathcal{L}_{\text{anchor}}$ flow backward through $g$ and stop at the $f^*$ boundary → the 20,000-step training loop updates only the final 4 ViT blocks.

3.3 Roadmap for the Deep Dive

• First, the data pipeline (§3.4, Data Pipeline and Modality Representation), because the specific way depth and segmentation maps are rendered into 3-channel images fundamentally determines whether the alignment task is solvable or trivially shortcutted. I will explain photometric augmentation, natural colorization (Algorithm 1), modality mixup, and normalization in order of application, with the rationale for each design choice.

• Second, the architecture and parameter-freezing strategy (§3.4, Architecture and Parameter-Efficient Teacher-Student Design), covering the ViT-B/14 block structure, the freeze/fine-tune split (L=8 frozen, 4 fine-tuned), the teacher-student sharing of $f^*$, and the ablation evidence for why 8 frozen blocks is optimal.

• Third, the loss functions (§3.4, Loss Functions), because they are the mathematical core of the method. I will walk through the symmetric InfoNCE alignment loss (Equations 1–2), the anchoring distillation loss (Equation 3), and the total weighted objective (Equation 4), explaining why each term exists, what failure mode it prevents, and how the $\lambda_{\text{anchor}}$ hyperparameter controls the alignment-discriminability tradeoff.

• Fourth, the training configuration and implementation details (§3.4, Training Configuration), providing the exact optimizer, learning rate, batch size, number of steps, hardware, and the dense-token subsampling strategy used to make the contrastive loss tractable.

• Fifth, a synthesis of design choices (§3.4, Design Rationale Summary), connecting the architectural, data, and loss decisions to the core challenge identified in the motivation section — preventing representational collapse while aligning severely imbalanced modalities.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that cross-modal alignment of visual modalities can be achieved post-hoc on a frozen foundation model through a carefully designed combination of data preprocessing, a symmetric contrastive loss, and a distillation-based anchoring loss, all applied in a parameter-efficient manner that updates only the final blocks of a ViT.

---

Data Pipeline and Modality Representation

The data pipeline transforms raw multimodal inputs (RGB images, depth maps as scalar arrays, and segmentation maps as integer label arrays) into three-channel image tensors that can be ingested by a ViT pretrained on RGB data. The pipeline is not a preprocessing afterthought — it is integral to the method because the specific rendering of non-RGB modalities determines whether the alignment problem is solvable or whether the model can exploit trivial shortcut signals. The pipeline has four sequential stages applied per training example.

Stage 1: Photometric augmentation (RGB only). The RGB image undergoes a sequence of standard color-space distortions designed to encourage invariance to lighting and color variations. The paper specifies exact sampling ranges: brightness is adjusted by adding a delta sampled uniformly from $[-0.1, 0.1]$, saturation is scaled by a factor drawn from $[0.8, 1.2]$, hue is shifted by a delta in $[-0.03, 0.03]$, and contrast is scaled by a factor from $[0.8, 1.2]$ (Appendix §6.1.1). Each distortion parameter is sampled independently. The output is the augmented RGB image $x^{\text{aug}}_r$. This stage ensures the encoder learns invariance to photometric variations (a standard self-supervised learning objective) and generates diverse color statistics for the subsequent natural colorization step.

Why this stage matters beyond standard augmentation: The photometrically augmented RGB image $x^{\text{aug}}_r$ serves dual duty — it is both a training input in its own right (fed directly to the encoder) and the source of the color palette used to re-render the depth and segmentation maps. If the RGB image were not augmented, the depth and segmentation maps would always be colorized with the same color distribution, reducing the diversity of the contrastive task and potentially allowing the model to memorize scene-specific color mappings rather than learning structural alignment.

Stage 2: Natural colorization (depth and segmentation maps). This is the paper's key data innovation. The core problem it solves: if you feed a depth map as a grayscale image (one intensity channel replicated to three) into a ViT pretrained on natural images, the encoder can immediately recognize it as "not RGB" based on low-level channel statistics — the correlation between the three channels is perfect (they are identical), the intensity distribution is different from natural images, and there are no chromatic variations. The contrastive learning loss would then align depth and RGB features not by learning shared scene geometry but by learning to collapse modality-specific color statistics. This is the "shortcut learning" problem identified in the motivation.

The natural colorization algorithm (Algorithm 1) eliminates this shortcut by re-rendering depth and segmentation maps using a scene-specific color palette extracted from the paired RGB image. Formally, let $x^{\text{raw}}_m$ be the raw scalar map for modality $m \in \{\text{Depth}, \text{Segmentation}\}$, represented as a 2D array of size $H \times W$. The algorithm proceeds in five substeps:

Substep 1: Normalization and discretization. The raw scalar values are normalized to the range $[0, 1]$:

$x^{\text{norm}}_m = \frac{x^{\text{raw}}_m - \min(x^{\text{raw}}_m)}{\max(x^{\text{raw}}_m) - \min(x^{\text{raw}}_m) + \epsilon}$

where $\epsilon = 10^{-6}$ prevents division by zero. Each normalized pixel value is then mapped to one of $B = 64$ discrete bins by computing $b_{u,v} = \text{clip}(\lfloor x^{\text{norm}}_m[u,v] \cdot B \rfloor, 0, B-1)$. This discretization groups pixels with similar depth or segmentation class index into the same bin.

What the binning does: For a depth map, the bins partition the depth range $[\min, \max]$ into 64 equal intervals; pixels at similar depths map to the same bin. For a segmentation map, the bins partition the arranged class indices into 64 groups (the paper does not specify the mapping from segmentation class labels to these 64 bins, but since segmentation maps are integer-valued, the binning effectively groups spatially adjacent class labels). The key property is that two pixels with the same bin index have similar structural/scalar values in the original modality.

Substep 2: Palette accumulation. For each bin $k \in \{0, \ldots, 63\}$, we accumulate the RGB colors from the augmented RGB image $x^{\text{aug}}_r$ that correspond to pixels in that bin:

$\mathbf{S}_k = \sum_{u,v} \mathbf{1}[b_{u,v} = k] \cdot x^{\text{aug}}_r(u,v), \quad \mathbf{N}_k = \sum_{u,v} \mathbf{1}[b_{u,v} = k]$

where $\mathbf{S}_k \in \mathbb{R}^3$ is the sum of RGB vectors for all pixels in bin $k$, and $\mathbf{N}_k \in \mathbb{R}$ is the count of pixels in that bin. The indicator $\mathbf{1}[b_{u,v} = k]$ is 1 if the pixel at $(u,v)$ maps to bin $k$, 0 otherwise.

What this does: For each depth interval, we collect the actual colors that appear at those depths in the RGB image. For example, if a range of depth values corresponds to a wall, all pixels in that depth range will have wall-like colors (beige, white, etc.). The palette entry for that depth bin will be the average wall color.

Substep 3: Palette smoothing. To handle bins that have very few pixels (creating noisy palette entries), a 1D convolution with a uniform kernel of size $K=5$ is applied across bins:

$\tilde{\mathbf{S}} = \text{Convolve1D}(\mathbf{S}, w), \quad \tilde{\mathbf{N}} = \text{Convolve1D}(\mathbf{N}, w)$

where $w_i = 1$ for $i = 1, \ldots, 5$. This smooths the accumulated color sums and counts across adjacent depth bins, filling in gaps where certain depth values are unrepresented in the scene.

Substep 4: Palette normalization. The average color for each bin is computed:

$\mathcal{P}[k] = \frac{\tilde{\mathbf{S}}[k]}{\tilde{\mathbf{N}}[k] + \epsilon}$

where $\mathcal{P} \in \mathbb{R}^{64 \times 3}$ is the scene-specific natural color palette. Each row is the average RGB color associated with a particular depth bin (or segmentation bin) in this scene.

Substep 5: Image re-rendering. The colorized map $x_m$ is generated by replacing each pixel's original scalar value with its bin's palette color:

$x_m(u,v) = \mathcal{P}[b_{u,v}]$

The result is a three-channel image where the spatial structure of the depth or segmentation map is preserved, but the colors are drawn from the RGB image's actual color distribution. A depth map of a kitchen now looks like a slightly smoothed version of the kitchen photo, with colors that match the actual scene.

Why this solves the shortcut problem: The colorized depth map $x_d$ and the augmented RGB image $x^{\text{aug}}_r$ now share the same color histogram (because the palette is derived from $x^{\text{aug}}_r$). If the encoder tried to distinguish them based on low-level color statistics, it would fail — the statistics are identical. The encoder is forced to attend to structural differences: the depth map lacks texture (it has smooth color regions corresponding to constant depth), while the RGB image has fine-grained texture. The only reliable way to align the two is to recognize that they depict the same spatial layout — edges, object boundaries, surface orientations — despite differing in textural detail. This creates what the paper calls "hard positives" for the contrastive objective.

Stage 3: Modality mixup (depth and segmentation only). Even with natural colorization, there remains a domain gap: the colorized depth map is a textured version of the scene where colors are constant within depth bins, while the RGB image has full photometric variation. To bridge this gap and create a continuous spectrum of modalities, the paper introduces modality mixup:

$x^{\text{mixup}}_d = (1 - \alpha_d) \cdot x_d + \alpha_d \cdot x^{\text{aug}}_r$

$x^{\text{mixup}}_s = (1 - \alpha_s) \cdot x_s + \alpha_s \cdot x^{\text{aug}}_r$

where $\alpha_d$ and $\alpha_s$ are independently sampled uniformly from $[0, \alpha_{\text{max}}]$ with $\alpha_{\text{max}} = 0.5$ (the default). The mixing is a pixel-wise convex combination: each pixel in the mixed image is a weighted average of the corresponding pixel in the colorized structural map and the augmented RGB image.

What this does: When $\alpha = 0$, the mixed image is the pure colorized depth/segmentation map (geometry with scene-derived colors but no texture). When $\alpha = 0.5$, the image is a 50-50 blend — the structural map's smooth regions are partially overlaid with RGB texture. The effect is to create a continuous interpolation between "pure geometry" and "pure photometry" in image space. The paper conceptualizes this as spanning a modality spectrum: Depth $\leftrightarrow$ RGB $\leftrightarrow$ Segmentation, with the RGB image sitting in the middle (as a blend point that could be reached by mixing from either side).

Why this is important: First, it provides dense negatives for contrastive learning. Without mixup, each scene contributes exactly three distinct modality-instantiations (RGB, depth, segmentation). With mixup, the continuous $\alpha$ sampling means the model sees an effectively infinite set of intermediate modalities, making it harder to memorize scene-specific modality mappings and encouraging learning of the underlying geometric invariance. Second, it teaches the encoder that the ratio of texture to structure is not a meaningful feature for scene identity — a scene should map to the same embedding whether it is presented as a pure depth map, a pure RGB photo, or a blend. Third, it handles imperfect data: real-world depth maps from sensors are noisy and contain artifacts that create unnatural color boundaries; mixup trains the model to be robust to imprecise depth values by exposing it to deliberately blurred versions of the modality boundary.

The choice of $\alpha_{\text{max}} = 0.5$: The paper restricts $\alpha$ to $[0, 0.5]$ rather than $[0, 1.0]$ during training "to prevent depth and segmentation images from looking too similar to the RGB image" (Section 3.2). If $\alpha$ were allowed to reach 1.0, the mixed depth image would be identical to the RGB image, making the contrastive task between "depth" and "RGB" trivially easy (they would be the same input) and defeating the purpose of learning cross-modal alignment. The 0.5 upper bound ensures the depth/segmentation signal remains dominant while introducing enough texture to span the modality gap. Table 6 shows that increasing $\alpha_{\text{max}}$ from 0 to 1.0 actually improves classification (83.1% → 83.5%), segmentation (62.4% → 63.2% mIoU), and 3D correspondence (28.40% → 29.03%), but degrades depth prediction (89.9% → 89.1% $\delta_1$). The default 0.5 balances these competing trends.

Stage 4: Normalization. All three-channel images (the augmented RGB $x^{\text{aug}}_r$, the mixed depth $x^{\text{mixup}}_d$, and the mixed segmentation $x^{\text{mixup}}_s$) are normalized using ImageNet-1k channel-wise mean and standard deviation: $\mu = (0.485, 0.456, 0.406)$, $\sigma = (0.229, 0.224, 0.225)$. This is the standard preprocessing for models pretrained on ImageNet and ensures the input statistics match what $f^*$ expects.

Dataset composition. Training data is drawn from six datasets spanning both real and synthetic domains: ScanNet (real indoor RGB-D scans), TartanAir (synthetic outdoor scenes with ground-truth depth), Hypersim (photorealistic synthetic indoor scenes), MOVi (synthetic object-centric videos from Kubric), PointOdyssey (synthetic long-term point tracking), and DynamicReplica (synthetic dynamic scenes). All images are preprocessed to a resolution of $224 \times 224$ pixels. RGB images are resized using bilinear interpolation; depth and segmentation maps use nearest-neighbor interpolation to preserve discrete structural boundaries, followed by a center crop to square aspect ratio (Table 7).

Why six datasets: The diversity of scenes (indoor, outdoor, object-centric, dynamic) prevents the encoder from learning dataset-specific alignment shortcuts. MOVi, for instance, contains simple geometric shapes on plain backgrounds, while ScanNet contains cluttered realistic rooms. By mixing these during training, the model learns to align modalities based on shared scene geometry that generalizes across domains.

---

Architecture and Parameter-Efficient Teacher-Student Design

The architecture is a teacher-student framework built on a single Vision Transformer (ViT-B/14, the "Base" variant with 12 transformer blocks and a patch size of 14 pixels, totaling 173 million parameters). The critical design choice is that the teacher and student share the vast majority of their parameters — only the final blocks differ — which provides a natural stabilization mechanism during training.

The frozen backbone $f^*$. The ViT is conceptually split into two components. The "backbone" $f^*$ consists of the patch embedding layer (which converts $224 \times 224$ input images into a $16 \times 16$ grid of patch tokens, since $224/14 = 16$) plus the first $L = 8$ transformer blocks. This component is fully frozen throughout Omnivorous training — its parameters never receive gradient updates. The rationale is that the early layers of a ViT primarily compute low-level features (edges, textures, color blobs) that are modality-general: whether the input is an RGB photo or a colorized depth map, the same edge-detection and texture-analysis operations apply. Freezing these layers preserves the pretrained quality of these features while preventing them from being destroyed by the alignment objective.

The trainable head $g$ (the "adapter"). The remaining $12 - L = 4$ transformer blocks (blocks 8 through 11, 0-indexed) constitute the trainable "head" $g$. These later layers are responsible for higher-level semantic processing — object parts, semantic categories, spatial relationships. The paper's hypothesis is that modality-specific information is primarily encoded in these upper layers (because DINOv2 was trained only on RGB, the high-level semantic features it learned are RGB-specific), and therefore these are the layers that need to be modified to achieve cross-modal alignment.

The teacher model $f_T$. The teacher is the union of the frozen backbone $f^*$ and the original, frozen last 4 blocks, denoted $g^*$. The teacher model is $f_T = g^* \circ f^*$ — the full pretrained DINOv2 ViT-B/14 with all parameters frozen. For any input $x_m$ of modality $m$, the teacher produces:

$h^*_m = f_T(x_m) = g^*(f^*(x_m))$

This is the pretrained representation that DINOv2 would produce. It has excellent semantic structure (good for classification and dense prediction) but poor cross-modal alignment (as evidenced by Figure 1 and Table 1). The teacher's outputs serve as targets for the anchoring loss — they define the feature space that the student should not drift too far from.

The student model $f_S$. The student shares the frozen backbone $f^*$ with the teacher but has a trainable head $g$ (initialized from the pretrained $g^*$ weights). The student model is $f_S = g \circ f^*$. For the same input $x_m$, the student produces:

$h_m = f_S(x_m) = g(f^*(x_m))$

Both $h^*_m$ and $h_m$ are $L_2$-normalized to unit length before any loss computation. The student's outputs are used for both the alignment loss (where they are pulled together across modalities) and the anchoring loss (where they are pulled toward the teacher's output for the same modality).

What "parameter-efficient" means in this context: Only 4 out of 12 transformer blocks are updated. In terms of parameter count, this is approximately one-third of the ViT's parameters (since transformer blocks are roughly equal-sized, and the patch embedding layer adds a small additional component). The frozen backbone $f^*$ is shared between teacher and student, meaning the total parameter overhead is only the student's trainable head $g$ — there is no separate teacher network to store. At inference time, only the student model $f_S = g \circ f^*$ is used; the teacher is discarded.

Ablation evidence for the freeze/fine-tune split. Table 12 presents results for freezing $L_{\text{stop-gradient}} = 4, 6, 8, 10$ blocks (out of 12). The configuration with $L = 8$ frozen blocks (the default) achieves the best overall performance across depth estimation (DPT head on NYUv2: $\delta_1 = 0.948$; linear head on NAVI: $\delta_1 = 0.706$) and semantic segmentation (DPT head on ADE20k: mIoU = $0.505$; linear head on Cityscapes: mIoU = $0.632$). Freezing fewer blocks ($L = 4$ or $6$) performs slightly worse, likely because too many trainable parameters allow the alignment loss to overwrite pretrained features. Freezing more blocks ($L = 10$) also performs worse, likely because there are insufficient trainable layers to learn the modality alignment. The choice of $L = 8$ represents the sweet spot where the student has enough capacity to reorganize features for alignment while retaining the pretrained low-level feature extractors.

Alternative parameterization: adapter on top vs. fine-tuning final blocks. Table 11 compares the default approach (fine-tuning the final 4 blocks of the ViT) against an alternative where the student is the fully frozen DINOv2 backbone plus an additional 4-block ViT "adapter" module trained from scratch on top. This alternative creates a deeper student (16 blocks total vs. 12) with a clear separation between frozen pretrained layers and newly learned layers. The results are comparable: adapter-on-top achieves 84.0% vs. 83.8% ImageNet linear-probe accuracy (a small advantage), but slightly worse depth prediction (NYUv2 $\delta_1 = 0.905$ vs. $0.896$) and 3D correspondence (28.15% vs. 29.00% PCK). This indicates the method is robust to the specific parameterization and the distillation-based objective works regardless of whether the student's trainable layers replace or augment the teacher's upper blocks.

Why a shared backbone matters modality-agnostically: The same $g$ processes features from all three modalities. There is no modality-specific routing, no separate depth-branch or segmentation-branch. The architecture is truly "omnivorous" — the function $f_S$ is applied identically to RGB, depth, and segmentation inputs. This is what enables the zero-shot cross-modal transfer demonstrated in Section 4.3: a depth prediction head trained on RGB features from $f_S$ can be applied directly to segmentation features from $f_S$ without any adaptation, because $f_S$ maps both inputs to the same embedding space.

TIPS as an alternative teacher. The paper also describes experiments replacing DINOv2 with TIPS (Text-Image Pretraining with Spatial Awareness; Maninis et al., 2025) as the teacher backbone. TIPS uses the same ViT-B/14 architecture but differs in two ways: its position encoding is $16 \times 16$ (vs. DINOv2's $37 \times 37$), and it uses two CLS tokens instead of one. The Omnivorous training procedure is applied identically, freezing the first 8 blocks and fine-tuning the last 4. Figure 9 shows qualitative behavior matching that observed with DINOv2: the alignment-discriminability Pareto frontier exists, and $\lambda_{\text{anchor}} = 100$ exceeds performance at higher values strongly anchored to the pretrained teacher. This demonstrates that the method generalizes across different pretrained foundation models, as long as they share the ViT architecture.

---

Loss Functions

The total training objective is a weighted sum of two losses, each designed to prevent a specific failure mode:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{align}} + \lambda_{\text{anchor}} \mathcal{L}_{\text{anchor}}$

where $\lambda_{\text{anchor}} = 10$ is the default hyperparameter. I will explain each loss in detail.

---

Symmetric Cross-Modal Alignment Loss ($\mathcal{L}_{\text{align}}$)

This loss is the primary mechanism that pulls embeddings of the same scene from different modalities together in feature space while pushing embeddings of different scenes apart. It is a symmetric, multi-modal extension of the InfoNCE (Information Noise-Contrastive Estimation) loss.

InfoNCE for a single modality pair. For a specific pair of modalities $(m_1, m_2)$ and a batch of $N$ scenes, the InfoNCE loss treating $m_1$ as the query and $m_2$ as the target is:

$\mathcal{L}_{\text{InfoNCE}}(m_1, m_2) = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp_{\tau}(\text{sim}(h_{m_1}^{(i)}, h_{m_2}^{(i)}))}{\sum_{j=1}^{N} \exp_{\tau}(\text{sim}(h_{m_1}^{(i)}, h_{m_2}^{(j)}))}$

where $h_{m_1}^{(i)}$ is the $L_2$-normalized student embedding of scene $i$ in modality $m_1$, $h_{m_2}^{(j)}$ is the $L_2$-normalized student embedding of scene $j$ in modality $m_2$, $\text{sim}(\cdot, \cdot)$ is the cosine similarity (equivalent to dot product since both vectors are unit-normed), $\exp_{\tau}(x) = \exp(x / \tau)$, and $\tau$ is a learned temperature parameter clipped to the range $[0, 100]$.

What it computes: For each scene $i$, the loss computes the cosine similarity between its $m_1$ embedding and all $N$ possible $m_2$ embeddings in the batch. The numerator is the exponentiated similarity of the correct pair (scene $i$ in $m_1$ matched with scene $i$ in $m_2$). The denominator is the sum of exponentiated similarities between scene $i$ in $m_1$ and all scenes $j = 1, \ldots, N$ in $m_2$ (including the correct one). The loss encourages the numerator to dominate the denominator, which happens when the correct pair has much higher similarity than any incorrect pair. The log and negative sign make this a minimization problem: lower loss means better discrimination. Averaged over all $N$ scenes in the batch.

Why this form: InfoNCE is the standard contrastive loss used in self-supervised learning (SimCLR, MoCo, CLIP). It can be derived as a lower bound on mutual information between the two views, making it a principled objective for learning representations that capture shared information. The temperature $\tau$ controls the concentration of the similarity distribution: low $\tau$ (sharp distribution) heavily penalizes any small confusion between the correct pair and hard negatives; high $\tau$ (flat distribution) is more lenient. Learning $\tau$ rather than fixing it (e.g., to the common value of 0.07 used in SimCLR) allows the model to adapt the sharpness to the difficulty of the modality pair.

Symmetric extension to three modalities. The total alignment loss is the average of InfoNCE losses over all unordered pairs of the three modalities (RGB, Depth, Segmentation):

$\mathcal{L}_{\text{align}} = \frac{1}{3} \sum_{k_1=1}^{3} \sum_{k_2 > k_1}^{3} \mathcal{L}_{\text{InfoNCE}}(m_{k_1}, m_{k_2})$

With modalities ordered as (RGB, Depth, Segmentation) $\equiv$ (R, D, S), this expands to:

$\mathcal{L}_{\text{align}} = \frac{1}{3} \left( \mathcal{L}_{\text{InfoNCE}}(R, D) + \mathcal{L}_{\text{InfoNCE}}(D, S) + \mathcal{L}_{\text{InfoNCE}}(S, R) \right)$

Note that $\mathcal{L}_{\text{InfoNCE}}(R, D)$ and $\mathcal{L}_{\text{InfoNCE}}(D, R)$ are not both computed — the sum over $k_2 > k_1$ picks each pair once. However, within each $\mathcal{L}_{\text{InfoNCE}}(m_1, m_2)$, the loss is asymmetric in its treatment: $m_1$ serves as the query and is compared against all $m_2$ targets. The paper describes this as "symmetric" because all three possible directional evaluations (RGB-as-query, Depth-as-query, Seg-as-query) are covered across the three terms, just not within each individual term.

What this symmetry achieves: By computing alignment across all three modality pairs, the loss creates a consistent embedding space where no single modality is privileged. If the loss only aligned RGB-to-Depth and RGB-to-Seg, depth and segmentation maps could still be misaligned with each other (they would both align to RGB but not directly). The triangular construction ensures transitivity: if Depth aligns with RGB and RGB aligns with Segmentation, then by the triangle inequality of the embedding space, Depth must approximately align with Segmentation. Computing the Depth-Seg loss explicitly reinforces this.

Inputs to the alignment loss. The features used are the student's $L_2$-normalized embeddings of the augmented and mixed-up images:

• $h_R$: embedding of the photometrically augmented RGB image $x^{\text{aug}}_r$
• $h_D$: embedding of the mixed-up depth image $x^{\text{mixup}}_d = (1 - \alpha_d) x_d + \alpha_d x^{\text{aug}}_r$
• $h_S$: embedding of the mixed-up segmentation image $x^{\text{mixup}}_s = (1 - \alpha_s) x_s + \alpha_s x^{\text{aug}}_r$

Dense token computation. The loss is computed separately for two types of features: the class token (CLS token, the standard ViT output used for image-level representations) and dense tokens (the $16 \times 16 = 256$ patch-level feature vectors). For the dense-token InfoNCE loss, 64 tokens are randomly subsampled from each image's $16 \times 16$ grid before computing the loss. This subsampling makes the computation tractable (using all 256 tokens per image in a batch of 512 scenes would require enormous similarity matrices) while still providing spatial supervision.

Preventing intra-image negatives. A crucial subtlety: when computing InfoNCE on dense tokens, two tokens from the same image could be treated as negatives (since they come from different scenes in the naive formulation if the scene index is attached to the full image rather than to individual tokens). The paper explicitly masks these out: "We use a mask to ensure we do not use intra-image dense tokens as negative examples for $\mathcal{L}_{\text{InfoNCE}}$" (Section 3.3). Without this mask, the loss would artificially push apart tokens that belong to different spatial locations of the same scene, which would degrade spatial coherence.

---

Anchoring Loss ($\mathcal{L}_{\text{anchor}}$)

The alignment loss alone is dangerous: a trivial solution that achieves perfect alignment is to collapse all student embeddings to a single constant vector (e.g., the zero vector after $L_2$ normalization, or any fixed vector). This would make $\mathcal{L}_{\text{align}}$ zero (all cosine similarities would be 1.0) but destroy all semantic information. The anchoring loss prevents this by penalizing the student's output for deviating from the teacher's pretrained representation.

$\mathcal{L}_{\text{anchor}} = \frac{1}{|M|} \sum_{m \in M} (1 - \text{sim}(h_m, h^*_m))$

where $M = \{\text{RGB}, \text{Depth}, \text{Seg}\}$ is the set of modalities, $h_m$ is the $L_2$-normalized student embedding for modality $m$, $h^*_m$ is the $L_2$-normalized teacher embedding for the same modality and same input, and $\text{sim}(\cdot, \cdot)$ is cosine similarity.

What it computes: For each modality, the loss is one minus the cosine similarity between the student's output and the teacher's output for the exact same input image. When the student's output perfectly matches the teacher's (cosine similarity = 1), the loss is zero. When the student's output is orthogonal to the teacher's (cosine similarity = 0), the loss is 1. When they point in opposite directions (cosine similarity = -1), the loss is 2. The loss is averaged over all three modalities.

Why this form: Cosine distance ($1 - \text{cosine similarity}$) is a natural metric for $L_2$-normalized vectors. It is bounded in $[0, 2]$ and focuses purely on angular deviation, ignoring magnitude (since vectors are normalized). This is appropriate because the InfoNCE loss also operates on cosine similarities — the two losses are in the same geometric space. Using $L_2$ distance instead would require careful tuning of the distance scale relative to the temperature $\tau$ in the alignment loss.

The role of $\lambda_{\text{anchor}}$. The hyperparameter $\lambda_{\text{anchor}}$ controls the trade-off between the two competing objectives. At $\lambda_{\text{anchor}} = 0$, the model optimizes only for alignment — it produces excellent cross-modal similarity but catastrophically forgets the pretrained semantics (the features "drift" to a degenerate representation). At very high $\lambda_{\text{anchor}}$ (e.g., 100.0), the anchoring loss dominates — the student's outputs stay very close to the teacher's, preserving most of the pretrained discriminability, but cross-modal alignment improves only marginally. Figure 4a visualizes this as a Pareto frontier on ScanNet: the x-axis measures alignment (cosine similarity of RGB-Depth pairs for the same scene), the y-axis measures cross-scene discernibility (1 minus cosine similarity of distinct RGB scenes). The frozen DINOv2 baseline sits at (0.28, 0.80) — poorly aligned but highly discriminative. As $\lambda_{\text{anchor}}$ increases, the adapted features trace a frontier from excellent alignment/poor discernibility toward the DINOv2 baseline, with the default $\lambda_{\text{anchor}} = 10$ achieving a balanced operating point.

How the anchoring loss prevents collapse differently from CMC. Contrastive Multiview Coding (Tian et al., 2020) prevents collapse by pushing apart embeddings of different scenes — this is also what the InfoNCE loss does through its denominator. However, in the highly imbalanced modality regime (RGB abundant, depth/segmentation scarce), the InfoNCE denominator may not provide enough "repulsive force" because the number of effective negatives from rare modalities is small. The anchoring loss adds an independent, per-sample regularization that directly penalizes deviation from the pretrained representation, regardless of the batch composition. This is crucial for training stability when one modality dominates the batch statistics.

Implementation detail: anchoring on all modalities. The loss is computed over all three modalities present in the batch, even though the teacher was originally trained only on RGB. For depth and segmentation inputs, the teacher $f_T$ still produces an output $h^*_d$ or $h^*_s$ — the teacher simply processes them as if they were RGB images (which they are, after natural colorization — they are three-channel images with naturalistic colors). The anchoring loss on these modalities says: "when the student sees a colorized depth map, its output shouldn't diverge too far from what the frozen teacher would output for that same image." Since the teacher processes the colorized depth map through its RGB-pretrained filters, this effectively preserves the teacher's structural understanding of the scene even though the teacher never explicitly learned that this particular input is "depth."

---

Total Objective and Implementation

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{align}} + 10 \cdot \mathcal{L}_{\text{anchor}}$

This is the full objective. The value $\lambda_{\text{anchor}} = 10$ was selected based on the Pareto frontier analysis in Figure 4a and the downstream task performance analysis in Figure 4b. The latter plots depth prediction $\delta_1$ (on NYUv2) against segmentation mIoU (on Cityscapes) for different $\lambda_{\text{anchor}}$ values, showing that $\lambda_{\text{anchor}} = 10$ achieves strong performance on both tasks simultaneously, while extreme values ($\lambda_{\text{anchor}} = 0$ or $\lambda_{\text{anchor}} = 100$) sacrifice one task for the other.

Gradient flow. Both losses are computed on the student's outputs $h_m$. The gradients flow from $\mathcal{L}_{\text{total}}$ backward through the student head $g$ and stop at the frozen backbone $f^*$. The teacher head $g^*$ never receives gradients. The frozen backbone $f^*$ processes the inputs to produce intermediate tokens, but these tokens are treated as constants for the backward pass.

Feature extraction. The losses are computed on $L_2$-normalized feature vectors. The paper evaluates both the CLS token embedding (the standard image-level representation output by the ViT) and spatially pooled dense features (global average pooling of the $16 \times 16$ patch tokens, denoted GAP). Both are used during training (the loss is computed separately for each) and evaluated in downstream experiments.

---

Training Configuration

The training hyperparameters are specified in Table 7 and are relatively lightweight compared to pretraining a foundation model:

Model architecture: DINOv2 ViT-B/14 with 12 transformer blocks, patch size 14, 173M parameters. Layers 0–7 (first 8 blocks) are frozen; layers 8–11 (last 4 blocks) are fine-tuned.

Optimizer: AdamW with learning rate $1 \times 10^{-4}$. The paper does not specify $\beta_1$, $\beta_2$, weight decay, or learning rate schedule details beyond this single value, but AdamW defaults typically use $\beta_1 = 0.9$, $\beta_2 = 0.999$, and weight decay values in the range $10^{-4}$ to $10^{-2}$.

Training duration: 20,000 steps with a global batch size of 512. At 512 scenes per step (each scene contributing 3 augmented modalities = 1,536 images), the total number of images seen during training is $20,000 \times 512 \times 3 = 30,720,000$ image views. This may include repeated sampling from the same datasets over multiple passes, as the total unique scene count across the six datasets is likely smaller than 10 million.

Hardware: TPU v4 in a $4 \times 4 \times 4$ topology (64 TPU v4 chips). Total runtime: 1 hour 14 minutes. This is a remarkably short training time compared to DINOv2's original pretraining (which used hundreds of GPUs for days), underscoring the efficiency of the post-hoc alignment approach.

Input resolution: $224 \times 224$ pixels for all modalities. RGB is resized using bilinear interpolation; depth and segmentation maps use nearest-neighbor interpolation to preserve sharp boundaries, followed by center cropping to square.

Datasets: ScanNet, TartanAir, Hypersim, MOVi, PointOdyssey, DynamicReplica (detailed in Appendix §6.1).

Data augmentation (training only): Photometric augmentation on RGB (brightness $\pm 0.1$, saturation $\times [0.8, 1.2]$, hue $\pm 0.03$, contrast $\times [0.8, 1.2]$), natural colorization on depth and segmentation (Algorithm 1, 64 bins, kernel size 5), normalization using ImageNet-1k statistics, and modality mixup with $\alpha_{\text{max}} = 0.5$.

Evaluation preprocessing: For evaluation (retrieval, downstream tasks), modality mixup is turned off ($\alpha = 0$), meaning depth and segmentation maps are purely colorized without RGB blending. This tests whether the model learned alignment of the "pure" modalities, not just the mixed versions it was trained on.

---

Design Rationale Summary

The design choices in this method collectively address a single core challenge: how do you align representations across severely imbalanced modalities without collapsing the feature space? Each component targets a specific failure mode or shortcut:

• Natural colorization prevents the encoder from using colormap statistics as a shortcut. By rendering depth and segmentation maps with scene-specific natural colors, the model cannot distinguish modalities by channel correlations or intensity distributions — it must attend to geometric structure.

• Modality mixup bridges the texture gap between structural maps (smooth, textureless) and RGB images (full texture). By blending these into a continuous spectrum, it prevents the model from learning discrete modality categories and instead teaches a smooth invariance to the texture-to-structure ratio. It also provides dense negatives for contrastive learning, crucial when some modalities are underrepresented.

• Symmetric InfoNCE alignment over all three modality pairs ensures a consistent shared space with no privileged modality. The learned temperature adapts the sharpness of the contrastive discrimination per batch.

• The anchoring loss prevents representational collapse — the trivial solution where all embeddings converge to a single point. By distilling from the frozen teacher, it preserves the pretrained semantic structure while allowing the student to reorganize features for alignment. The $\lambda_{\text{anchor}}$ hyperparameter provides explicit control over the alignment-discriminability tradeoff.

• Parameter-efficient fine-tuning (freezing the first 8 of 12 blocks) preserves low-level feature detectors that are modality-agnostic (edges, blobs, textures) while giving the upper layers enough capacity to learn alignment. This is both computationally efficient (1 hour 14 minutes on TPU v4) and prevents catastrophic forgetting of the expensive pretrained features.

• Dense token supervision (computing the loss on both CLS tokens and 64 randomly sampled patch tokens) ensures that alignment happens at both the image level (is this the same scene?) and the spatial level (do these patches correspond to the same region?). This is important for downstream dense prediction tasks like depth estimation and segmentation that rely on spatially localized features.

The architecture's simplicity — a single ViT with frozen early layers and fine-tuned late layers, no modality-specific branches — is itself a design choice that enables the zero-shot cross-modal transfer property. A depth head trained on RGB features from this encoder automatically works on segmentation inputs because the same $g$ processes both, mapping them to the same space. Any architecture with modality-specific processing would require explicit transfer mechanisms (adapters, domain translation) to achieve this property.

4. Key Insights and Innovations

Innovation 1: Post-Hoc Cross-Modal Alignment as a Viable Alternative to From-Scratch Multimodal Pretraining

The dominant paradigm for building multimodal vision encoders has been co-training from scratch: design a unified architecture that ingests multiple input types, then train it jointly on mixed-modality data so that shared parameters naturally develop cross-modal invariances. This is the approach of Omnivore (Girdhar et al., 2022), ImageBind (Girdhar et al., 2023), and Unified-IO (Lu et al., 2023). The implicit assumption is that cross-modal alignment must be baked into the model during its foundational pretraining phase — that it cannot be added later without catastrophic forgetting or representational collapse.

This paper challenges that assumption directly by demonstrating that high-quality cross-modal alignment can be achieved post-hoc, on top of a frozen unimodal foundation model, with minimal training and parameter updates. The method does not retrain DINOv2; it fine-tunes only the final 4 of 12 transformer blocks for 20,000 steps (1 hour 14 minutes on a modest TPU pod; Table 7). The result is an encoder that substantially outperforms the original DINOv2 on cross-modal retrieval (R@1 on ScanNet improves from 4.6% to 46.1%; Table 1) while simultaneously improving single-modality task performance (ImageNet linear-probe accuracy rises from 80.4% to 83.8%; Table 3).

The conceptual significance of this finding is that it decouples two properties of a good representation — semantic discriminability (knowing what a scene contains) and cross-modal invariance (recognizing the same scene through different sensors) — that the field had implicitly assumed were intertwined. DINOv2 was already highly discriminative but not cross-modally invariant. The Omnivorous fine-tuning procedure adds invariance without sacrificing discriminability, and in fact slightly improves it. This implies that the cross-modal alignment signal acts as a form of representation regularization that enriches the feature space rather than merely compressing it — the structural modalities provide geometric supervision that clarifies ambiguous RGB semantics.

This is not merely an engineering convenience (avoiding expensive retraining). It is a reconceptualization of what foundation model training should produce. A pretrained encoder need not solve every invariance problem during its initial training; it need only produce a rich enough feature basis that later, targeted fine-tuning can reorganize into modality-agnostic form. This mirrors the evolution in NLP, where pretrained monolingual models were retrofitted for cross-lingual transfer through post-hoc alignment techniques (Artetxe et al., 2017; Conneau and Lample, 2019), rather than requiring multilingual pretraining from scratch. The paper explicitly invokes this parallel (Section 1), but the intellectual contribution is demonstrating that the analogy holds empirically in the vision domain — that structural modalities (depth, segmentation) behave analogously to low-resource languages that benefit from alignment with a high-resource "pivot" (RGB).

Innovation 2: The Anchoring Loss as a General Mechanism for Preventing Collapse in Imbalanced Multi-View Contrastive Learning

The problem of representational collapse in contrastive learning is well-known: if you maximize agreement between views of the same data without sufficient "repulsive force," the encoder learns to output a constant vector. The standard solution, pioneered by Contrastive Multiview Coding (Tian et al., 2020) and embedded in InfoNCE-style losses, is to push apart embeddings of different scenes — the denominator in Equation 1 serves exactly this purpose. The negative examples create a repulsive term that prevents collapse by requiring the encoder to maintain enough distinctiveness to tell scenes apart.

The paper's key insight is that when modalities are severely imbalanced — RGB is abundant, depth and segmentation are scarce — the repulsive force from negative examples is insufficient. In a batch of 512 scenes, the InfoNCE denominator for a rare modality pair may contain mostly RGB negatives that are easy to distinguish from depth queries (because the pretrained features are modality-misaligned), providing weak gradient signal for learning shared structure. Simultaneously, the alignment numerator pulls embeddings of the same scene together, and without adequate repulsion, the encoder can drift toward a collapsed representation that solves the alignment task trivially.

The anchoring loss (Equation 3) addresses this with a fundamentally different mechanism: rather than relying on batch statistics (negatives), it adds a per-sample regularization that directly penalizes deviation from a frozen teacher's output. This is distillation, but with a specific purpose that goes beyond standard knowledge distillation (which aims to transfer capabilities from a larger to a smaller model). Here, the teacher and student architectures are identical; the purpose is not compression but stabilization. The frozen teacher provides a fixed reference frame that prevents the student's representations from wandering into degenerate regions of the embedding space, regardless of batch composition.

The conceptual novelty is in recognizing that collapse can occur even when a contrastive loss is properly formulated, because the balance of forces (alignment pressure vs. repulsion from negatives) depends on the effective number and difficulty of negatives, which are modality-dependent. The anchoring loss decouples the anti-collapse mechanism from the batch — it is a "local" regularizer that works per-sample, complementing the "global" repulsive force from InfoNCE negatives. Figure 4a provides the clean empirical demonstration: as λ_anchor varies from 0 to 100, the model traces a continuous Pareto frontier between alignment quality and cross-scene discriminability, with the anchoring loss acting as an explicit control knob. This is not just a hyperparameter sweep; it is evidence that the two forces (alignment and preservation) are independently controllable through this loss design, which is not possible with InfoNCE alone.

This finding has implications beyond this paper: any multi-view contrastive learning scenario with imbalanced view frequencies or varying view difficulty could benefit from a similar per-sample anchoring mechanism. The teacher need not be a frozen pretrained model — it could be an exponential moving average of the student (as in BYOL or MoCo), providing a slowly-updating reference that prevents rapid drift.

Innovation 3: Data-Centric Hard-Positive Construction as a Principled Solution to Modality-Specific Shortcut Learning

A recurring challenge in multimodal learning is that networks exploit trivial correlations to solve alignment tasks without learning the intended invariance. In vision, this manifests as "shortcut learning" from colormaps: an encoder fed grayscale depth maps and RGB photos can distinguish modalities by channel statistics (grayscale has identical R, G, B values) rather than geometric content, and can then "align" features by learning a modality-specific transformation that collapses these statistics without ever learning shared scene structure.

Prior cross-modal alignment methods (CLIP2Point, CoMAE, Mask3D) acknowledged the modality gap but addressed it primarily through architectural or loss design — contrastive objectives, reconstruction losses, or modality-specific encoders. They did not systematically address the data representation of non-RGB modalities as a source of shortcuts. The default rendering of depth as grayscale or segmentation as a fixed colormap was implicitly accepted as the input format, and the model was expected to learn invariance despite these low-level cues.

The paper's natural colorization strategy (Algorithm 1) makes a fundamentally different move: it changes the input representation to actively remove the shortcut signal before it reaches the model. By rendering depth and segmentation maps with a color palette extracted from the paired RGB image, the colorized maps share the same channel statistics as the RGB photo. The encoder cannot use color to distinguish modalities; it must attend to structural differences — texture presence vs. absence, edge sharpness, spatial layout — which are exactly the geometric properties that cross-modal alignment should be based on.

This is a data-centric insight rather than a modeling insight, but its significance is that it reframes the problem: shortcut learning is not something to be overcome through better architectures or regularization; it is something to be prevented through better input design. The paper demonstrates that this matters empirically: without natural colorization, the InfoNCE loss would be largely optimizing a modality-identification task (is this input RGB, depth, or segmentation?) rather than a scene-identification task (is this the same scene as that other modality?). The fact that the resulting encoder generalizes to completely unseen modalities (NOCS maps; Table 5) — which were never colorized with a natural palette during training — suggests that the learned alignment is genuinely structural rather than tied to the specific colorization procedure.

The modality mixup extension of this idea — blending structural maps with RGB to create a continuous modality spectrum — further demonstrates that data augmentation, when designed with the specific failure mode in mind, can substitute for architectural complexity. The model never sees "pure" depth or segmentation during training (α_max = 0.5 ensures at least 50% structural signal), yet generalizes to pure modalities at test time (α = 0). This is a non-trivial result: it means the model learned to extrapolate from mixed to pure inputs, which implies it developed an internal representation of "modality-ness" as a continuous variable that can be zeroed out when needed.

The practical implication is that data preprocessing choices — which are often treated as mundane engineering details — can be the determining factor in whether a cross-modal alignment task is solvable at all. This is a lesson that transfers to any domain where alignment must be learned across heterogeneous data types with identifiable superficial differences (e.g., aligning sketches with photos, simulated with real images, or different medical imaging modalities).

Innovation 4: Empirical Proof That Cross-Modal Alignment Improves, Rather Than Compromises, Single-Modality Representations

A natural concern with any cross-modal alignment method is that pulling representations of different modalities together will "blur" the feature space — that the loss of modality-specific information will degrade performance on tasks that rely on the fine-grained structure of the original unimodal representation. This concern is particularly acute when the aligned modalities provide complementary rather than redundant information: depth maps capture 3D geometry that is only implicit in RGB; segmentation maps provide categorical boundaries that may not correspond to visual edges. Forcing these distinct signal types into a shared embedding could, in principle, harm the encoder's ability to represent either one well.

The paper addresses this concern empirically and arrives at a striking result: cross-modal alignment not only preserves but actually improves single-modality downstream performance. On ImageNet linear-probe classification, the Omnivorous encoder achieves 83.8% top-1 accuracy compared to DINOv2's 80.4% (Table 3). On NYUv2 depth estimation with a linear head, RMSE drops from 0.405 to 0.377 (Table 2). On ADE20k segmentation with a linear head, mIoU rises from 0.463 to 0.475. These are not marginal gains — they are substantial improvements on the original encoder's primary modality (RGB) achieved by a training procedure that never used ImageNet, NYUv2, or ADE20k labels and only optimized for cross-modal alignment and teacher distillation.

The intellectual significance of this result is that it challenges the zero-sum assumption about representational capacity. The finding suggests that structural modalities provide a form of self-supervision that enriches RGB representations — the geometric regularities learned from aligning with depth and segmentation maps transfer to better semantic understanding even when those modalities are absent at test time. This aligns with the broader hypothesis in multimodal learning that "auxiliary" modalities act as inductive biases that help the model discover structure it might otherwise miss (e.g., depth provides cues about object boundaries and spatial relationships; segmentation provides cues about object categories and their spatial extent).

The mechanism is likely that the symmetric cross-modal alignment loss forces the RGB pathway to produce features that are predictable from depth and segmentation features (and vice versa). This means the RGB encoder must learn to represent geometric and categorical properties explicitly enough that they can be matched across modalities, which in turn makes these properties more accessible for downstream tasks that depend on them. In effect, the alignment loss acts as a form of multimodal pretraining for single-modality deployment — training with extra modalities improves the representation for the primary modality alone.

The k-NN classification results (Table 4) provide a more nuanced picture: while ImageNet soft-voting accuracy is essentially at parity (81.94% vs. 81.97%), transfer to fine-grained datasets shows mixed results — improvements on RP2K (+3.65%) but regressions on iNaturalist and Google Landmarks v2. This suggests that the alignment procedure's benefits are not universal; they depend on whether the structural regularities present in the training data (which includes synthetic multi-object scenes) transfer to the target domain. The authors acknowledge this (Section 4.2): "while 'omnivorous' alignment generally preserves semantics, the choice of training data nevertheless matters." This honest reporting of negative results strengthens rather than weakens the contribution, because it establishes the boundary conditions for the "alignment improves single-modality performance" claim.

The practical implication is significant: practitioners considering cross-modal alignment need not fear that they are trading single-modality performance for multi-modality transfer. At least for the vision modalities studied here, alignment is a free lunch — it improves cross-modal capabilities while either maintaining or boosting single-modality task performance. This removes a major barrier to adoption and suggests that post-hoc alignment could become a standard fine-tuning step for deployed vision encoders, analogous to how instruction tuning is now standard for LLMs.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses six datasets for training (ScanNet, TartanAir, Hypersim, MOVi, PointOdyssey, DynamicReplica) spanning real and synthetic indoor/outdoor scenes with paired RGB, depth, and segmentation modalities. Evaluation datasets vary by task: ScanNet (3,072 scenes), MOVi (128 scenes), and TartanAir (128 scenes) for cross-modal retrieval; NYUv2 and NAVI for monocular depth estimation; ADE20k, Cityscapes, and Pascal VOC for semantic segmentation; ImageNet-1k, iNaturalist, SOP, Google Landmarks v2, RP2K, and Food2k for classification; and PACE for zero-shot cross-modal transfer. All training images are processed to 224×224 resolution (Table 7).

• Base model(s). The primary teacher and initialization is DINOv2 ViT-B/14 (173M parameters, 12 transformer blocks, patch size 14), a state-of-the-art self-supervised vision encoder chosen because it represents "the capabilities of many contemporary LLMs" (Section 4) in the vision domain and exhibits the specific cross-modal misalignment that the paper aims to address (Figure 1). An ablation replaces DINOv2 with TIPS (ViT-B/14, Text-Image Pretraining with Spatial Awareness) to test generality. The student network is the same ViT-B/14 architecture with the first 8 blocks frozen and the last 4 fine-tuned.

• Metrics. Cross-modal retrieval uses Recall@k (R@1, R@5), Mean Average Precision (mAP, equivalent to Mean Reciprocal Rank since exactly one positive match exists per query), and Median Rank (MedR), averaged over all 6 directed modality pairs (RGB→Depth, Depth→RGB, RGB→Seg, Seg→RGB, Depth→Seg, Seg→Depth). Monocular depth estimation uses RMSE (root mean squared error in meters) and δ₁ accuracy (percentage of pixels where predicted depth is within a factor of 1.25 of ground truth). Semantic segmentation uses mean Intersection-over-Union (mIoU) computed by aggregating confusion matrices across the full validation set. Classification uses top-1 accuracy for both linear probes and k-NN. 3D correspondence uses Percentage of Correct Keypoints (PCK) at a strict threshold of 0.0, requiring exact patch-level matches on a 16×16 grid.

• Baselines. The primary baseline throughout is the frozen, unmodified DINOv2 ViT-B/14 (Oquab et al., 2023), serving as both the teacher for distillation and the comparison point for all downstream tasks. For the zero-shot cross-modal transfer experiment, the baseline is the SAME DINOv2 backbone with depth prediction heads trained identically on RGB inputs from NYUv2. For classification, the paper also compares against DINOv2's published k-NN results on standard transfer datasets. No other multimodal alignment methods (CLIP2Point, CoMAE, ImageBind, etc.) are directly evaluated as baselines—the comparison is exclusively to the unimodal DINOv2 teacher.

• Generation budget / compute accounting. Training uses 20,000 steps with global batch size 512 on TPU v4 (4×4×4 topology, 64 chips), totaling 1 hour 14 minutes (Table 7). At test time, all methods use a single forward pass per input (no test-time compute scaling, no ensemble). The parameter efficiency of the method is measured by the number of trainable parameters (~1/3 of the ViT-B/14, approximately 58M out of 173M) and training time relative to full pretraining.

• Cross-validation / statistical protocol. No k-fold cross-validation is reported. For linear-probe classification on ImageNet, the authors sweep over five learning rates (base values: [0.15, 0.2, 0.5, 1.0, 2.0]) and report the best accuracy achieved. For k-NN classification, the best k is selected from {5, 10, 20, 50, 100} and reported. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any metric. The downstream evaluation protocols (depth, segmentation, classification) follow the procedures established by DINOv2 (Oquab et al., 2023) or Probe3D (El Banani et al., 2024), inheriting their validation splits and preprocessing pipelines.

---

Main Quantitative Results

Cross-Modal Retrieval (Section 4.1, Table 1)

The headline result: On ScanNet (3,072 scenes), the Omnivorous encoder improves R@1 from 4.6% to 46.1% using global average pooled (GAP) features—a 10× improvement—while reducing Median Rank from 401.8 to 2.0. The frozen DINOv2 baseline exhibits essentially random alignment: its R@1 of 4.6% is close to what random chance would produce (~0.03% for 3,072-way retrieval). The Omnivorous encoder achieves near-perfect alignment on synthetic datasets: R@1 of 86.2% on MOVi and 90.6% on TartanAir (GAP features), compared to DINOv2's 15.5% and 46.6% respectively.

Detailed comparison across feature types: The CLS token embedding consistently underperforms GAP pooling for both models, but the gap is larger for the Omnivorous encoder. On ScanNet, Omnivorous Tok achieves R@1 of 30.2% vs. GAP's 46.1%—a substantial difference suggesting that the alignment signal is stronger in spatially averaged features than in the global summary token. DINOv2 shows a smaller gap (3.9% Tok vs. 4.6% GAP). The same pattern holds on MOVi (76.6% Tok vs. 86.2% GAP) and TartanAir (84.5% vs. 90.6%).

The paper reports results averaged over all six directed modality pairs. Table 8 in the appendix provides the underlying per-pair cosine similarities, revealing an asymmetry: on ScanNet, the Omnivorous encoder achieves <R,D> similarity of 0.600 but <D,S> similarity of 0.663, suggesting that depth and segmentation are more easily aligned with each other (both being structural modalities lacking RGB texture) than with RGB. DINOv2 shows <R,S> similarity of only 0.216—segmentation and RGB are the most misaligned pair in the baseline.

The cross-scene discernibility metric (Table 8) confirms the trade-off: DINOv2's <R₁,R₂> similarity (between unrelated RGB scenes) is 0.198 on ScanNet (lower is better for discriminability), while the Omnivorous encoder's is 0.259—a degradation indicating that some between-scene discrimination has been sacrificed for cross-modal alignment. This is the Pareto trade-off visualized in Figure 4a.

Downstream Task Performance (Section 4.2, Tables 2-4)

Monocular depth estimation (Table 2). The Omnivorous encoder matches or exceeds DINOv2 across all metrics. With a linear head: on NYUv2, RMSE drops from 0.405 to 0.377 (7% relative improvement), δ₁ rises from 0.875 to 0.896. On NAVI, improvements are marginal (RMSE 0.074 vs. 0.076, δ₁ 0.706 vs. 0.697). With the more expressive DPT head, performance is effectively identical: NYUv2 δ₁ = 0.948 for both models, RMSE = 0.297 for both—the alignment procedure neither helps nor hurts when the decoder has sufficient capacity to extract depth-relevant features from either representation.

Semantic segmentation (Table 2). With a linear head, the Omnivorous encoder shows consistent small gains: ADE20k mIoU improves from 0.463 to 0.475 (+2.6%), Cityscapes from 0.622 to 0.632 (+1.6%), Pascal VOC from 0.814 to 0.826 (+1.5%). With the DPT head, results are mixed: ADE20k improves (0.496 → 0.505, +1.8%), Cityscapes slightly degrades (0.737 → 0.732, -0.7%), Pascal VOC is at parity (0.855 vs. 0.857). The qualitative results in Figures 7-8 suggest reduced over-segmentation and greater resilience to textural details in the input images, consistent with the hypothesis that alignment with depth encourages the encoder to focus on geometric boundaries rather than photometric edges.

Classification—linear probe (Table 3). On ImageNet-1k, the Omnivorous encoder achieves 83.5% accuracy using only the CLS token (vs. 80.1% for DINOv2, a +3.4 percentage point improvement), and 83.8% using CLS concatenated with GAP features (vs. 80.4%, +3.4pp). This is a substantial improvement—effectively "free" accuracy gained by training on non-ImageNet, non-labeled data—and is one of the paper's most striking results. It implies that the cross-modal alignment objective acts as a powerful regularization that enriches ImageNet-relevant semantics even though ImageNet labels and images were never used during Omnivorous training.

Classification—k-NN (Table 4). The results are nuanced. On ImageNet with soft voting, performance is at parity (81.974% Omnivorous vs. 81.936% DINOv2), confirming that representational drift has been effectively mitigated by the anchoring loss. On transfer datasets: RP2K shows a large gain (70.48% vs. 66.83%, +3.65pp), Food2k shows a small gain (52.14% vs. 51.90%), SOP is essentially flat (54.69% vs. 54.39%), while iNaturalist (77.49% vs. 78.53%, -1.04pp) and Google Landmarks v2 (50.13% vs. 51.90%, -1.77pp) show regressions. The authors attribute this to training data composition ("our training mix includes a significant amount of simulated multi-object data"), which benefits object-centric datasets like RP2K but may harm fine-grained recognition on natural-species datasets like iNaturalist.

Zero-Shot Cross-Modal Transfer (Section 4.3, Table 5)

This experiment is the paper's most direct test of modality-agnostic representation. A depth prediction head (Linear or DPT) is trained on NYUv2 RGB images only, then evaluated on PACE dataset inputs in three modalities: RGB (in-distribution for both backbones), Segmentation maps (in-distribution for Omnivorous, out-of-distribution for DINO as anything other than RGB), and NOCS maps (out-of-distribution for both models).

On RGB inputs (the training modality), the Omnivorous encoder outperforms DINOv2: Linear head δ₁ improves from 0.108 to 0.146, RMSE drops from 0.842 to 0.671; DPT head δ₁ improves from 0.420 to 0.463, RMSE drops from 0.318 to 0.290. This confirms that the alignment procedure improves even in-modality transfer under domain shift (NYUv2 → PACE).

On Segmentation inputs, DINOv2 catastrophically fails: Linear head δ₁ = 0.003, RMSE = 1.536 (effectively random, since depth values in PACE are unlikely to be exactly predicted by chance). The Omnivorous encoder achieves δ₁ = 0.184, RMSE = 0.532—far from perfect but clearly extracting meaningful depth from a modality never seen by the depth head. The DPT head shows a similar pattern: DINOv2 δ₁ = 0.042, RMSE = 0.792 vs. Omnivorous δ₁ = 0.169, RMSE = 0.507.

On NOCS maps (completely out-of-distribution—neither backbone was trained on normalized object coordinate spaces), both models struggle, but Omnivorous maintains an advantage: Linear head RMSE 1.075 vs. 1.996, DPT head RMSE 0.822 vs. 0.979. The absolute performance is poor (δ₁ = 0.029 for Omnivorous DPT vs. 0.014 for DINOv2 DPT), but the relative improvement suggests that the alignment procedure has learned a more general notion of scene geometry that partially transfers even to coordinate-field representations it never encountered.

The qualitative results in Table 5's bottom panel show the DPT predictions: DINOv2 produces nearly uniform depth estimates on Segmentation and NOCS inputs (reflecting catastrophic failure), while Omnivorous produces depth maps with visible object boundaries and spatial structure on Segmentation inputs, though with significant artifacts on NOCS.

PCA Visualizations (Section 5, Figure 5)

Figure 5 provides qualitative evidence by visualizing the top three Principal Components of the frozen (DINOv2) and adapted (Omnivorous) features for two example scenes from MOVi and ScanNet. The frozen features show disjoint color distributions across modalities—RGB, Depth, and Segmentation feature maps appear in different hues, confirming they occupy separate feature subspaces. The adapted features show Depth and Segmentation aligned closely with RGB in both color palette and spatial structure, providing a visual confirmation that the adapter has successfully unified the modalities into a shared semantic space "without discarding spatial geometry."

Diagnostic Metrics (Appendix 7.1, Table 8)

Table 8 quantifies the per-modality-pair cosine similarities that underlie the retrieval results. The Omnivorous encoder at λ_anchor = 10 (the default checkpoint) increases all cross-modal similarities substantially: on ScanNet, <R,D> rises from 0.285 to 0.600, <R,S> from 0.216 to 0.550, <D,S> from 0.413 to 0.663. The cross-scene discriminability degrades: <R₁,R₂> similarity (between unrelated RGB images, where lower is better) rises from 0.198 to 0.259 on ScanNet, from 0.237 to 0.279 on MOVi, and from 0.172 to 0.223 on TartanAir. This confirms the Pareto trade-off: alignment comes at some cost to between-scene discrimination, but the magnitude is far smaller than the alignment gains.

---

Ablation Studies and Robustness Checks

Anchoring loss weight (λ_anchor) in Figure 4: Figure 4a plots the Pareto frontier between cross-modal alignment (cosine similarity of <RGB, Depth> pairs) and cross-scene discernibility (1 - cosine similarity of distinct RGB scenes) on ScanNet. The frozen DINOv2 baseline is at (alignment ≈ 0.28, discernibility ≈ 0.80). As λ_anchor varies from 0.0 to 100.0, the adapted features trace a curve: at λ_anchor = 0, alignment ≈ 0.70 but discernibility drops to ≈ 0.65; at λ_anchor = 100, alignment ≈ 0.55 and discernibility ≈ 0.78. The default λ_anchor = 10 achieves alignment ≈ 0.60 and discernibility ≈ 0.74—a balanced operating point. Figure 4b examines the effect on downstream tasks: λ_anchor = 10 achieves the best joint depth prediction (NYUv2 δ₁) and segmentation (Cityscapes mIoU) performance. The datapoint for λ_anchor = 0 is omitted from Figure 4b "for clarity, as it was too far below the remaining datapoints" (segmentation mIoU ≈ 0.356)—this is the collapsed representation regime where alignment is excellent but semantics are destroyed.

Modality mixup blending range (α_max) in Table 6: The mixing coefficient α_max controls how much RGB texture is blended into depth and segmentation inputs during training (0 = no blending, 1.0 = full blending where structural inputs become identical to RGB at α = 1). Results on four tasks:

• Classification (ImageNet linear probe, TOK): 83.1% at α_max = 0 → 83.5% at α_max = 1.0 (monotonically increasing, +0.4pp)
• Depth (NYUv2 δ₁, linear head): 89.9% at α_max = 0 → 89.1% at α_max = 1.0 (monotonically decreasing, -0.8pp)
• Segmentation (Cityscapes mIoU, linear head): 62.4% at α_max = 0 → 63.2% at α_max = 0.75 and 1.0 (improving from 0 to 0.5, then plateauing)
• 3D correspondence (NAVI PCK@0): 28.40% at α_max = 0 → 29.04% at α_max = 0.75, then 29.03% at α_max = 1.0 (improving and plateauing)

The default α_max = 0.5 is chosen as a compromise: it captures most of the gains for classification, segmentation, and 3D correspondence while limiting the regression on depth prediction. The paper notes that "depth prediction is an outlier" in the trend, potentially because blending RGB texture into depth images during training makes the depth signal less pure, slightly harming the encoder's ability to extract precise geometric information from depth inputs at test time (when α = 0, no blending).

Alternative foundation model (TIPS instead of DINOv2) in Figure 9: Replacing DINOv2 with TIPS as the teacher backbone produces qualitatively similar behavior. Figure 9a shows the same alignment-discernibility Pareto frontier on TartanAir, with the frozen TIPS baseline at a different position (TIPS has better native alignment than DINOv2) but exhibiting the same trade-off as λ_anchor varies. Figure 9b shows that higher λ_anchor values (stronger anchoring to the frozen teacher) improve depth and segmentation performance relative to low λ_anchor. The paper states this "attests to the generality of the Omnivorous framework regardless of the choice of pretrained teacher network."

Adapter-on-top vs. fine-tuning final blocks in Table 11: Comparing the default approach (fine-tuning the last 4 of 12 ViT blocks) against training a separate 4-block adapter on top of the fully frozen backbone:

• Classification (ImageNet linear probe, TOK & GAP): 84.0% (adapter) vs. 83.8% (fine-tuned blocks)—small advantage for adapter
• k-NN classification (ImageNet): 81.832% vs. 81.974%—essentially tied
• Depth (NYUv2 δ₁, linear head): 0.905 vs. 0.896—small advantage for adapter
• Depth (NAVI δ₁, linear head): 0.679 vs. 0.706—notable advantage for fine-tuned blocks
• Segmentation (Cityscapes mIoU, linear head): 0.628 vs. 0.632—essentially tied
• 3D correspondence (NAVI PCK@0): 28.15% vs. 29.00%—moderate advantage for fine-tuned blocks

The methods are broadly comparable, with fine-tuned blocks showing advantages on geometric tasks (depth on synthetic NAVI data, 3D correspondence) while the adapter shows a slight edge on ImageNet classification. This demonstrates robustness to the specific parameterization.

Number of frozen blocks in Table 12: Ablating L_stop-gradient (the number of early ViT blocks kept frozen) across values {4, 6, 8, 10} out of 12 total blocks:

• With a DPT head: freezing 4 blocks gives NYUv2 depth δ₁ = 0.948, ADE20k segmentation mIoU = 0.495. Freezing 8 blocks (default) gives δ₁ = 0.948, mIoU = 0.505—improvement on segmentation, parity on depth. Freezing 10 blocks gives δ₁ = 0.949, mIoU = 0.504—slightly better depth but regressed segmentation.
• With a linear head: freezing 4 blocks gives NYUv2 δ₁ = 0.894, Cityscapes mIoU = 0.622. Freezing 8 blocks gives δ₁ = 0.896, mIoU = 0.632—improvement on both. Freezing 10 blocks gives δ₁ = 0.895, mIoU = 0.628—regression on both relative to 8.

The default L = 8 provides the best overall balance with the linear head, and near-best performance with the DPT head. The pattern is not monotonic: freezing too few blocks (4) allows too many parameters to update, causing some loss of pretrained features; freezing too many (10) leaves insufficient trainable capacity to learn alignment.

Loss design—necessity of the anchoring loss: The paper does not ablate removing ℒ_anchor entirely and relying only on InfoNCE negatives to prevent collapse, likely because the λ_anchor = 0 datapoint in Figure 4b shows catastrophic degradation on downstream tasks (the omitted datapoint at segmentation mIoU ≈ 0.356). This is itself an informative ablation: symmetric InfoNCE alone, without anchoring, collapses representations despite having a negative-driven repulsive term, confirming the paper's claim that imbalanced modalities weaken the repulsive force.

Feature type comparison (CLS token vs. GAP): Tables 1-3 consistently show that globally average-pooled (GAP) features outperform CLS token features for cross-modal retrieval, while for classification, the combination of both (TOK & GAP) performs best. This pattern holds across all datasets and is consistent with DINOv2's own findings that dense features carry richer spatial information than the global token.

---

Critical Assessment

Claim 1: The Omnivorous encoder dramatically improves cross-modal alignment while preserving or improving single-modality performance. This claim is well-supported by multiple independent evaluation axes. The retrieval results (Table 1) unambiguously show the alignment improvement: R@1 on ScanNet goes from essentially random (4.6%) to substantially above chance (46.1%). The downstream task results (Tables 2-4) demonstrate that single-modality performance is at minimum preserved (DPT depth on NYUv2: identical 0.297 RMSE; DPT segmentation on Pascal VOC: 0.855 vs. 0.857), and in several cases meaningfully improved (ImageNet linear probe: 80.4% → 83.8%; NYUv2 depth linear head RMSE: 0.405 → 0.377). The classification improvement is particularly notable because it is a completely held-out task—the model was never trained on ImageNet labels or even ImageNet images during Omnivorous training—yet gains 3.4 percentage points in linear separability. This is strong evidence that the cross-modal alignment signal genuinely enriches the representation, as the paper claims.

However, a careful reader should note what this claim does NOT mean: it does not mean the Omnivorous encoder outperforms DINOv2 on all tasks. The k-NN transfer results (Table 4) show regressions on iNaturalist (-1.04pp) and GLDv2 (-1.77pp). The DPT head on Cityscapes shows a slight regression (0.732 vs. 0.737). The paper is transparent about these, but they establish that the "free lunch" is not universal—it depends on whether the structural modalities in the training data provide relevant inductive bias for the target task. Fine-grained species recognition (iNaturalist) may require photometric features that are partially smoothed over by alignment with depth and segmentation.

Claim 2: The anchoring loss prevents representational collapse and provides a tunable trade-off between alignment and discriminability. This claim is strongly supported by Figure 4, which directly visualizes the Pareto frontier. The λ_anchor = 0 datapoint (pure alignment, no anchoring) is catastrophically bad on downstream tasks (segmentation mIoU drops to ~0.356, so low it was omitted from the plot for clarity), confirming that collapse would occur without anchoring. The smooth variation in both alignment and discernibility as λ_anchor sweeps from 0 to 100 demonstrates that the trade-off is controllable. However, the paper does not explain why λ_anchor = 10 is the specific chosen value beyond showing it balances downstream performance in Figure 4b. There is no sweep over intermediate values (e.g., 5, 20) to justify that 10 is optimal rather than merely adequate. Given the sensitivity of the trade-off (small changes in λ_anchor produce visible shifts along the Pareto curve), this is a genuine gap.

Claim 3: Post-hoc alignment is a parameter-efficient alternative to multimodal pretraining from scratch. The training efficiency is real: 1 hour 14 minutes on 64 TPU v4 chips, updating only ~1/3 of the ViT parameters (Table 7). This is dramatically cheaper than training DINOv2 from scratch on multimodal data. However, the paper does not provide a direct comparison to any from-scratch multimodal pretraining method. No Omnivore, ImageBind, or Uni-Perceiver model is trained and evaluated under the same protocol. The claim of "efficiency" is therefore demonstrated only relative to the implicit cost of full pretraining, not through a controlled FLOPs-matched or parameter-matched comparison. This is understandable—replicating Omnivore at scale is expensive—but it means the claim is supported by inference about relative costs, not by empirical head-to-head evaluation.

Claim 4: Natural colorization and modality mixup are essential for preventing shortcut learning. The modality mixup ablation (Table 6) provides partial support: performance on classification, segmentation, and 3D correspondence improves with increasing α_max, confirming that modality blending is beneficial. However, there is no ablation of natural colorization itself—no experiment where the encoder is trained using standard grayscale depth maps and fixed-colormap segmentation masks (the "shortcut-prone" baseline). Without this ablation, we cannot know whether natural colorization is actually preventing a shortcut or merely providing a convenient input format. The paper argues that standard colormaps "allow models to shortcut alignment by relying on low-level channel statistics" (Section 1), but this claim is never empirically validated within the paper. It is a plausible hypothesis grounded in prior work on shortcut learning, but it is not an experimental finding. This is the most significant missing ablation.

Claim 5: The Omnivorous encoder enables zero-shot cross-modal transfer to unseen modalities. The NOCS result (Table 5) provides genuine evidence for this claim: a depth head trained on RGB transfers to normalized object coordinates (a modality neither backbone ever saw) with RMSE of 0.822 (Omnivorous DPT) vs. 0.979 (DINOv2 DPT). However, the absolute accuracy is extremely low (δ₁ = 0.029 for Omnivorous). The model is doing something better than random, but it is not producing useful depth predictions. The segmentation transfer is more practically meaningful: RMSE of 0.507 (Omnivorous DPT on Segmentation inputs) is not far from the 0.290 achieved on RGB inputs, indicating genuine cross-modal generalization. But the test is limited to a single dataset (PACE) and a single task (depth prediction). We do not know whether the same property holds for segmentation heads transferred across modalities, or for classification on modality-switched inputs.

Additional weaknesses and missing experiments:

• Single model scale (ViT-B/14 only). All experiments use the Base variant. There are no results for ViT-Small, ViT-Large, or ViT-Giant variants of DINOv2. The scaling behavior of the alignment procedure—does it work better or worse with larger models? does the optimal number of frozen blocks change with scale?—is unexplored.

• No comparison to alternative alignment methods. The paper contrasts with CLIP2Point, CoMAE, and Mask3D in the related work (Section 2) but never empirically compares against them. This is a significant omission because the paper's central positioning is that post-hoc, parameter-efficient alignment is practically advantageous. Showing that the Omnivorous encoder matches or exceeds the cross-modal retrieval performance of a method like CoMAE (which requires full-model pretraining) would substantiate this claim. Without such comparison, the paper demonstrates that the method improves over DINOv2 but not that it is better than alternative approaches to the same problem.

• Training data contamination concerns. The downstream evaluation datasets (NYUv2, ADE20k, Cityscapes, Pascal VOC) are standard benchmarks that may appear in DINOv2's pretraining data. The paper does not discuss whether the alignment training data includes scenes from these evaluation datasets, which could inflate downstream performance through memorization rather than representation quality. The method's robustness to this concern depends on DINOv2's data curation, which is not fully documented.

• Modality mixup creates a train-test distribution shift. During training, depth and segmentation inputs are blended with RGB (α_max = 0.5). During evaluation (retrieval, downstream tasks), α = 0 (pure structural maps). The model must extrapolate from mixed to pure modalities. The paper does not analyze how performance changes as a function of the gap between training α and test α, nor does it evaluate on mixed-modality test inputs to see whether the extrapolation is smooth. This is a robustness concern: if a deployed system encounters partially blended modalities (e.g., sensor fusion producing mixed inputs), performance could degrade unpredictably.

• No ablation of the three-modality requirement. The method trains on RGB, depth, and segmentation jointly with a symmetric loss over all three pairs. Is three modalities necessary, or would two (e.g., RGB + Depth) suffice? Does the triangular alignment (including Depth-Seg) provide additional benefit, or is RGB as a pivot enough? This is not ablated, leaving unclear whether the "omnivorous" claim requires all three or would work with any pair.

• Statistical significance is never reported. No confidence intervals, standard deviations, or error bars appear on any metric. For the retrieval results on MOVi and TartanAir (N=128), the differences between R@1 values of 86.2% and 90.6% could be statistically indistinguishable. For the downstream task improvements (e.g., Cityscapes mIoU 0.622 → 0.632), the gain of 0.01 mIoU is small and may fall within evaluation noise. Without error estimates, we cannot assess the reliability of the reported improvements, particularly the marginal ones.

• The k-NN results use different protocols across datasets. ImageNet uses soft voting with temperature τ = 0.07 and sweeps over k; transfer datasets use "hard" k-NN (equivalent to R@1) with a self-match exclusion. This inconsistency makes cross-dataset comparison difficult and complicates the interpretation of the mixed transfer results (gains on some datasets, regressions on others).

When the claims hold and when they don't: The central claim—that post-hoc cross-modal alignment improves both alignment and single-modality performance—holds robustly for the specific setting tested: DINOv2 ViT-B/14, vision modalities (RGB, depth, segmentation), parameter-efficient fine-tuning of the upper layers, and evaluation on standard benchmarks following DINOv2/Probe3D protocols. The claim of zero-shot cross-modal transfer holds directionally (Omnivorous > DINOv2 on novel modalities) but not practically (absolute performance on NOCS is very poor). The claim of training efficiency is convincing in absolute terms (1.2 hours on TPU v4) but unvalidated in relative terms (no comparison to multimodal alternatives). The claim that data preprocessing prevents shortcut learning is plausible but not experimentally demonstrated due to the missing natural colorization ablation. A reader should accept the paper's core empirical result—that this specific recipe produces an encoder with vastly better cross-modal alignment and preserved or improved single-modality performance—while recognizing that the paper does not establish this method as superior to existing multimodal alignment techniques, does not validate the necessity of its data preprocessing innovations through controlled ablation, and does not characterize statistical uncertainty in its measurements.

6. Limitations and Trade-offs

The Natural Colorization Claim Is Empirically Unvalidated

The assumption or constraint. The paper's entire data pipeline rests on a central claim: that standard depth and segmentation colormaps (grayscale, jet) would allow the encoder to "shortcut alignment by relying on low-level channel statistics" rather than learning genuine geometric correspondence (Section 1). Natural colorization (Algorithm 1) is presented as the solution to this problem — it renders non-RGB modalities with scene-specific color palettes to force the encoder to attend to structure. The paper states:

"standard colormaps (e.g., grayscale or jet colormaps) allow models to shortcut alignment by relying on low-level channel statistics. To counter this, we colorize depth and segmentation maps using a natural color palette derived from the corresponding RGB image." (Section 1)

The consequence. No experiment in the paper tests whether this claim is true. There is no ablation comparing Omnivorous training with natural colorization against Omnivorous training with standard grayscale depth maps and fixed-colormap segmentation masks. Without this ablation, it is impossible to know whether natural colorization is preventing a real shortcut or merely providing a convenient input format that happens to work well. The method's core data innovation could be unnecessary — the alignment might succeed equally well with simpler preprocessing — or it could be actively harmful in ways that are invisible without the comparison. A practitioner implementing this method must either trust the untested hypothesis or conduct this ablation themselves, since the paper provides no guidance on whether the complexity of Algorithm 1 (64-bin quantization, palette accumulation, 1D smoothing convolution) is warranted.

What evidence exists in the paper. None. The paper provides extensive ablations for modality mixup (Table 6), the anchoring loss weight (Figure 4), the number of frozen blocks (Table 12), and the adapter parameterization (Table 11), but natural colorization — arguably the most complex component of the data pipeline — is never varied or removed. The qualitative PCA visualizations (Figure 5) show that the adapted features are well-aligned, but this confirms that the full pipeline works, not that each component is necessary.

Mitigation status. Not addressed. The paper treats natural colorization as a justified design choice rather than a testable hypothesis. Given that Algorithm 1 involves multiple design decisions (64 bins, kernel size 5, 1D convolution smoothing, per-scene palette computation), a practitioner cannot know which aspects matter or whether simpler alternatives (e.g., using the RGB image's per-channel mean and variance to normalize depth maps) would suffice.

The Difficulty Estimation Cost Is Unaccounted for in the Practical Deployment Picture

The assumption or constraint. The method is presented as a lightweight, parameter-efficient fine-tuning procedure: 1 hour 14 minutes on a 64-chip TPU v4 pod, updating only ~1/3 of the ViT parameters (Table 7). This efficiency claim is central to the paper's positioning as a practical alternative to multimodal pretraining from scratch. However, the training procedure requires paired RGB, depth, and segmentation data for every scene in the training set. Six datasets are used (ScanNet, TartanAir, Hypersim, MOVi, PointOdyssey, DynamicReplica), which collectively provide this pairing through a combination of real depth sensors, simulation engines, and human annotation pipelines.

The consequence. In practice, paired multimodal data of this quality is expensive and scarce. Real-world RGB images vastly outnumber RGB-D or RGB-segmentation pairs. The method does not address the regime where depth or segmentation data is unavailable for the target domain — it assumes access to sufficient paired data to learn cross-modal alignment. This limits applicability to domains where structural modalities are available (indoor scenes with depth sensors, driving scenes with LiDAR, simulated environments) and excludes the vast majority of natural images where only RGB is available. Furthermore, the paper does not investigate how many paired scenes are needed: would 1,000 scenes suffice, or is the full six-dataset mixture (potentially millions of scenes) required? A practitioner considering adoption for a new domain cannot estimate the data collection cost from the paper's results.

What evidence exists in the paper. The training data composition is documented in Section 3.2 and Appendix 6.1, but no experiment varies the quantity or diversity of paired training data. The TIPS ablation (Figure 9) demonstrates that the method works with a different pretrained backbone, but does not test data efficiency. The zero-shot cross-modal transfer experiment (Table 5) shows that the learned alignment generalizes to unseen modalities (NOCS maps), which is promising, but this is after training on six datasets with paired structural modalities — we do not know whether the same generalization would emerge from fewer or less diverse paired scenes.

Mitigation status. Not addressed. The paper makes no claim about data efficiency and does not frame the paired-data requirement as a limitation. The authors do not discuss the cost of acquiring or generating paired RGB-D-segmentation data, nor do they suggest strategies for reducing this requirement (e.g., semi-supervised alignment with unpaired RGB images, or leveraging synthetic data more aggressively). A practitioner reading the paper must independently assess whether sufficient paired data exists for their domain.

Hard Problems Requiring Fine-Grained Photometric Discrimination Show Regressions

The assumption or constraint. The method optimizes for cross-modal alignment by pulling embeddings of different modalities together in feature space while anchoring to the frozen teacher's outputs. The implicit assumption is that the structural information in depth and segmentation maps is complementary to RGB features and that aligning these modalities will enrich the representation for all downstream tasks. Section 4.2 states that "enforcing alignment between RGB, depth, and segmentation maps does not degrade the high-level semantic understanding required for segmentation tasks; in fact, the multimodal regularization appears to offer slight benefits in generalization."

The consequence. This claim does not hold universally. The k-NN transfer results (Table 4) reveal that the Omnivorous encoder degrades performance on fine-grained recognition tasks that rely on photometric detail: iNaturalist (species classification) drops from 78.53% to 77.49% (-1.04 percentage points), and Google Landmarks v2 (landmark instance recognition) drops from 51.90% to 50.13% (-1.77 pp). These regressions are not marginal — for Landmarks v2, the 1.77 pp decrease represents a meaningful degradation in instance-level discrimination. The authors acknowledge this obliquely: "we note slight regressions on fine-grained datasets like iNaturalist and Google Landmarks v2. This could be explained by our training mix, which includes a significant amount of simulated multi-object data" (Section 4.2).

The deeper issue is that alignment with structural modalities appears to smooth over precisely the photometric features that fine-grained tasks depend on. iNaturalist classification requires distinguishing nearly identical species based on subtle color and texture differences (e.g., two similar-looking bird species differentiated by slight plumage variations). Aligning RGB features with depth and segmentation — which are inherently textureless — may encourage the encoder to downweight photometric details in favor of geometric structure, harming tasks where those details are discriminative. The anchoring loss ($\mathcal{L}_{\text{anchor}}$) is designed to prevent this drift, but the regressions indicate it does not fully preserve the fine-grained photometric sensitivity of the original DINOv2 features.

What evidence exists in the paper. Table 4 directly shows the regressions. The classification linear-probe results (Table 3) show large improvements on ImageNet (+3.4 pp), but ImageNet is a coarse-grained classification benchmark (1,000 object categories with significant visual diversity within each) where geometric structure may be more informative than fine photometric detail. The contrast between ImageNet gains and iNaturalist/GLDv2 regressions reveals a task-dependent trade-off: alignment helps coarse-grained, structure-dependent tasks but can harm fine-grained, texture-dependent tasks. The paper does not characterize this trade-off beyond noting the regressions exist.

Mitigation status. Partially acknowledged but not addressed. The authors attribute the regressions to training data composition ("our training mix includes a significant amount of simulated multi-object data") rather than to the alignment objective itself. This leaves open the question of whether training on different data (e.g., datasets with more natural fine-grained categories) would mitigate the regression, or whether the alignment objective inherently downweights photometric detail regardless of training data. No experiment tests different training mixtures or measures how photometric feature preservation varies with $\lambda_{\text{anchor}}$.

The Method Is Validated Only on a Single Model Family at a Single Scale

The assumption or constraint. All experiments in the paper use DINOv2 ViT-B/14 (173M parameters, 12 blocks) as the teacher and backbone. The TIPS ablation (Figure 9) replaces DINOv2 with another ViT-B/14 model, but this is still the same architecture class (Vision Transformer, Base size) from a closely related pretraining lineage. The paper states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is made in the context of the Introduction and is not empirically supported for vision encoders more broadly.

The consequence. The findings may not generalize to other model families, scales, or pretraining objectives. Several aspects of the method could be architecture-dependent:

• The freeze/fine-tune split (L=8 out of 12 blocks) is tuned for ViT-B. Larger ViTs (ViT-L with 24 blocks, ViT-G with 40 blocks) have different ratios of low-level to high-level features. Freezing 8 blocks in a ViT-G (20% of layers) would be very different from freezing 8 blocks in ViT-B (67% of layers). The optimal L likely scales with model depth, but no guidance is provided.
• The 4-block trainable capacity might be insufficient for larger models to learn alignment across more diverse modalities or domains, or might be excessive for smaller models, leading to overfitting.
• Convolutional architectures (ConvNeXt, ResNet) lack the clear layer-wise hierarchy of ViTs and might require different freeze/fine-tune strategies. The method's parameter-efficient design relies on the transformer block structure.
• CLIP-pretrained or supervised encoders have different feature geometries than DINOv2 (self-supervised). The anchoring loss assumes the teacher's output is a good representation to preserve; if the teacher already has better native cross-modal alignment (as TIPS appears to; Figure 9a shows higher baseline alignment than DINOv2), the optimal $\lambda_{\text{anchor}}$ and the necessity of alignment training might change.

What evidence exists in the paper. None beyond ViT-B. The TIPS experiment (Figure 9) confirms that the method works on a second ViT-B model with a different pretraining objective (text-image contrastive), which is encouraging but still within the same architecture class and parameter scale. There are no experiments with ViT-S, ViT-L, ViT-G, or non-ViT architectures. The adapter-on-top ablation (Table 11) tests a different parameterization within the same ViT-B, not a different backbone.

Mitigation status. Not addressed. The paper does not claim to study scaling behavior or architectural generality, but it also does not caution readers about the untested nature of these dimensions. The title and framing ("An Omnivorous Vision Encoder") imply a general framework, but the experimental validation is narrow. A practitioner using a different backbone (e.g., a ConvNeXt-based model for mobile deployment, or a ViT-L for higher accuracy) would need to independently validate that the method transfers.

Sequential Dependency on DINOv2's Specific Pretraining Pipeline Creates a Reproducibility and Deployment Barrier

The assumption or constraint. The method distills from a frozen DINOv2 teacher and freezes the first 8 blocks of the DINOv2 backbone. This assumes access to a DINOv2 model with the specific architecture (ViT-B/14), patch size (14), position encoding shape ($37 \times 37$), and pretraining procedure (self-supervised learning on a curated dataset of 142M images with high-resolution fine-tuning; Oquab et al., 2023). The paper acknowledges that "DINOv2 undergoes high-resolution fine-tuning as a final training step. It is unclear whether this step would be required after training Omnivorous DINO too" (Section 5, Limitations).

The consequence. Reproducing or adapting this method requires not just any pretrained ViT, but specifically DINOv2 with its particular pretraining recipe. If DINOv2 is updated (new versions, different architectures), deprecated, or unavailable due to licensing changes, the method's effectiveness is tied to that specific checkpoint. More practically, a practitioner with their own pretrained encoder (e.g., a company-proprietary model trained on internal data) cannot simply apply the Omnivorous recipe — they must first verify that their encoder exhibits the same cross-modal misalignment phenomenon (Figure 1), that its layer-wise feature hierarchy supports a freeze/fine-tune split analogous to DINOv2's 8/4 division, and that the anchoring loss provides sufficient stabilization. None of these properties are guaranteed for arbitrary pretrained encoders.

Furthermore, the default hyperparameters ($\lambda_{\text{anchor}} = 10$, $L = 8$, $\alpha_{\text{max}} = 0.5$, 64 colorization bins, kernel size 5) were tuned jointly with DINOv2 ViT-B/14. There is no evidence that these values transfer to other settings. The TIPS experiment (Figure 9) shows that the alignment-discriminability trade-off curve exists for TIPS, but does not specify whether $\lambda_{\text{anchor}} = 10$ remains optimal — the paper only states that "$\lambda_{\text{anchor}} = 100$ nevertheless does exceed the depth and segmentation performance of higher values of $\lambda_{\text{anchor}}$," which is a qualitative observation, not a hyperparameter optimization.

What evidence exists in the paper. The TIPS ablation (Section 7.3.1, Figure 9) provides partial evidence for generality across pretraining objectives within the ViT-B architecture class. The adapter-on-top vs. fine-tuning ablation (Table 11) shows robustness to the student parameterization. However, neither experiment addresses the core dependency: the method has only been validated starting from DINOv2 or TIPS, both ViT-B/14 models from the same research group with similar pretraining philosophies. The paper does not evaluate on publicly available alternatives like CLIP ViT-B/16 (OpenAI), EVA-02, or MAE-pretrained ViTs, which would be the minimum evidence for broader applicability.

The high-resolution fine-tuning concern (Section 5, Limitations) is explicitly flagged by the authors but left unresolved. DINOv2's high-resolution fine-tuning stage adapts the model to process images at larger resolutions (up to 518×518 or higher) with modified position encodings. Since Omnivorous training is done at $224 \times 224$, it is unclear whether the alignment learned at this resolution would degrade when the adapter is applied to higher-resolution inputs, or whether an additional high-resolution fine-tuning step (matching DINOv2's original protocol) would be needed to recover performance on dense prediction tasks that benefit from larger inputs.

Mitigation status. Partially acknowledged. The paper mentions the high-resolution fine-tuning concern in the Limitations section, but does not propose a solution or conduct experiments to characterize the resolution sensitivity. The dependency on DINOv2 specifically is not framed as a limitation, even though the paper's title and positioning ("Omnivorous Vision Encoder") imply a general framework. Future work suggestions (Section 5) focus on pre-training an omnivorous encoder from scratch rather than post-hoc alignment, implicitly acknowledging that the frozen-backbone approach is limited by the base model's properties, but this is presented as a forward-looking research direction rather than a warning about the current method's assumptions.

The Method Cannot Create Cross-Modal Alignment Where the Teacher Lacks Any Cross-Modal Signal

The assumption or constraint. The anchoring loss (Equation 3) distills from the frozen teacher's output $h^*_m$ for each modality. This assumes that the teacher can produce a meaningful representation of non-RGB inputs — depth maps and segmentation maps processed through natural colorization — even though the teacher was trained exclusively on RGB images. The paper relies on the fact that colorized depth and segmentation maps look similar enough to RGB images that the teacher's RGB-trained filters extract useful features from them. The PCA visualizations (Figure 5, "Frozen features") show that DINOv2 does produce structured feature maps for depth and segmentation inputs, even though they occupy different subspaces from RGB features.

The consequence. If the teacher produces degenerate, uninformative, or extremely noisy features for a particular modality, the anchoring loss provides no useful signal — it distills noise. Worse, the alignment loss might pull the student's representations for that modality toward a noisy teacher anchor, degrading overall representation quality. This failure mode is acute for modalities that are far from RGB in appearance, even after colorization. The NOCS experiment (Table 5) hints at this boundary: the Omnivorous encoder achieves only $\delta_1 = 0.029$ on NOCS inputs (DPT head), only marginally better than DINOv2's $\delta_1 = 0.014$. NOCS maps represent 3D coordinates as RGB channels — a fundamentally different signal than natural images, even after aggressive colorization. The teacher (trained on natural RGB images) likely produces poorly structured features for NOCS inputs, limiting how much the student can learn to align them.

More broadly, this limitation defines a capability ceiling: the Omnivorous encoder can only align modalities that the frozen teacher can already partially represent. For modalities where the teacher fails entirely (e.g., sonar, radar, event camera data, non-visual sensor modalities), the anchoring loss provides no foundation to build on, and the alignment loss alone (without anchoring) leads to representational collapse (as shown by the $\lambda_{\text{anchor}} = 0$ datapoint in Figure 4b). This means the method cannot be extended to radically different sensing modalities without changing the teacher — which would require a multimodal foundation model that already embeds those modalities, defeating the purpose of post-hoc alignment.

What evidence exists in the paper. The NOCS results (Table 5) provide indirect evidence. The diagnostic metrics (Table 8) show that DINOv2 does have non-trivial, albeit misaligned, cross-modal similarities (e.g., <R,D> = 0.285 on ScanNet), confirming that the teacher extracts some structure from non-RGB inputs. However, no experiment deliberately tests the method with a modality where the teacher's features are known to be poor, to establish the failure boundary. The paper does not measure the quality of the teacher's features on depth, segmentation, or NOCS inputs beyond the cosine similarity in Table 8 (which captures alignment, not semantic quality). We do not know whether the teacher's depth features, for instance, contain useful geometric information independent of their alignment with RGB — the teacher could be producing features that are internally consistent for depth inputs but simply occupy a different subspace, or it could be producing features that are essentially random with respect to scene geometry.

Mitigation status. Not addressed. The paper presents the method as post-hoc alignment built on any frozen backbone, but does not characterize the boundary conditions under which the frozen backbone provides an adequate foundation. The authors do not discuss what properties the teacher must satisfy (minimum cross-modal similarity? minimum within-modality discriminability?) for the method to succeed. A practitioner applying this to a novel modality pair must independently verify that their frozen encoder produces sufficiently structured features for both modalities — a diagnostic that the paper does not provide tools or thresholds for.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological reframing rather than a paradigm shift. It does not introduce a new architecture, a new pretraining objective, or a new theoretical framework for multimodal learning. Instead, it demonstrates that a specific combination of existing techniques—parameter-efficient fine-tuning, symmetric InfoNCE alignment, and distillation-based anti-collapse regularization—can achieve something that the field had implicitly assumed required from-scratch multimodal pretraining: a single encoder that maps diverse visual modalities to a shared, semantically rich embedding space. This reframing matters because it decouples the investment in unimodal foundation models from the problem of cross-modal alignment. The enormous cost of training DINOv2 (hundreds of GPU-days on a curated 142M-image dataset) need not be repeated every time a new modality needs to be integrated; instead, a 1-hour fine-tuning step on modest hardware suffices to retrofit alignment onto the existing backbone.

The practical consequence is a shift in how organizations should think about their model development pipeline. Rather than the prevailing implicit model of "pretrain a multimodal encoder from scratch if you need cross-modal capabilities," the paper's results suggest a two-stage strategy: first, invest heavily in a high-quality unimodal encoder (or adopt an existing one like DINOv2); second, perform lightweight, parameter-efficient alignment to add modality-agnostic behavior. This is cheaper, faster, and—crucially—does not risk degrading the unimodal performance that the original encoder was designed for. The evidence for this last point is one of the paper's most important contributions: the Omnivorous encoder not only preserves but improves DINOv2's ImageNet linear-probe accuracy from 80.4% to 83.8% (Table 3). The alignment signal acts as a beneficial regularizer, not a compromise.

The paper reconciles a tension in the multimodal learning literature that had not been clearly articulated before. On one side, unified co-training approaches (Omnivore, ImageBind, Unified-IO) argued that multimodal encoders must be trained from scratch with mixed-modality data to develop genuine cross-modal invariances. On the other side, cross-modal distillation methods (SOCKET, CMKD variants) and adapter-based approaches (ViT-Adapter, MA-AVT) showed that post-hoc alignment was possible for specific task-modality pairs, but typically required task-specific architectures or training procedures. The Omnivorous encoder shows that general, task-agnostic cross-modal alignment can be achieved post-hoc with a single shared adapter and a simple dual-objective loss, bridging the gap between the expensive generality of from-scratch co-training and the narrow specificity of task-level distillation. This positions the method in a sweet spot: it inherits the rich unimodal representations of the original foundation model while adding the cross-modal transfer capabilities that co-training aims for, all at a fraction of the computational cost.

The identification of the anchoring loss as an explicit control mechanism for the alignment-discriminability trade-off (Figure 4a) is a diagnostic contribution that changes how researchers should think about multi-view contrastive learning with imbalanced modalities. Before this work, preventing collapse in InfoNCE-based alignment was typically handled by engineering the negative sampling strategy (e.g., large batch sizes, memory banks, hard negative mining). The paper shows that these strategies may be insufficient when modalities are severely imbalanced, because the effective repulsive force from negatives is modality-dependent. The anchoring loss provides an independent, per-sample regularization that directly penalizes drift from a reference representation, making the trade-off between alignment quality and semantic preservation explicitly tunable through a single hyperparameter ($\lambda_{\text{anchor}}$). This is not just a training trick; it is a conceptual insight about the geometry of multi-view contrastive learning that could inform the design of future alignment methods across domains (vision-language, audio-visual, cross-lingual) where view imbalance is the norm rather than the exception.

The data-centric insight about shortcut prevention—that rendering non-RGB modalities with natural color palettes derived from the paired RGB image can force the encoder to learn structural rather than statistical correspondences—is a perspective that may influence how practitioners approach cross-modal alignment problems more broadly. Rather than treating modality-specific input statistics as immutable properties that the model must learn invariance to, the paper demonstrates that actively manipulating input representations to remove shortcut signals can simplify the learning problem. This is analogous to the role of data augmentation in standard self-supervised learning (where color jitter and cropping remove trivial invariances so the model learns meaningful ones), but applied at the modality level rather than the instance level. Whether this insight generalizes to non-visual modalities (e.g., converting spectrograms to match natural image statistics for audio-visual alignment) is an open question that the paper enables but does not answer.

The work makes two research directions less attractive: first, the claim that from-scratch multimodal pretraining is necessary for cross-modal transfer is weakened—the paper shows that post-hoc alignment can achieve meaningful cross-modal generalization (zero-shot depth prediction from segmentation inputs, RMSE 0.532 vs. random-guessing DINOv2's 1.536 in Table 5) without multimodal pretraining. Second, the pursuit of increasingly complex auxiliary losses or modality-specific architectural modules for cross-modal alignment is challenged by the paper's demonstration that a simple symmetric InfoNCE + anchoring loss, applied to a shared set of fine-tuned upper layers, suffices for substantial alignment gains. The method's simplicity and training efficiency (1 hour 14 minutes; Table 7) sets a high bar for more complex alternatives to justify their additional complexity.

Follow-Up Research This Work Enables

The missing natural colorization ablation: isolating the contribution of shortcut prevention. The paper claims that standard grayscale/jet colormaps enable shortcut learning, but never experimentally validates this claim by training an Omnivorous encoder with standard colormaps and comparing performance. A controlled ablation would train three variants: (1) the full Omnivorous method with natural colorization, (2) Omnivorous training with standard grayscale depth maps and fixed-colormap segmentation masks, and (3) Omnivorous training with RGB-only color jitter applied to depth and segmentation maps (a simpler shortcut-removal strategy). If variant (2) achieves comparable cross-modal alignment but inferior downstream task performance (suggesting the model learned modality-specific colormap projections rather than structural alignment), the natural colorization hypothesis is validated. If variant (2) performs comparably to variant (1) on all metrics, the paper's most complex data preprocessing step is unnecessary—a finding that would simplify adoption and shift focus to the loss design and anchoring mechanism. If variant (3) performs well, a simpler, more general colorization strategy exists. This experiment is low-cost (the training pipeline already exists) and would substantially clarify the method's active ingredients.

Scaling behavior with model size (ViT-S, ViT-L, ViT-G) and the optimal freeze/fine-tune ratio. All experiments in the paper use ViT-B/14 with the first 8 of 12 blocks frozen. The optimal L likely depends on model depth: the ratio of frozen to trainable layers determines how much pretrained knowledge is preserved vs. how much capacity is available for alignment learning. A scaling study would apply the Omnivorous training procedure to DINOv2 ViT-S (12 blocks, freeze 4/6/8), ViT-L (24 blocks, freeze 16/18/20), and ViT-G (40 blocks, freeze 28/32/36), measuring cross-modal retrieval (ScanNet R@1) and downstream task performance (ImageNet linear probe, NYUv2 depth) as a function of the freeze ratio. The key question is whether the alignment procedure's effectiveness saturates or degrades at larger scales—do bigger models benefit more (more capacity to reorganize features) or less (harder to overcome strong pretrained priors) from alignment training? The answer would determine whether the method is a practical tool for state-of-the-art large vision encoders or primarily useful at the Base scale. The computational cost is tractable: the paper's training time scales roughly linearly with model size for fixed-step fine-tuning.

Generalization beyond the DINOv2 training lineage: CLIP, EVA, MAE, and supervised backbones. The method's dependence on DINOv2's specific pretraining properties (self-supervised ViT with particular feature geometry) is untested. A robustness study would apply the identical Omnivorous training recipe (same losses, λ_anchor = 10, L = 8 for 12-block models, same datasets, same 20k steps) to a diverse set of publicly available ViT-B backbones: OpenAI CLIP ViT-B/16 (contrastive language-image pretraining), EVA-02 ViT-B/14 (masked image modeling + CLIP), MAE ViT-B/16 (reconstruction pretraining), and a supervised ImageNet-21k ViT-B/16. The primary metrics would be: (1) does the baseline cross-modal misalignment (Figure 1) exist for all backbones, or is it specific to DINOv2? (2) does the anchoring loss prevent collapse equally well across pretraining objectives, or does its effectiveness depend on the teacher's feature geometry? (3) do the downstream task improvements (Table 3's +3.4pp ImageNet gain) replicate, or are they specific to DINOv2's self-supervised features? If the method transfers broadly, it becomes a general-purpose tool; if it fails on certain backbones, the paper's title and framing ("An Omnivorous Vision Encoder") needs qualification. The negative result—identifying which backbones resist alignment—would be as informative as the positive transfer.

Data efficiency: how many paired scenes are needed for effective alignment? The paper uses six datasets with potentially millions of paired RGB-D-Seg scenes, but never varies the quantity or diversity of training data. A data-scaling experiment would train Omnivorous encoders on random subsets of the training data at sizes [100, 500, 1000, 5000, 10000, 50000, full] scenes, measuring cross-modal retrieval (ScanNet R@1), depth prediction (NYUv2 δ₁ with linear head), and zero-shot cross-modal transfer (segmentation → depth on PACE). The key practical question: can a practitioner with a small paired dataset (e.g., 1000 RGB-D scenes from their domain) achieve meaningful alignment, or is the method data-hungry despite its parameter efficiency? The paper's zero-shot NOCS result (Table 5) suggests that even limited alignment training produces generalizable invariances, but the scaling curve is unknown. A finding that 1000 scenes achieves 80% of the full-data alignment performance would dramatically lower the barrier to adoption; a finding that 50k scenes are needed would restrict the method to organizations with large paired datasets.

Combining Omnivorous alignment with the revision model from other test-time compute strategies. The paper studies post-hoc alignment as a standalone fine-tuning procedure, but does not explore whether the aligned representations enable new forms of test-time adaptation or iterative refinement. A natural extension would be to use the Omnivorous encoder as the backbone for a system that, at inference time, receives an input in one modality and iteratively "refines" its representation by imagining the scene in other modalities—essentially using the shared embedding space to perform cross-modal mental rotation. Concretely: given an RGB image, encode it with the Omnivorous encoder, use a generative decoder to produce a depth map from the shared embedding, re-encode that depth map, and check whether the depth-derived embedding is consistent with the RGB-derived embedding (violations indicate model uncertainty or out-of-distribution inputs). This connects to the broader literature on test-time adaptation and consistency regularization, but is made newly tractable by the Omnivorous encoder's ability to map different modalities to the same space without paired data at test time. The paper's cross-modal retrieval results (Table 1) show this is feasible in principle (depth retrieves the correct RGB scene), but an active refinement loop has not been tested.

Practical Applications and Downstream Use Cases

Robotic perception with sensor-agnostic task heads. A robot deployed in a building might encounter RGB cameras in well-lit areas, depth sensors in low-light corridors, and segmentation masks from a lightweight semantic segmentation model running on edge hardware. Currently, each sensor modality requires its own task-specific perception head (e.g., a grasping model trained on RGB, a separate one for depth). With an Omnivorous encoder, a single downstream model—say, a grasp success predictor trained only on RGB features from NYUv2—can be deployed on depth inputs (achieving δ₁ = 0.184, RMSE = 0.532 for depth prediction from an RGB-trained head; Table 5) or segmentation inputs without retraining. This reduces the engineering burden of maintaining per-modality models and enables graceful degradation when sensors fail or switch (e.g., switching from RGB to depth in darkness). The 10× improvement in cross-modal retrieval on ScanNet (R@1 from 4.6% to 46.1%; Table 1) directly translates to better retrieval of relevant training examples or object templates across modalities—a robot can find the closest matching scene in its database regardless of which sensor captured the query.

Efficient fine-tuning of deployed vision systems for new modalities. A company with a production DINOv2-based image classification or object detection pipeline receives a new camera type (e.g., a thermal infrared sensor) and needs to adapt their models. Rather than collecting a large labeled dataset for the new modality and retraining all downstream heads from scratch, they can: (1) collect a modest set of paired RGB-thermal images (hundreds to thousands of scenes, not millions), (2) run Omnivorous fine-tuning for ~1 hour to align thermal features with the existing RGB feature space, and (3) deploy existing RGB-trained task heads directly on thermal inputs. The paper's results with NOCS maps (Table 5) demonstrate that alignment generalizes partially even to never-seen modalities (RMSE 0.822 vs. DINOv2's 0.979), suggesting that thermal-to-RGB alignment—an easier problem since thermal images share structural similarity with grayscale images—would likely succeed. The key practical metric missing from the paper is the minimum number of paired scenes needed (see future work above), but the training efficiency (1 hour 14 minutes; Table 7) makes rapid iteration on this hyperparameter feasible.

Multi-view 3D reconstruction with heterogeneous input sources. Structure-from-motion and neural radiance field (NeRF) pipelines typically assume consistent input modalities (all RGB images). An Omnivorous encoder would enable these pipelines to ingest mixed-modality image collections—some RGB photos, some depth maps, some segmentation masks—and map them all to a shared feature space before triangulation or volumetric rendering. The 3D correspondence results (Table 10, PCK@0 improves from ~28.6% to 29.0% on NAVI) suggest that the aligned features preserve spatial correspondence at the patch level. The depth estimation results (Table 2, NYUv2 δ₁ = 0.948 with DPT head, matching DINOv2) confirm that the aligned features retain the geometric precision needed for 3D tasks. A practical system could use the Omnivorous encoder as the feature extractor for a DUSt3R or MASt3R-style stereo matching pipeline, training on RGB pairs but deploying on mixed RGB-depth pairs, leveraging depth maps as a direct geometric signal while using RGB for texture and appearance matching.

When to Prefer This Method

The paper implicitly defines a choice between scaling pretraining compute (training a larger or multimodal model from scratch) and post-hoc alignment (fine-tuning an existing unimodal encoder with the Omnivorous recipe). The following decision boundaries are derived from the paper's findings:

Prefer post-hoc Omnivorous alignment when:

• You already have a deployed unimodal encoder (DINOv2 or similar) and need to add cross-modal capabilities without risking regression on existing single-modality tasks—the anchoring loss preserves discriminability (k-NN parity at 81.97% vs. 81.94%; Table 4) while adding alignment.
• Your paired multimodal data is limited or expensive to collect—the method's parameter efficiency (only ~1/3 of ViT-B parameters updated, 20k steps; Table 7) suggests it may work with modest data, though the exact minimum is unknown (see future work above).
• Compute budget for training is constrained—1 hour 14 minutes on TPU v4 vs. the weeks or months required for from-scratch multimodal pretraining.
• The target modalities are structurally similar to RGB after appropriate colorization (depth, segmentation, surface normals, edge maps)—the method assumes the frozen teacher can extract meaningful features from colorized versions of these inputs, which holds when the modality captures scene geometry that correlates with RGB appearance.
• Your downstream tasks benefit from geometric regularization—the ImageNet linear-probe improvement (+3.4pp; Table 3) and depth/segmentation gains (Tables 2) suggest alignment enriches representations for structure-dependent tasks.

Prefer from-scratch multimodal pretraining (Omnivore, ImageBind, Unified-IO) when:

• The target modalities are fundamentally different from RGB in appearance and information content, such that even aggressive colorization cannot make them resemble natural images (e.g., sonar, radar, event camera data, audio spectrograms)—the frozen teacher will produce poor features, and the anchoring loss has nothing useful to distill.
• You need native multimodal understanding rather than cross-modal transfer—if the task requires joint reasoning over RGB and depth simultaneously (not transferring from one to another), a co-trained model may learn richer multimodal interactions than post-hoc alignment enables.
• The unimodal teacher's representations are known to be poor for the target modalities—the method's effectiveness is bounded by the teacher's ability to partially represent non-RGB inputs (the <R,D> similarity of 0.285 for DINOv2 on ScanNet; Table 8, provides a baseline to improve upon; if this were near zero, alignment would lack a foundation).
• Training data for all modalities is abundant and you can afford to train from scratch—the Omnivorous approach is a pragmatic compromise for data- or compute-constrained settings; it is not claimed to outperform a hypothetical model trained optimally on all modalities from the start (the paper never makes this comparison).

Prefer task-specific cross-modal distillation (SOCKET, CMKD variants) when:

• You only need transfer for a single task-modality pair (e.g., RGB → depth semantic segmentation) and have a task-specific student model architecture that can be optimized for that particular transfer—Omnivorous alignment produces a general-purpose encoder that may be overkill if only one transfer direction matters.
• You have no access to the RGB training data (source-free transfer)—SOCKET explicitly handles this regime, while Omnivorous training requires paired multimodal data during alignment training.