ArXiv: 2601.14256

🎯 Pitch

HUVR proves that a single ViT-based hyper-network—trained to output tiny 8–32 dimensional tokens that are genuinely decodable back to pixels—can beat DINOv3 at classification while rivaling a Stable Diffusion VAE on reconstruction, shattering the assumption that recognition and generation demand separate latents.

1. Executive Summary

This paper proposes HUVR (Hyper-network for Unified Visual Representation), a first-of-its-kind model that learns image representations simultaneously useful for recognition and generation by training a Vision Transformer as a hyper-network for implicit neural representation (INR). Unlike prior INR hyper-networks that discard output image tokens, HUVR introduces three key design innovations—using patch tokens directly as weight tokens, introducing a learnable global token that both serves as a classification token and modulates INR weights via matrix multiplication, and learning compressed TinToks (Tiny Tokens) through learned downsampling/upsampling layers—enabling the model to natively perform classification, semantic segmentation, depth estimation, and reconstruction from a single architecture. When paired with knowledge distillation from DINOv3, HUVR matches or exceeds DINOv3's ViT-B/16 performance (+0.4% ImageNet classification, +1.2 mIoU on ADE20K segmentation, +4.84 PSNR reconstruction), while its 32-dimensional TinToks—achieving up to 96× compression—offer +48% ImageNet classification over a DINOv3 PCA baseline at 8 dimensions and +1.26 PSNR over the Stable Diffusion VAE at equal embedding size, establishing that compressed INR-based representations can unify recognition and generation but that current generative quality still trails dedicated autoencoder approaches.

2. Context and Motivation

The Core Problem: Recognition and Generation Use Separate, Incompatible Representations

The fundamental split this paper tackles is that modern computer vision has developed along two largely independent tracks that produce representations with fundamentally different properties. Recognition-focused models (trained via contrastive learning, self-distillation, or masked image modeling) learn embeddings that excel at high-level semantic tasks—image classification, object detection, semantic segmentation—but these embeddings cannot reconstruct images. They discard pixel-level information as a byproduct of their training objectives, which explicitly encourage invariance to low-level details like texture, color, and exact spatial position. A DINOv3 embedding can tell you with high confidence that an image contains a "golden retriever playing in a park," but you cannot recover the park's layout, the dog's pose, or the lighting conditions from that embedding.

Conversely, generative models (VAEs, diffusion model autoencoders, GAN encoders) learn latent spaces optimized for pixel-wise reconstruction fidelity, measured by metrics like PSNR, SSIM, and LPIPS. These latents enable high-quality image synthesis but typically perform poorly on semantic recognition tasks. The Stable Diffusion VAE latent space preserves sufficient detail to reconstruct a 512×512 image with ~25 PSNR, but linear probing classification accuracy on those same latents is dramatically lower than what dedicated recognition models achieve.

This schism means that any practical system requiring both capabilities—for instance, a visual assistant that must classify objects and generate visual explanations, or a compression pipeline that needs to support both storage-efficient reconstruction and content-based retrieval—must either maintain two separate models or accept degraded performance on one axis. The paper frames this concretely in Section 1:

"A natively unified encoder's features should have good high level (image classification), mid level (semantic segmentation), low level (depth estimation), and pixel level (reconstruction) information out-of-the-box."

The phrase "out-of-the-box" is critical: some prior works attempt to bolt recognition capabilities onto generative models post-hoc, but these are retrofits, not native solutions.

Why This Problem Matters: Practical Deployment and Representation Learning Theory

The motivation is both practical and theoretical.

Practical motivation: embedding size compression. The paper introduces an additional axis of unification that sharpens the practical stakes. Different downstream tasks have wildly different computational budgets, and embedding size is a primary cost driver for large-scale retrieval, storage, and transmission. As the authors note in Section 1:

"Tasks like retrieval become very difficult as the amount of data increases at scale, and reducing the embedding size introduces massive savings."

A unified representation that works at multiple compression levels—from standard ViT token dimensions (768 for ViT-B) down to tiny vectors (8–32 dimensions)—would enable a single model to serve both high-accuracy recognition (using full-size embeddings) and efficient large-scale retrieval or storage (using compressed embeddings), all while retaining reconstruction capability. This is not merely about convenience; it has direct implications for the economics of deploying vision systems at scale.

Theoretical motivation: what does a "unified" representation require? The problem touches a deeper question in representation learning: can a single representational format capture the full spectrum from pixel-level to semantic information? The paper hypothesizes that implicit neural representation hyper-networks are structurally suited to this because they perform compression along multiple axes: first by encoding an image into latents, and second by learning a shared "base" INR that represents the dataset's common structure. The authors reference classical ideas about compression and intelligence:

"We hypothesize this is helpful for learning high quality embeddings not only at the pixel-level, but also for low-, middle-, and high-level image information [34, 54, 84, 90]."

The citations to Schmidhuber (2006), Gregor et al. (2016), Solomonoff (1964), and Lee et al. (2021) invoke the long-standing intuition that good compression requires understanding—that representations which can reconstruct their inputs must capture the underlying structure, and this structure might naturally span semantic levels.

Prior Approaches and Where They Fall Short

The paper identifies four families of prior work, each with specific limitations that HUVR aims to address.

Post-Hoc Unification of Pre-Trained Models

Several recent works take pre-trained generative models (typically diffusion models) and adapt them for recognition tasks after the fact. The paper cites works that "attempt to unify these recognition-focused and generative-focused models post-hoc [25, 57, 68, 117]" (Section 1). These approaches—for example, using the internal representations of a diffusion model's denoising U-Net as features for classification or segmentation—demonstrate that generative models contain semantically useful information, but they require retrofitting: training additional linear probes, designing extraction protocols, or fine-tuning. The representations were not optimized for recognition during pre-training, so performance typically lags behind purpose-built recognition models. The paper describes these works as pointing to a "promising synergy" but not achieving native unification.

Joint Training for Both Objectives

Some methods train models with objectives that combine recognition and generation from the start. The paper references works on adversarial representation learning (Donahue et al., 2016; Dumoulin et al., 2016; Donahue and Simonyan, 2019), InfoGAN (Chen et al., 2016), and MAGE (Li et al., 2022). These approaches typically use adversarial training or masked generative objectives alongside discriminative losses. The limitation is that these methods have not achieved state-of-the-art results on both axes simultaneously—they tend to make a trade-off, performing well on one family of tasks at the expense of the other, or performing adequately on both without excelling at either.

Pure INR Hyper-Networks for Reconstruction

The paper builds directly on the INR hyper-network literature, particularly TransINR (Chen and Wang, 2022) and its successors (IPC by Kim et al., 2022; LA-IPC by Lee et al., 2023; ANR by Zhang et al., 2024). These methods train a transformer encoder to predict modulation matrices that adapt a shared base INR to reconstruct specific input images. They achieve impressive reconstruction quality and avoid the per-sample training cost of standard INRs. However, these prior works have several critical shortcomings that HUVR directly addresses:

  1. Wasted output tokens. Prior hyper-networks introduce learnable "weight tokens" as inputs to the transformer encoder, then discard the output image/patch tokens entirely—using only the output weight tokens to produce the INR modulation. The paper states this explicitly (Section 3.1): "Existing INR hyper-networks discard the output image tokens, using them neither to calculate loss nor perform inference." These discarded tokens represent wasted capacity and compute.

  2. No spatial correspondence for dense tasks. Because the weight tokens lack a clear correlation with spatial locations in the input image, dense prediction tasks like semantic segmentation are "very challenging." The learned weight tokens mix information across the entire image, so recovering per-patch semantics from them is non-trivial.

  3. No classification token. The original hyper-network formulation lacks a [CLS] token or equivalent global summary, making it ill-suited for image-level classification without architectural modification.

  4. Poor semantic representations. Most critically, INR hyper-networks trained purely for reconstruction do not learn high-level semantics. The paper acknowledges this directly: "INR hyper-networks do not natively learn good high level semantics" (Section 1). The reconstruction objective alone is insufficient to produce features that work for ImageNet classification or semantic segmentation.

  5. No support for compressed representations. Prior hyper-networks operate at a single token dimension (typically the ViT's native dimension). They provide no mechanism for producing variable-size embeddings that could serve different downstream budgets.

Contrastive and Self-Distillation Methods (Recognition-Only)

The dominant paradigm for image representation learning—represented by DINOv2, DINOv3 (Siméoni et al., 2025), SigLIP 2 (Tschannen et al., 2025), and C-RADIOv3 (Heinrich et al., 2024)—achieves outstanding recognition performance across classification, segmentation, and depth estimation. These methods use variations of self-distillation (student-teacher consistency with different augmentations) or image-text contrastive learning. However, they are fundamentally incapable of reconstruction: their training objectives actively discard pixel-level information, and their architectures provide no decoding pathway back to image space. The paper demonstrates this concretely in Table 1, where even after training a separate decoder on frozen DINOv3 PCA-compressed features, reconstruction PSNR is dramatically lower than HUVR's (e.g., 16.75 PSNR for DINOv3 PCA at 16 dimensions vs. 26.25 for HUVR at the same size). This is not a failure of DINOv3—it was never designed for reconstruction—but it illustrates the gap that a unified model could fill.

How the Paper Positions Itself

HUVR positions itself as filling the specific gap between these families. Its design philosophy, articulated in Section 1, is that INR hyper-networks provide the right structural foundation for unification because they natively handle the reconstruction axis, but they need targeted architectural modifications and training procedures to add the recognition axis.

The positioning can be understood as a three-part argument:

Structurally, INR hyper-networks can be refactored to support recognition. The paper argues that the wasted-token problem is not fundamental but an artifact of prior designs. By making patch tokens serve double-duty as the weight tokens (Key Innovation #1), and introducing a global token that both summarizes the image for classification and participates in INR modulation via matrix multiplication (Key Innovation #2), the architecture becomes capable of supporting both task families without fundamental tension.

Semantically, knowledge distillation bridges the gap. The paper does not claim that reconstruction training alone yields good recognition. Instead, it explicitly addresses the semantic deficiency through knowledge distillation from a pre-trained recognition model (DINOv3). The distillation loss (Equation 1) aligns HUVR's encoder and decoder outputs with DINOv3's feature space, effectively importing semantic structure that the reconstruction objective alone would not produce. This is Key Innovation #4: "Distillation for Unified Representation."

Practically, compressed representations (TinToks) expand the unification to the resource axis. By introducing learnable downsampling and upsampling layers between the encoder backbone and the INR prediction head (Key Innovation #3), HUVR can produce embeddings at multiple dimensionalities from a single model. The compressed TinToks are not just smaller versions of the standard tokens; they are explicitly trained to retain both semantic and reconstruction information through their position in the architecture (they feed into both the distillation pathway and the INR modulation pathway).

The paper also makes clear what it is not claiming. It acknowledges that the generative quality of HUVR's latents does not match dedicated autoencoders for diffusion (Table 2: FID of 24.72 vs. 23.05 for SD VAE), and that its pre-training scale is smaller than state-of-the-art recognition models (Section 4.8: "We do not train with specialized curated data" like DINOv3, and we train on less data than SigLIP 2). The contribution is not "we beat everyone at everything" but rather "we demonstrate that native unification is possible and that the INR hyper-network framework, properly modified, is the right vehicle for it." This modesty strengthens the paper's credibility: the authors are transparent about where their method excels (unified representation quality, especially under compression) and where it falls short (pure generative modeling relative to specialized autoencoders), which frames the work as opening a research direction rather than closing one.

3. Technical Approach

3.1 Reader Orientation

HUVR is an INR hyper-network—a Vision Transformer that takes an image as input and produces as output not a single embedding vector but the weights of a neural network (an implicit neural representation, or INR) that, when evaluated at pixel coordinates, reconstructs the input image. The core problem it solves is that current vision models are specialized: recognition models produce embeddings useful for classification and segmentation but cannot reconstruct images, while generative models reconstruct pixels but produce embeddings with poor semantic quality. The "shape" of HUVR's solution is to refactor the standard INR hyper-network architecture so that the same tokens that drive reconstruction also serve as powerful recognition features, and to use knowledge distillation from a pre-trained recognition model to inject high-level semantics that the reconstruction objective alone does not provide.

3.2 Big-Picture Architecture (Diagram in Words)

The HUVR system has five major components, which process information in sequence:

  1. Transformer Encoder (ViT backbone) — takes an image split into patches as input and produces standard-dimensionality patch tokens and a learnable global ([CLS]) token. This is the main representational backbone.

  2. Compression Module (TinToks) — a learned downsampling linear layer that projects each token from the encoder dimension d_ViT to a much smaller dimension d_t, producing compressed representations called TinToks (Tiny Tokens). These are the compact embeddings used for efficient downstream tasks.

  3. Transformer Decoder — a shallow transformer that processes the TinToks, allowing information mixing among compressed patches before they are used for reconstruction. It upsamples tokens back to a decoding dimension d_dec.

  4. INR Modulation Head — projects decoder output tokens (patch tokens and global token) to form modulation matrices that adapt a shared, learnable base INR θ_b into an image-specific INR θ'. The modulation uses the global token projected to one dimension and patch tokens projected to another, multiplied to form a matrix, which is then elementwise-multiplied with the base INR's second-layer weights.

  5. Patch-wise INR (the output network) — for each image patch, a small MLP with modulated weights takes strided (x, y) coordinates as input and produces (r, g, b) color values as output, reconstructing that patch. All patches are concatenated to reconstruct the full image.

Information flows as follows: an image enters the ViT encoder → encoder produces standard patch tokens and a global token → tokens are linearly compressed to TinToks → TinToks are processed by a transformer decoder → decoder outputs are projected to form patch-specific and global modulation vectors → the outer product of global and patch vectors forms a modulation matrix → the matrix elementwise-multiplies the base INR's second layer → the modulated INR reconstructs each patch from coordinates → patches are stitched together into the reconstructed image. Meanwhile, the encoder and decoder outputs are also fed into distillation projection layers that align them with a frozen teacher model's (DINOv3's) feature space, providing the semantic training signal.

3.3 Roadmap for the Deep Dive

  • First, I will explain what an implicit neural representation (INR) is and why hyper-networks are necessary—this establishes the foundation that makes the paper's architectural choices non-trivial.
  • Second, I will walk through the base INR hyper-network design inherited from prior work (TransINR, IPC), highlighting exactly which components existed before and what their limitations were. This is essential context, because HUVR's contributions are all modifications to this baseline.
  • Third, I will detail Key Innovation #1: Patch Tokens as Weight Tokens—how HUVR eliminates the wasted-token problem by turning output patch tokens into the mechanism for INR modulation, enabling dense spatial tasks.
  • Fourth, I will detail Key Innovation #2: Global Token for Modulation and Summarization—how HUVR introduces a [CLS] token that simultaneously serves recognition and participates in modulation via a matrix multiplication design.
  • Fifth, I will detail Key Innovation #3: Tiny Tokens (TinToks)—how the compression module works, how it interacts with the decoder, and how it enables variable-dimensionality embeddings from a single model.
  • Sixth, I will detail Key Innovation #4: Knowledge Distillation for Unified Representation—the distillation loss formulation (Equation 1), teacher selection, token-type and block-level loss weighting, and why distillation is applied to both encoder and decoder but not to TinToks directly.
  • Seventh, I will explain the complete training objective, including the reconstruction losses (MSE, optionally SSIM and LPIPS) and how they coexist with distillation losses.
  • Eighth, I will walk through the inference pathways for different downstream tasks, showing how the same model serves classification, segmentation, depth estimation, and reconstruction.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems/architecture paper whose core idea is that the INR hyper-network framework—when refactored to eliminate wasted tokens, add global context, support compression, and incorporate semantic distillation—becomes a viable vehicle for unified recognition and generation in a single model.

What an Implicit Neural Representation (INR) Is and Why Hyper-Networks Are Necessary

An implicit neural representation stores a signal (an image, a video, a 3D scene) not as an array of pixels or a compressed bitstream, but as the weights of a neural network. Concretely, for an image I of size H × W, an INR is a function f_θ (typically a small MLP) that maps 2D coordinates (x, y) to RGB color values (r, g, b). The function is trained to overfit the single image so that, for every pixel coordinate in the image, f_θ(x, y) produces the correct color. After training, the image is "stored" implicitly in the network's weights θ—you can reconstruct any pixel by evaluating f_θ at its coordinates.

The problem with standard INRs is that they require per-sample training: for every new image, you must run stochastic gradient descent from scratch, iterating over all pixels multiple times (sometimes hundreds of passes) until the MLP memorizes that specific image. This makes INRs computationally prohibitive for anything beyond one-off applications.

A hyper-network solves this by training a second, larger network h that predicts the INR weights for any input image in a single forward pass. Instead of learning f_θ for each image individually, you train h: I → θ' on a large dataset, where h is typically a transformer encoder. After pre-training, encoding a new image requires only one forward pass of h, and reconstructing it requires evaluating the predicted INR θ' at all coordinates. The key mechanism is that h does not predict θ' from scratch; instead, it predicts modulation matrices M_i that adapt a shared, learnable base INR θ_b (trained across the entire dataset). The modulation is applied via elementwise multiplication:

Wi′=Wi⊙MiW_i' = W_i \odot M_i

where W_i is the weight matrix of layer i in the base INR θ_b, M_i is the modulation matrix predicted by the hyper-network for that layer, and W_i' is the modulated weight in the image-specific INR θ'. The \odot operator denotes elementwise (Hadamard) product.

This modulation design is efficient because the base INR captures knowledge common to all images (e.g., that natural images have smooth color variations, edges, etc.), while the modulation matrices capture image-specific deviations. The hyper-network only needs to predict these relatively small modulation matrices rather than dense weight tensors.

The Baseline INR Hyper-Network (TransINR Design) and Its Limitations

Before explaining HUVR's innovations, I need to establish exactly what the prior state-of-the-art looked like, because HUVR's four key innovations are all targeted modifications to this baseline. The TransINR framework (Chen and Wang, 2022), refined by IPC (Kim et al., 2022) and LA-IPC (Lee et al., 2023), has four learnable components:

Component 1: A transformer encoder E. This is the main neural network. It takes image patches as input tokens (standard ViT-style patch embedding) plus additional learnable weight tokens—a fixed number of special input tokens that do not correspond to any image region. The encoder processes all tokens together through self-attention and produces output image tokens and output weight tokens.

Component 2: A shared base INR θ_b. This is a small MLP with n layers that maps coordinates to colors. It is learned during hyper-network training as a global parameter, not specific to any image. If you evaluated θ_b directly (without modulation), it would produce a dataset-average-like blur.

Component 3: Learnable weight tokens. These are input tokens to the encoder, separate from the patch tokens. Their number must be a factor of the INR's layer dimensions to facilitate modulation matrix construction. This creates a coupling between hyper-network design and INR design that is difficult to tune.

Component 4: Fully-connected projection layers. These project the encoder's output dimension d_ViT to the INR's input dimension d_in, enabling the construction of modulation matrices.

How modulation works in the baseline. For each INR layer W_i (of shape d_in × d_out), a corresponding output weight token of dimension d_ViT is projected via a linear layer to dimension d_in, then repeated d_out times to form a matrix M_i of shape d_in × d_out. This matrix is multiplied elementwise with W_i. Critically, only the second layer is modulated in the IPC design (the authors adopt this practice, as stated in the supplementary Section 6.2: "IPC [44] shows that it is sufficient to modulate only the second layer").

What gets discarded. The output image tokens from the encoder are thrown away. They are not used for any loss computation, any downstream task, or any part of the INR prediction. As the paper states: "Existing INR hyper-networks discard the output image tokens, using them neither to calculate loss nor perform inference" (Section 3.1).

This baseline has four critical limitations that motivate HUVR:

  1. Wasted compute and capacity. Half of the encoder's output (the image tokens) serves no purpose, meaning the model's representational capacity is underutilized.

  2. Spatial blindness. Because only the weight tokens drive reconstruction, and these weight tokens mix information across the entire image without spatial alignment, the learned representation does not maintain a clean mapping between input spatial locations and output features. Dense tasks like semantic segmentation, which require per-patch feature quality, are "very challenging" (Section 3.1).

  3. No global representation for classification. The design lacks a [CLS] token or any mechanism for producing a single vector that summarizes the entire image for image-level tasks.

  4. Rigid dimension coupling. The number of weight tokens must divide d_in or d_out, and the encoder dimension must match either d_in, d_out, or some carefully chosen multiple. This limits architectural flexibility.

Key Innovation #1: Patch Tokens as Weight Tokens

HUVR's first—and most consequential—design change is to eliminate the distinction between image tokens and weight tokens. Instead of introducing separate learnable weight tokens and discarding the output patch tokens, HUVR uses the output patch tokens as the weight tokens. Every patch token that exits the encoder participates directly in modulating the INR for its corresponding image patch.

How the per-patch prediction works. Rather than predicting a single INR for the entire image, HUVR predicts a separate INR for each patch. For a ViT-B/16 with 14×14 patches (for a 224×224 input), this means predicting 196 small INRs, each responsible for reconstructing its own 16×16 patch of pixel space. The output patch tokens from the encoder (or decoder, if TinToks are used) are the representations that drive each patch's INR modulation.

Why this solves the wasted-token problem. Every token produced by the transformer serves a purpose. The patch tokens drive reconstruction; the global token (introduced next) drives classification and participates in modulation. No output is discarded. This simultaneously addresses the spatial blindness issue: because each patch token corresponds to a known spatial location in the input, the per-patch representations have a clear spatial mapping. Dense tasks like segmentation can extract features directly from these patch tokens without needing to infer spatial correspondence from a global weight-token mixture.

The dimension challenge and its partial resolution. The paper acknowledges a remaining tension: "The output token has dimension d_ViT, but W_i has dimension d_in × d_out" (Section 3.1). Following the prior-work approach, one could set d_ViT = d_in and copy the output token d_out times to form the modulation matrix. However, the paper states "in practice this is suboptimal (see Section 4.5)." The reason, explored in the ablations, is that this constrains the INR's hidden dimension to match the ViT's token dimension, which is a poor inductive bias: ViT token dimensions (768 for ViT-B) are far larger than effective INR hidden dimensions (the paper uses 256 in Table 16). Forcing them to match wastes INR capacity or forces the ViT to be underparameterized.

HUVR resolves this through the global token mechanism (Innovation #2) and the compression/decompression pathway (Innovation #3), both of which provide degrees of freedom to decouple d_ViT from d_in and d_out.

Key Innovation #2: Global Token for Modulation and Summarization

HUVR introduces a learnable global token (g) that is appended to the patch tokens at the encoder input, analogous to the standard [CLS] token in ViT classification models. This token serves two functions simultaneously:

Function 1: Image-level representation for recognition. After passing through the encoder (and optionally the decoder), the global token's output embedding is a summary of the entire image, suitable for image-level tasks like classification. This is the direct replacement for a standard [CLS] token.

Function 2: Participating in INR modulation via matrix multiplication. This is the architecturally clever part. Rather than constructing the modulation matrix M by simply repeating a patch token, HUVR forms M as the outer product of the global token and the patch token. Specifically:

  • The global token g (of dimension d_dec after decoder processing) is linearly projected to dimension d_out (the output dimension of the INR layer being modulated).
  • Each patch token p (also of dimension d_dec) is linearly projected to dimension d_in (the input dimension of the INR layer being modulated).
  • The modulation matrix for patch p's INR is computed as:

Mp=projglobal(g)×projpatch(p)TM_p = \text{proj}_{\text{global}}(g) \times \text{proj}_{\text{patch}}(p)^T

where proj_global(g) is a vector of shape d_out, proj_patch(p) is a vector of shape d_in, and their outer product yields a matrix of shape d_in × d_out. This matrix is then elementwise-multiplied with the base INR's weight matrix W (specifically the second layer) to produce the patch-specific modulated weights.

What this operation computes. For a concrete example: let d_in = 256 and d_out = 256 (the INR hidden dimension in HUVR's default setting, per Table 10). The global token is projected to a 256-dimensional vector; each patch token is projected to a 256-dimensional vector; their outer product is a 256×256 matrix. This matrix tells the INR how to adjust its weight connections—which input neurons to strengthen or weaken for which output neurons—based on both the global image context (what the image depicts overall) and the local patch content (what that specific 16×16 region contains).

Why this design is superior. The outer product formulation elegantly solves three problems at once. First, it decouples d_ViT from d_in and d_out because the projections are learned linear layers that can map from any dimension to any dimension—the ViT dimension no longer constrains the INR architecture. Second, it injects global context into every patch's reconstruction: the modulation for a patch on the left side of the image can be influenced by what appears on the right side, enabling the INR to exploit long-range dependencies (e.g., consistent lighting, coherent object boundaries). Third, it provides a clean separation of concerns: the global token captures image-level information, the patch tokens capture local information, and their interaction via the outer product captures how global structure constrains local appearance.

Connection to Table 6. The ablation in Table 6 quantifies the impact of the global token on reconstruction. Adding the global token to the patch-wise design improves PSNR from 51.96 to 53.36, SSIM from 0.9974 to 0.9985, and LPIPS from 0.0026 to 0.0007 on ImageNette. These are small absolute improvements (the patch-wise design is already near-perfect at this image size), but they demonstrate that global context does help even for pure reconstruction, validating the design beyond its recognition benefits.

The Base INR and Patch-Wise Reconstruction

The shared base INR θ_b is a small MLP whose architecture is detailed in the supplementary material (Table 16). The configuration is:

  • Sinusoidal positional embedding: dimension 128, applied to input coordinates
  • 3 MLP layers: all with hidden dimension 256
  • ReLU activation
  • 1 convolutional layer: kernel size 3, padding 1, stride 1, followed by PixelShuffle upsampling

The convolutional layer and PixelShuffle are important departures from a pure MLP INR. Instead of querying the INR at every pixel coordinate individually (which would be H × W forward passes per image), HUVR uses strided coordinates with stride 4. This means the INR receives (H/4) × (W/4) coordinate inputs, produces features at that reduced resolution, and the convolutional layer with PixelShuffle upsamples by a factor of 4 to recover the full H × W image. Per the supplementary: "we use strided coordinates (stride = 4) as inputs to f_{θ'} to reduce the computation by 16×."

Why PixelShuffle instead of transposed convolution. PixelShuffle (also known as sub-pixel convolution) rearranges elements from the channel dimension into the spatial dimensions, effectively performing learned upsampling without introducing checkerboard artifacts common in transposed convolutions. It produces smoother outputs for reconstruction tasks. The choice follows intuitions from prior INR works in the NeRV family (Chen et al., 2021) which use PixelShuffle for video reconstruction.

Which layers get modulated. Following IPC (Kim et al., 2022), only the second MLP layer is modulated. The first and third MLP layers, the positional embedding, and the convolutional layer are all shared across patches for a given image (they are part of θ_b and are not modulated per-patch). The paper states this explicitly in the supplementary Section 6.2: "IPC [44] shows that it is sufficient to modulate only the second layer, and since we adopt that practice in this paper, we will explain the process from the perspective that we only modulate a single layer."

Why modulate only the second layer. This is an efficiency-motivated design choice validated by prior work. The first layer processes raw coordinates (mostly a "positional" transformation), and the last layer projects to output space (a "readout" transformation)—these are largely task-agnostic. The middle layer performs the core feature transformation that distinguishes one image from another, so modulating it captures most of the image-specific variation while keeping the number of predicted parameters small. This substantially reduces the hyper-network's output dimensionality: modulating all layers would require predicting tens of thousands of additional values per patch.

Key Innovation #3: Tiny Tokens (TinToks)

The paper introduces a compression pathway between the encoder and the INR modulation head to produce TinToks—compressed token representations with dimension d_t much smaller than the encoder dimension d_ViT. This enables a single model to produce embeddings at multiple resolutions for different downstream budgets.

The compression architecture. The compression pathway consists of three components in sequence:

  1. Downsample layer: LayerNorm followed by a fully-connected linear layer that projects from d_ViT to d_t. For the main experiments, d_t takes values of 8, 16, or 32 (shown in Table 1 for ViT-B/16 and ViT-L/16). Per the supplementary Section 8.1: "our downsample layer... consists of a LayerNorm [2] followed by fully-connected layer which projects from d_ViT to d_t."

  2. Transformer decoder: A shallow transformer (4 blocks in the default configuration, detailed in Table 15) that processes the compressed TinToks. This allows inter-patch communication at the compressed dimension, enabling the model to resolve ambiguities that might arise from extreme compression (e.g., a 32-dimensional vector per patch must carry enough information for both reconstruction and recognition of that patch, and neighboring patch context helps disambiguate). The decoder uses sinusoidal positional embeddings (not RoPE, unlike the encoder).

  3. Upsample layer: LayerNorm followed by a fully-connected linear layer that projects from d_t back up to d_dec, the decoder's token dimension. Per the supplementary: "Our upsample layer is the same but projects from d_t to d_dec where d_dec is the token dimension of the transformer decoder that we use to predict matrices to modulate the INR."

After the decoder, separate linear projection layers map the decoder's output tokens to d_in and d_out for the INR modulation step, and to d_teacher for the distillation loss. These projection layers also include LayerNorms.

Why a decoder is necessary. The paper shows through ablation (Table 10) that reducing the decoder from 4 layers to 1 layer or removing attention from the decoder both degrade reconstruction performance. Specifically, reducing layers from 4 to 1 drops PSNR from 25.35 to 24.78, and removing attention (converting the decoder to an MLP with residual connections) drops PSNR to 24.90. However, interestingly, removing attention improves TinTok classification accuracy (from 69.7% to 73.5% at d_t = 32), suggesting a trade-off: attention helps reconstruction by enabling global information flow for INR prediction, but it may over-smooth patch-level semantic features that are useful for classification.

The compression ratio. For ViT-B/16 with d_ViT = 768, using d_t = 8 yields a 96× compression factor. The paper's headline results in Table 1 use d_t = 8, 16, 32 for both ViT-B and ViT-L. At d_t = 8, HUVR's compressed tokens achieve 64.1% ImageNet accuracy (linear probe) vs. 16.1% for DINOv3 PCA at the same dimension—a 48 percentage point absolute improvement. This is the most visually striking result in the paper and validates the core claim that INR-based training produces more information-dense compressed representations.

Parameter count for the compression module. Per the supplementary Section 8.1, for ViT-B, the total additional parameters for the compression pathway (downsample, upsample, projections) are 3.16M, of which only 305k are used at inference time (the rest are training-only projections to the teacher's feature space). This is a small fraction of the 85.9M encoder parameters.

Key Innovation #4: Knowledge Distillation for Unified Representation

The reconstruction objective alone trains features that capture pixel-level information, but these features lack high-level semantics. HUVR bridges this gap by distilling knowledge from a pre-trained, frozen recognition model (primarily DINOv3) into both its encoder and decoder outputs.

Teacher model selection. The paper primarily uses DINOv3 (Siméoni et al., 2025) as the teacher, with experiments comparing against DINOv2, SigLIP 2, and C-RADIOv3 in Table 7. For the main results, the paper distills from a teacher one size larger than the student: "for our ViT-B, we distill from a ViT-L, and for our ViT-L, we distill from a ViT-H" (Section 4.6). This choice follows the finding that larger teachers produce better final performance but require longer training to show their advantage (a crossover effect demonstrated in Table 11, where at 10 epochs the ViT-B teacher outperforms the ViT-L teacher for standard-size token classification, but by 30 epochs the ViT-L teacher pulls ahead).

The distillation loss formulation. The paper defines a multi-component L2 distillation loss that operates on specific token types at specific block outputs:

Ldistillation=∑t∈T∑o∈OL2(θt,o(Ft,o)−Ft,teacher)⋅αt,o\mathcal{L}_{\text{distillation}} = \sum_{t \in \mathcal{T}} \sum_{o \in \mathcal{O}} \text{L2}\left(\theta_{t,o}(F_{t,o}) - F_{t,\text{teacher}}\right) \cdot \alpha_{t,o}

where:

  • \mathcal{T} = \{\text{type}_g, \text{type}_p\} is the set of token types: the global (type_g) token and the patch (type_p) tokens
  • \mathcal{O} = \{o_{\text{enc}}, o_{\text{dec}}\} is the set of outputs: the final encoder block output and the final decoder block output
  • F_{t,o} are HUVR's features of token type t at output o
  • \theta_{t,o} is a learned linear projection (with LayerNorm) that maps HUVR's features to the teacher's feature dimension d_teacher
  • F_{t,\text{teacher}} are the teacher's corresponding features (DINOv3's [CLS] token for the global token, DINOv3's patch tokens for patch tokens)
  • \alpha_{t,o} are per-component loss weights

What this computes. For each of four combinations (global-encoder, global-decoder, patch-encoder, patch-decoder), a linear projection maps HUVR's features to the teacher's feature space, and the L2 distance between the projected HUVR features and the teacher's features is computed. The per-component losses are weighted by \alpha values and summed. The default weights given in Table 12 are: \alpha_{g,enc} = 2.0, \alpha_{p,enc} = 2.0, \alpha_{g,dec} = 0.5, \alpha_{p,dec} = 0.5. This means encoder outputs are weighted 4× more heavily than decoder outputs in the distillation loss.

Why this form. The L2 (mean squared error) loss directly regresses HUVR's features to match the teacher's features in a shared projection space. The paper does not use cosine similarity or contrastive losses; L2 is simpler and effective. The per-component weighting reflects the observation that encoder features are more semantically meaningful (they haven't passed through the compression-decompression bottleneck), so matching them to the teacher is more important. The decoder features receive a lower weight but are still included because they drive the INR modulation and thus need some semantic guidance to prevent the reconstruction objective from destroying semantic structure.

Why distillation is applied to encoder and decoder, not TinToks directly. The paper states in Section 3.2: "Note that we do not perform distillation on the compressed tokens directly. Combined with the INR reconstruction objective, we find that distillation to the encoder and decoder is sufficient to imbue the compressed tokens with good semantics for downstream reconstruction tasks." This is an important design choice: the TinToks are an information bottleneck, and forcing them to directly match teacher features would create a tension with the reconstruction objective (which requires pixel-level information to survive compression). Instead, the distillation signals applied at the encoder output (before compression) and decoder output (after decompression) create a semantic "envelope" that shapes the compressed representation indirectly.

Projection layer design. Each of the four \theta_{t,o} projections consists of a LayerNorm followed by a single linear layer that maps from HUVR's dimension (d_ViT for encoder outputs, d_dec for decoder outputs) to the teacher's dimension. These projection layers account for the 3.16M additional parameters mentioned earlier.

The crossover effect and why larger teachers need longer training. Table 11 in the supplementary shows a nuanced finding: when distilling from DINOv3 ViT-L into ViT-B, the smaller DINOv3 ViT-B teacher actually gives better results at early training stages (10 epochs). For instance, at 10 epochs, standard token ImageNet accuracy is 82.5% for the ViT-B teacher vs. 80.3% for the ViT-L teacher. However, by 30 epochs, the ViT-L teacher pulls ahead (83.1% vs. 83.6%), and by 50 epochs the gap is 84.1% vs. 84.6%. TinTok accuracy, however, shows the ViT-L teacher as superior even at 10 epochs (70.6% for both, but by 20 epochs it's 71.4% for ViT-B vs. 72.8% for ViT-L). The paper hypothesizes that larger teachers provide a richer but harder-to-match target signal; with insufficient training, the student cannot align well to the more complex feature space, but given enough optimization steps, the richer signal yields better final features.

Complete Training Objective

The full training loss combines three components, described across Sections 3.1 and 3.2:

Component 1: Reconstruction loss (pixel-level). Mean squared error (MSE) between the input image pixels and the reconstructed pixels from the concatenated patch INR predictions:

Lrecon=1H×W×3∑i=1H∑j=1W∑c=13(Ii,j,c−I^i,j,c)2\mathcal{L}_{\text{recon}} = \frac{1}{H \times W \times 3} \sum_{i=1}^{H} \sum_{j=1}^{W} \sum_{c=1}^{3} \left(I_{i,j,c} - \hat{I}_{i,j,c}\right)^2

where H and W are the image dimensions, c indexes color channels, I is the original image, and \hat{I} is the reconstruction. This loss drives the INR hyper-network to produce weights that accurately reconstruct the input.

Component 2: Optional perceptual losses. The paper states: "We can also optionally train with SSIM [99] and LPIPS [111] losses to further improve the reconstruction" (Section 3.1). SSIM (Structural Similarity Index Measure) compares local patches of the original and reconstructed images, capturing structural information that pixel-wise MSE misses (e.g., luminance, contrast, and structure). LPIPS (Learned Perceptual Image Patch Similarity) computes distances in the feature space of a pre-trained deep network (typically AlexNet or VGG), capturing semantic-level similarity. For the main results (Tables 1, 3, 4), the paper uses only MSE, but for the diffusion experiments (Figure 4), SSIM and LPIPS are included to improve generative quality.

Component 3: Distillation loss (semantic). The multi-component L2 loss described above (Equation 1). The total loss is:

Ltotal=Lrecon+λperceptual⋅Lperceptual+Ldistillation\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{recon}} + \lambda_{\text{perceptual}} \cdot \mathcal{L}_{\text{perceptual}} + \mathcal{L}_{\text{distillation}}

where \lambda_{\text{perceptual}} weights the optional perceptual losses. The paper does not specify explicit overall loss weights beyond the per-component \alpha_{t,o} values in the distillation term; reconstruction and distillation appear to be weighted equally (implicit weight of 1.0 each) in the default configuration.

Why MSE is sufficient for the main reconstruction results. The paper shows in Table 1 that using only MSE, HUVR achieves PSNR of 24.66 at d_t = 8 and 27.83 at d_t = 32 for ViT-B, significantly higher than the PCA baseline. This suggests that the INR hyper-network structure, combined with the global-patch modulation mechanism, is sufficiently expressive to produce good pixel-level reconstructions from the L2 objective alone. The optional perceptual losses improve visual quality (as needed for generative tasks) but are not necessary for the paper's core unified representation claim.

Training Configuration and Hyperparameters

The paper provides detailed training settings in Section 4.1 and the supplementary Section 8.1.

Data. Pre-training uses a mix of DataComp (Gadre et al., 2023, image-only subset) and ImageNet22k (Russakovsky et al., 2015, without labels). At least 10% of samples come from ImageNet22k (inspired by DINOv3's practice). Training proceeds for "the equivalent of 50 epochs on ImageNet22k" (Section 4.1), which given 13.7M training images in ImageNet22k, means approximately 685M images seen.

Optimization. AdamW optimizer with a base learning rate of 0.0005, rescaled linearly by batch size: lr = lr_base × B/256. Cosine annealing schedule after a linear warm-up of 5 epochs (or the equivalent number of iterations). Gradient clipping with a max norm of 0.01. Per the supplementary: "For data augmentations, we take a random resized crop, with a minimum rescale ratio of 0.2. We then resize to our input resolution, 256×256 for pretraining." No other augmentations (no color jitter, no blurring, no horizontal flip) are used. This is a deliberate choice: the reconstruction objective provides its own form of invariance learning, and heavy augmentation could corrupt the pixel-level training signal.

Normalization. HUVR uses its own normalization: mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5), mapping pixel values from [0, 1] to [-1, 1]. When interacting with other models (teachers or evaluation pipelines), the paper uses those models' original normalization settings.

Model sizes. ViT-B/16: 85.9M parameters (encoder), ViT-L/16: 304M parameters (encoder). The decoder adds 50.3M (ViT-B) or 68.2M (ViT-L) parameters in the default configuration, with a token dimension of 1024 or 1280, 4 transformer blocks, 16 attention heads, and a feedforward dimension of 4096. The base INR θ_b has 275k parameters across all variants.

High-resolution fine-tuning. For dense tasks (ADE20K segmentation at 512×512, NYUv2 depth estimation at 480×480), the paper adds a second training stage: 3 additional epochs on ImageNet22k only, with a mix of 256×256 and 512×512 resolution images, "following DINOv3" (Section 4.4).

Inference Pathways for Different Tasks

The unified architecture enables multiple inference modes depending on the downstream task, without any architectural changes or fine-tuning (other than training a linear probe head):

Image classification. The global token from the encoder output (standard, d_ViT dimensions) or the decoder output serves as the image representation. A linear classifier is trained on top of these frozen features. The paper evaluates both standard tokens and compressed TinToks (after decoder processing) for classification in Table 1.

Semantic segmentation. Patch tokens from the encoder or decoder output are extracted and bilinearly upsampled to the original image resolution. A linear segmentation head (1×1 convolution) is trained on these frozen per-patch features. The paper reports ADE20K mIoU using the standard AMRADIO evaluation protocol (Section 4.1). TinToks can also be used—their upsampled decoder output tokens serve as the segmentation features, achieving 29.7 mIoU for ViT-B at d_t = 32 (Table 4).

Depth estimation. Similar to segmentation: patch tokens are extracted, and a linear depth prediction head is trained. The metric is RMSE on NYUv2. At d_t = 32, TinToks achieve 0.5980 RMSE for ViT-B, competitive with DINOv3's 0.7056 at the same dimension (Table 4).

Image reconstruction. For each patch, the decoder output patch token and global token are projected to form the modulation matrix M_p, which modulates the base INR's second layer to produce the patch-specific INR θ'_p. The INR is evaluated at strided coordinates (stride 4) and the PixelShuffle layer upsamples to 16×16 pixels. All patch reconstructions are concatenated to form the full image. This is the native output pathway—no additional training is needed.

Diffusion-based generation. The TinToks (after decoder processing) serve as the latent representation for training a diffusion model (DiT-XL). The paper demonstrates this in Table 2 using TinTok dimensions of 16 and 256, training a class-conditional DiT-XL on ImageNet. The latents have spatial structure (e.g., 16×16 patches × 256 channels = 16×16×256 latent grid for the d_t = 256 setting), allowing standard diffusion architectures to operate on them.

Summary of Architectural Design Choices

The paper's architectural decisions can be understood as resolving tensions between reconstruction fidelity (which benefits from high-dimensional, spatially precise features with global context), recognition accuracy (which benefits from semantically structured features with both global and local representations), and compression efficiency (which benefits from low-dimensional bottleneck representations).

  • Patch-wise INR prediction enables spatial precision for reconstruction and dense recognition.
  • Global token + outer product modulation injects global context into reconstruction while providing a [CLS] token for classification.
  • TinTok compression + decoder creates a bottleneck that forces information density, producing representations that work for both pixel-level and semantic tasks even at extreme compression ratios.
  • Knowledge distillation to encoder and decoder (but not TinToks directly) imports semantic structure without conflicting with the reconstruction objective at the bottleneck.
  • Modulating only the second INR layer (following IPC) keeps the hyper-network's output dimensionality manageable while capturing most image-specific variation.
  • Strided coordinates + PixelShuffle (following NeRV-family practices) reduces INR evaluation cost by 16× without sacrificing reconstruction quality.
  • MSE-only reconstruction loss for main results keeps training simple, with optional perceptual losses reserved for generation-focused experiments.

4. Key Insights and Innovations

Innovation 1: The INR Hyper-Network is Reframed as a Unified Architecture Rather Than a Reconstruction-Only Tool

The paper's most fundamental conceptual move is not any single architectural modification but the reframing of what an INR hyper-network is for. Prior work (TransINR, IPC, LA-IPC, ANR) treated INR hyper-networks purely as efficient alternatives to per-sample INR training — the goal was faster or better reconstruction, period. The components were engineered exclusively for that purpose: learnable weight tokens that extract global image information, a base INR that captures dataset-level structure, and modulation matrices that adapt it per-image. Any tokens that didn't participate in reconstruction (the patch tokens output by the encoder) were discarded as irrelevant.

HUVR asks a different question: what if the INR hyper-network architecture, properly restructured, is actually a natural vehicle for unified representation learning? The insight is not that you can bolt recognition onto an INR hyper-network post-hoc (that would be the post-hoc unification approach the paper critiques). It's that the INR hyper-network, by its nature, forces the model to learn representations that span the full information hierarchy — from pixel-level detail (needed for reconstruction) to semantic structure (needed for efficient compression of many images into a shared base network). The paper argues, through citations to Schmidhuber, Gregor et al., and Solomonoff, that compression and understanding are linked: a representation that can reconstruct its input from a bottleneck must capture its essential structure. The INR hyper-network, which compresses images twice (once into latents, once into a shared base INR), is structurally aligned with this principle.

What distinguishes this from prior "joint training" approaches (MAGE, adversarial representation learning) is that the architectural structure itself — not just the loss function — enforces the unification. In MAGE, a single ViT is trained with a masked reconstruction objective and evaluated on both generation and recognition, but the architecture is a standard ViT with no dedicated reconstruction pathway. In HUVR, the INR prediction head is a separate, structured output (a neural network) that directly consumes the learned representations. This means the representations must be good enough to serve as instructions for building another network — a much stronger requirement than being good enough to classify or segment. The paper validates this in Table 13: removing the INR prediction objective (training with distillation only) degrades TinTok classification from 68.2% to 64.2%, showing that reconstruction training is synergistic with recognition, not merely compatible.

This is a fundamental conceptual shift, not an incremental refinement. It repurposes an entire subfield (INR hyper-networks) from a narrow compression/efficiency framing to a representation learning framing, and argues that the architectural properties that made them good for reconstruction also make them good for recognition — once the wasted-token and missing-global-token design flaws are fixed.

Innovation 2: The Distinction Between Image Tokens and Weight Tokens is Identified as the Architectural Barrier to Unified Representation

The paper makes a diagnostic contribution that is easy to overlook: it identifies exactly why prior INR hyper-networks couldn't serve as general-purpose vision encoders. The culprit is the separation between image tokens (discarded) and weight tokens (used for reconstruction), which the paper diagnoses as creating three simultaneous failures: wasted compute, broken spatial correspondence, and no global representation.

Prior work in the INR hyper-network literature (TransINR, IPC, LA-IPC, ANR) treated this separation as an implementation detail, not a problem. The learnable weight tokens were a design choice inherited from the meta-learning literature, where the "context" for adaptation comes from a separate set of parameters. No prior paper identified this as the reason INR hyper-networks couldn't perform dense prediction or image-level classification. The fact that these models "accidentally" had patch tokens passing through the encoder and being discarded was, as far as the literature was concerned, an unremarkable consequence of using a ViT as the hyper-network backbone.

HUVR's contribution here is recognizing that this separation is not a necessary feature of INR hyper-networks but a contingent design choice, and that collapsing the distinction — making patch tokens be the weight tokens, and adding a global token that participates in modulation — eliminates all three failures simultaneously. This is a conceptual insight about the architecture's degrees of freedom, not a new loss function or training procedure. The ablation in Table 6 quantifies its impact on reconstruction: the patch-wise design alone (before adding global tokens or compression) takes PSNR from 27.15 to 51.96 on ImageNette — a 24.81 dB jump that dominates all other architectural improvements combined. This suggests that the patch-wise formulation is not merely "better" but fundamentally changes what the model can represent: a single global INR must encode the entire image's variation into one set of weights, while per-patch INRs can specialize locally, making the reconstruction problem dramatically easier.

The significance goes beyond performance. By showing that the patch-token-as-weight-token design works — and works dramatically better than the global-weight-token design it replaces — the paper establishes a design principle for future unified architectures: the tokens that drive generation should be spatially grounded in the input, not abstracted into a separate learned representation. This principle likely extends beyond INR hyper-networks to any architecture that tries to unify recognition and generation.

Innovation 3: Compressed Representation Quality Emerges from the Reconstruction Bottleneck, Not Just from Distillation

The paper demonstrates something surprising about the TinToks: they don't just survive extreme compression (96× at d_t = 8); under compression, they dramatically outperform representations produced by dimensionality-reducing stronger recognition models. At 8 dimensions, HUVR achieves 64.1% ImageNet accuracy vs. 16.1% for DINOv3 PCA (Table 1) — a 48 percentage point gap. This gap narrows as dimensionality increases but remains substantial: at 32 dimensions, HUVR achieves 79.4% vs. 72.2% for DINOv3 PCA on ViT-B.

What makes this intellectually distinctive is that HUVR's TinToks achieve this despite the teacher model (DINOv3) being the very model whose PCA-compressed features they're being compared against. The distillation target is the full 768-dimensional DINOv3 features; the TinToks are a 32-dimensional bottleneck, and yet a linear probe on those 32 dimensions outperforms a linear probe on PCA-compressed DINOv3 at 32 dimensions. This cannot be explained by distillation alone — if the TinToks were merely learning to mimic the teacher, they'd at best match its PCA-compressed performance (since PCA is the optimal linear compression for reconstruction of the teacher's feature space). The fact that they substantially exceed it means the reconstruction objective is forcing information into the compressed representation that distillation alone would not preserve.

The paper's explanation, though not fully unpacked, is that the reconstruction objective creates an information-dense bottleneck: every bit of the 32-dimensional TinTok must carry both semantic content (to match the teacher) and pixel-level content (to reconstruct the image). This dual pressure prevents the representation from collapsing to only the most discriminative semantic dimensions (which is what PCA captures — the directions of maximum variance in the teacher's feature space). Instead, the TinToks must encode a richer set of information, some of which turns out to be useful for classification even though it wouldn't survive an unsupervised dimensionality reduction of a pure recognition model.

This has implications for representation learning beyond this paper: it suggests that reconstruction objectives are not just compatible with semantic representation learning but can produce complementary information that pure semantic objectives miss. The reconstruction pressure acts as a regularizer that prevents the representation from discarding structurally relevant information that happens to be orthogonal to the teacher's principal components. This is a concrete empirical demonstration of the compression-understanding hypothesis the paper invokes theoretically.

Innovation 4: The Global Token × Patch Token Outer Product Is Positioned as a General Mechanism for Injecting Global Context into Local Generation

The paper's decision to form modulation matrices via the outer product of a projected global token and a projected patch token (rather than, say, concatenation followed by an MLP, or repeating the patch token) is presented as a practical solution to a dimension-matching problem, but it encodes a deeper design principle: global context should modulate local generation multiplicatively, not additively. The outer product proj_global(g) × proj_patch(p)^T means that the global token controls how much each dimension of the patch token influences each output dimension of the INR layer. If the global token's projection is near-zero in some dimension, that dimension of the patch token has no effect on the INR's output, regardless of its value. This is a gating mechanism: global context determines which aspects of local information are relevant for reconstruction.

Contrast this with an additive approach (concatenate global and patch tokens, pass through an MLP to produce modulation values). In the additive case, global and local information are on equal footing — the MLP can learn to weight them differently, but it cannot learn that in some contexts local information should be ignored entirely for certain output dimensions. The multiplicative formulation makes the interaction explicitly conditional: the global token defines the conditions under which local features matter.

The empirical evidence that this matters is subtle but present. Table 6 shows that adding the global token to the already-patch-wise design improves reconstruction from 51.96 to 53.36 PSNR on ImageNette — a small absolute gain, but notable because the patch-wise design is already near-perfect (SSIM of 0.9974, LPIPS of 0.0026). The remaining errors are likely in regions where local information is ambiguous (e.g., a patch at an object boundary where the local texture could belong to either the foreground or background). Global context resolves this ambiguity by telling the INR which "mode" the patch is in. The outer product is structurally suited to this disambiguation in a way that additive conditioning is not.

This insight, while demonstrated specifically for INR modulation, generalizes: any architecture that generates structured outputs from a combination of global and local features should consider multiplicative interaction, not just concatenation, when the global context should gate the use of local information. The paper doesn't make this generalization explicitly, but the design choice and its justification in Section 3.1 (solving the dimension-coupling problem "at once") imply it.

Innovation 5: Verifier Over-Optimization Analogy for a Different Domain — the Distillation vs. Reconstruction Trade-off as Two Competing "Verifiers"

A subtle conceptual contribution emerges from the paper's treatment of the trade-off between reconstruction quality and recognition accuracy. In Table 8, distilling to both global and patch tokens (the "all tokens" setting) gives the best classification at d = 768 (83.2%) but the worst reconstruction (25.33 PSNR) compared to distilling to patches only (26.95 PSNR but 81.9% classification). In Table 9, the block selection that maximizes standard classification (distilling from encoder block 12, decoder block 1: 83.2% accuracy, 26.18 PSNR) is different from the selection that maximizes TinTok classification (encoder block 11, decoder block 4: 72.6% TinTok accuracy, 25.40 PSNR). And in Table 10, removing attention from the decoder improves TinTok classification (69.7% → 73.5%) at significant cost to reconstruction (25.35 → 24.90 PSNR).

The paper doesn't frame it this way, but these results collectively demonstrate a phenomenon analogous to reward hacking in RLHF or verifier over-optimization in test-time compute scaling: there are two competing training signals (distillation loss and reconstruction loss), and optimizing too aggressively for either one degrades the other. The distillation loss pushes features to match the teacher's semantic structure; the reconstruction loss pushes features to preserve pixel-level information. When you strengthen the distillation signal (by adding more distillation components, or distilling to blocks closer to the output), the features over-specialize to semantics and lose reconstruction fidelity. When you strengthen the reconstruction pathway (by adding more decoder layers, or keeping attention in the decoder), the features over-specialize to pixel-level detail and lose semantic discriminability, especially in the compressed bottleneck.

The insight is not that this trade-off exists (that's expected), but that the paper systematically maps it and shows that the optimal operating point depends on which downstream metric you care about — and, importantly, that the architecture provides knobs (distillation target selection, decoder design, block selection) to tune this trade-off without changing the model architecture or retraining from scratch. Tables 8, 9, and 10 are effectively a control panel for the semantics-vs-reconstruction frontier. This is a practically useful finding: a practitioner deploying HUVR can adjust these knobs based on whether their application prioritizes retrieval accuracy (favor TinTok classification, use attention-free decoder, distill to earlier encoder blocks) or generation quality (favor reconstruction, use full decoder with attention, distill to later blocks).

More broadly, this trade-off characterization suggests that unified representation learning is fundamentally a multi-objective optimization problem where the Pareto frontier is controlled by architectural choices about where and how much each objective signal is injected. This framing positions the specific HUVR design as one point on a broader frontier that future work can explore.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary benchmark is ImageNet-1k (Deng et al., 2009), using the standard 1,000-class validation set of 50,000 images for classification and reconstruction evaluation. Linear probing classification is reported with both original labels and ReaL labels (Beyer et al., 2020). ObjectNet (Barbu et al., 2019) tests out-of-distribution robustness. Five fine-grained visual classification (FGVC) datasets are evaluated: Caltech-UCSD Birds 200 (Wah et al., 2011), Stanford Cars (Krause et al., 2013), Describable Textures Dataset (Cimpoi et al., 2014), Oxford 102 Flowers (Nilsback and Zisserman, 2008), and Food-101 (Bossard et al., 2014). Dense tasks use ADE20K (Zhou et al., 2017) for semantic segmentation and NYUv2 (Nathan Silberman et al., 2012) for depth estimation. Reconstruction metrics are computed on the 50,000 ImageNet-1k validation images, which are never seen during training. Pre-training data mixes DataComp (Gadre et al., 2023, image-only subset) and ImageNet22k (Russakovsky et al., 2015) without labels, with at least 10% of samples from ImageNet22k following DINOv3 practice (Section 4.1). For INR hyper-network comparisons (Table 5), the paper uses ImageNette (Howard), CelebA (Liu et al., 2015), and LSUN Churches (Yu et al., 2015).

Base models. The paper uses ViT-B/16 (85.9M parameters) and ViT-L/16 (304M parameters) with Rotary Positional Embeddings (RoPE; Su et al., 2024) as encoder backbones. These sizes are chosen due to their "prevalence of practical adoption" and to manage training cost (Section 4.1). The ViT-B has 12 encoder blocks, token dimension 768, feedforward dimension 2048 (SwiGLU; Shazeer, 2020), and 16 attention heads. The ViT-L has 24 blocks, token dimension 1024, feedforward dimension 3072, and 16 attention heads. No register tokens are used. The decoder (where applicable) has 4 transformer blocks with sinusoidal positional embeddings, token dimension 1024 (ViT-B) or 1280 (ViT-L), feedforward dimension 4096, and 16 attention heads. The base INR has 275k parameters across all variants, with 3 MLP layers of hidden dimension 256, sinusoidal positional embedding dimension 128, and a single convolutional layer with PixelShuffle upsampling (stride 4 coordinate inputs for 16× computation reduction). The total model with encoder, decoder, and INR head is approximately 136M parameters for ViT-B.

Metrics. For classification, the primary metric is ImageNet-1k top-1 accuracy (%) using linear probing (training a linear classifier on frozen features). The paper also reports ReaL accuracy (a re-annotated ImageNet label set that corrects for multi-object images), ObjectNet top-1 accuracy, and per-dataset accuracy on the five FGVC benchmarks. All classification evaluations use L-BFGS (Liu and Nocedal, 1989) for FGVC datasets and standard SGD for ImageNet/ObjectNet. For semantic segmentation, the metric is mean Intersection-over-Union (mIoU) and mean Accuracy (mAcc) on ADE20K, using linear probing with the AMRADIO protocol (Ranzinger et al., 2024). For depth estimation, the metric is Root Mean Square Error (RMSE) on NYUv2, following the probing setup from El Banani et al. (2024). For reconstruction, the metrics are Peak Signal-to-Noise Ratio (PSNR in dB), Structural Similarity Index Measure (SSIM; Wang et al., 2004), and Learned Perceptual Image Patch Similarity (LPIPS; Zhang et al., 2018), all computed on the 50,000 ImageNet-1k validation images. For generation via diffusion (Table 2), the metrics are Fréchet Inception Distance (FID), spatial FID (sFID), Inception Score (IS), Precision, and Recall, following the DiT protocol (Peebles and Xie, 2023).

Baselines. The paper compares against multiple strong recognition models: DINOv3 (Siméoni et al., 2025) with ViT-B/16 and ViT-L/16, C-RADIOv3 (Heinrich et al., 2024) with ViT-B and ViT-L, and SigLIP 2 (Tschannen et al., 2025) with ViT-B and ViT-L. For compressed representation comparisons (Table 1), PCA baselines are constructed by fitting principal component analysis on the ImageNet-1k training set features from each model and applying the same transform to all evaluation datasets. For reconstruction, a decoder is trained on frozen DINOv3 PCA features as a baseline, with "roughly the same number of parameters as ours" (Section 4.2). For generation (Table 2), the Stable Diffusion VAE (Rombach et al., 2022) serves as the autoencoder baseline, with DiT-XL trained on its 32×32×4 latents. For INR hyper-network comparisons (Table 5), baselines are TransINR (Chen and Wang, 2022), IPC (Kim et al., 2022), LA-IPC (Lee et al., 2023), and ANR (Zhang et al., 2024).

Generation budget / compute accounting. For the main experiments, training budget is measured in epochs on ImageNet22k (50 epochs standard, or equivalent iterations based on 13.7M images). For INR hyper-network comparisons (Table 5), budget is measured in training epochs on each dataset (ImageNette/CelebA/LSUN), with the paper explicitly noting that HUVR uses fewer epochs (400 on ImageNette vs. 4000 for TransINR) while achieving better results. For diffusion experiments (Table 2), DiT-XL is trained for 400k steps (vs. the 7000k steps used for DiT-XL/2 in the original paper). The paper does not report FLOP counts for inference-time reconstruction, though it notes the strided coordinate evaluation reduces INR computation by 16× relative to per-pixel queries.

Cross-validation / statistical protocol. No formal cross-validation is reported. The ImageNet-1k validation set (50,000 images) serves as a held-out evaluation for all metrics; it is never used during pre-training. For the PCA baselines, the PCA transform is fit on the ImageNet-1k training set and applied to the validation set (and other datasets) to avoid contamination. There is no reported confidence interval, standard error, or statistical significance testing for any metric. The test set sizes are: 50,000 for ImageNet classification/reconstruction, 50,000 for ObjectNet, ~200-11,000 per FGVC dataset, 2,000 for ADE20K validation, and 654 for NYUv2 test set. For the INR hyper-network comparisons (Table 5), evaluation is on the standard test splits of ImageNette (3,925 images), CelebA (19,962 images), and LSUN Churches (300 images).

Main Quantitative Results

TinTok Compression: Unified Recognition and Reconstruction from Tiny Embeddings

The paper's most distinctive result is that its compressed TinToks simultaneously achieve good recognition and reconstruction, dramatically outperforming PCA-compressed versions of stronger pure-recognition models (Table 1). The headline finding: at 8 dimensions (96× compression from ViT-B's 768-dimensional tokens), HUVR achieves 64.1% ImageNet accuracy vs. 16.1% for DINOv3 PCA — a 48 percentage point gap. At 16 dimensions, HUVR achieves 75.6% vs. 40.6% for DINOv3 PCA. At 32 dimensions, the gap narrows to 79.4% vs. 64.1% (ViT-B). For ViT-L at 32 dimensions, HUVR achieves 78.1% — slightly lower than DINOv3 PCA at 72.2%, but DINOv3's PCA features cannot reconstruct images.

The reconstruction advantage is equally stark. At 8 dimensions, HUVR achieves 24.66 PSNR / 0.6918 SSIM vs. 15.51 PSNR / 0.4975 SSIM for DINOv3 PCA (with a separately trained decoder). At 32 dimensions (ViT-B), HUVR achieves 27.83 PSNR / 0.7845 SSIM vs. 17.68 PSNR / 0.5398 SSIM for DINOv3 PCA. For context, the Stable Diffusion VAE (a dedicated compression autoencoder) achieves 24.99 PSNR / 0.7078 SSIM — HUVR at 32 dimensions exceeds this (27.83 PSNR). HUVR at 8 dimensions essentially matches the SD VAE's PSNR (24.66 vs. 24.99) while providing semantically meaningful features that the VAE does not.

The fine-grained classification results (Table 1, right columns) show that TinToks retain detailed category information. At 32 dimensions (ViT-B), HUVR achieves 80.8% on Cars, 86.5% on CUB, 70.1% on DTD, 99.6% on Flowers, and 89.7% on Food. These are competitive with or exceed C-RADIOv3 PCA at the same dimension (82.0%, 73.3%, 73.3%, 98.6%, 89.4%) and SigLIP 2 PCA (90.6%, 77.8%, 67.0%, 99.3%, 89.2%), despite HUVR offering reconstruction capability that neither baseline possesses.

The paper compares against C-RADIOv3 and SigLIP 2 PCA baselines in addition to DINOv3 to establish that HUVR's advantage is not an artifact of using DINOv3 as a teacher. SigLIP 2 PCA at 32 dimensions (ViT-B) achieves only 63.4% ImageNet accuracy — substantially lower than HUVR's 79.4% — despite SigLIP 2 being trained on "25 times more data" (Section 4.3). This suggests the reconstruction objective provides a complementary training signal that produces more information-dense compressed representations than what survives PCA compression of even a very strong (and much more expensively trained) recognition model.

For dense tasks under compression (Table 4), HUVR's 32-dimensional TinToks achieve 29.7 mIoU on ADE20K for ViT-B, matching DINOv3 (29.7) and slightly exceeding C-RADIOv3 (28.9) and SigLIP 2 (17.5) at the same dimension. For depth estimation, HUVR achieves 0.5980 RMSE vs. 0.7056 for DINOv3 at 32 dimensions. The fact that TinToks match or exceed the baselines on dense tasks is notable because these tasks require spatial precision — all information for both pixel-level reconstruction and mid-level segmentation must fit within 32 floating-point numbers per patch. The paper explicitly flags this: "all information for both patch reconstruction (pixel-level) and segmentation (mid-level) must fit within 32 floating point values" (Section 4.4).

For generation (Table 2), training a DiT-XL on HUVR's compressed latents (16×16×16 tokens) yields FID of 24.72 and sFID of 76.09, compared to 23.05 and 68.65 for the SD VAE baseline (32×32×4 latents). This is a noticeable gap in FID but a larger gap in sFID (spatial FID, which measures structural coherence at multiple scales), suggesting that HUVR's latents produce less spatially coherent generations. When the TinTok dimension is increased to 256 (producing 16×256×256 — though the paper states 16×256×256 for this configuration, which appears to be a spatially-flattened representation), FID improves to 24.53 and sFID to 68.37, much closer to the VAE baseline. However, Inception Score remains substantially lower (66.13 vs. 70.34), and Precision drops (0.4307 vs. 0.4318) while Recall degrades more noticeably (0.4367 vs. 0.4775), indicating that HUVR latents produce less diverse samples. The paper acknowledges these results are not state-of-the-art for generation but considers them "promising" as a proof-of-concept that unified representations can support generative modeling (Section 4.2). Figure 4 shows visual examples with "many degradations and artifacts" but demonstrates recognizable class-conditional samples.

Standard-Size Token Recognition: HUVR Competes with State-of-the-Art

Beyond compressed tokens, HUVR's standard 768-dimensional (ViT-B) and 1024-dimensional (ViT-L) token embeddings achieve competitive recognition performance (Table 3). For ViT-B/16, HUVR achieves 85.0% ImageNet top-1 accuracy, exceeding DINOv3 (84.6%), C-RADIOv3 (82.4%), and SigLIP 2 (84.5%). For ReaL labels, HUVR achieves 89.2% vs. 88.9% (DINOv3), 87.6% (C-RADIOv3), and 89.0% (SigLIP 2). On ObjectNet, HUVR achieves 62.0% — better than DINOv3 (59.4%) and C-RADIOv3 (54.5%), but below SigLIP 2 (68.4% — though SigLIP 2 trains on 25× more data, as noted in Section 4.3).

On FGVC datasets (Table 3, ViT-B), HUVR achieves 93.1% on Cars (vs. 93.4% DINOv3, 92.8% SigLIP 2), 89.4% on CUB (vs. 89.7% DINOv3), 84.3% on DTD (vs. 83.9% DINOv3), 99.7% on Flowers (matching DINOv3), and 94.3% on Food (vs. 93.7% DINOv3, 94.2% SigLIP 2). The pattern is consistent: HUVR is within ±0.3–0.6% of the best prior method on most datasets, with no systematic weakness. For ViT-L/16, HUVR achieves 86.9% ImageNet accuracy — slightly below DINOv3 at 87.1% and SigLIP 2 at 87.1%. On FGVC datasets, HUVR shows some larger gaps: 91.7% on Cars (vs. 94.9% SigLIP 2), 85.7% on CUB (vs. 91.1% DINOv3), and 84.5% on DTD (vs. 86.6% DINOv3). The paper attributes these gaps to pre-training data curation: "DINOv3 mines pretraining data that is similar to many of these, whereas we use pre-training data that is not specifically curated for these" (Section 4.3).

For dense recognition with standard tokens (Table 4, top section), HUVR achieves 52.0 mIoU on ADE20K for ViT-B, exceeding DINOv3 (50.8), C-RADIOv3 (49.5), and SigLIP 2 (40.0). For ViT-L, HUVR achieves 53.5 mIoU vs. 54.2 for DINOv3 — a 0.7 point gap. On depth estimation, HUVR's ViT-B achieves 0.3263 RMSE vs. 0.3305 for DINOv3 (lower is better) — a marginal improvement. For ViT-L, HUVR achieves 0.3287 vs. 0.3235 for DINOv3 — a marginal disadvantage.

The overall picture for standard tokens is that HUVR essentially matches or slightly exceeds DINOv3 on most recognition benchmarks while also providing reconstruction capability that DINOv3 lacks entirely. The gap to SigLIP 2 on ObjectNet (62.0% vs. 68.4%) is the largest single-metric deficit, but as noted, SigLIP 2 uses 25× more pre-training data, making the comparison inexact.

INR Hyper-Network Reconstruction: State-of-the-Art with Faster Training

Table 5 compares HUVR's INR hyper-network design directly against prior INR hyper-networks (TransINR, IPC, LA-IPC, ANR) on three standard reconstruction benchmarks: ImageNette (178×178), LSUN Churches (256×256), and CelebA (178×178). The critical result: HUVR achieves the best PSNR on all three datasets while using significantly fewer training epochs (400 on ImageNette vs. 4000 for prior works) — effectively achieving state-of-the-art reconstruction quality with 10% of the training time.

On ImageNette, HUVR achieves 48.44 PSNR vs. 46.10 for LA-IPC (the previous best). On CelebA, the gap is larger: 56.91 vs. 50.74 for LA-IPC — a 6.17 dB improvement. On LSUN Churches, HUVR achieves 34.00 PSNR; the paper does not report LA-IPC or IPC results for LSUN in Table 5, but the best comparison is ANR at 28.30 PSNR. The paper is careful to establish fairness: all models are matched on total encoder parameters, shared base INR parameters, and unique predicted parameters (see supplementary Section 8.2). HUVR's encoder has one fewer layer to compensate for the added decoder, maintaining parameter parity.

Table 6 unravels how HUVR achieves these gains through a cumulative ablation on ImageNette, starting from TransINR + RoPE and adding components incrementally. The baseline TransINR + RoPE achieves 23.78 PSNR. Switching to modulation of only the second layer (adopting the IPC approach) improves PSNR to 27.15. The critical jump comes from the patch-wise design: making each patch token predict its own INR takes PSNR from 27.15 to 51.96 — a 24.81 dB improvement that dominates all subsequent changes combined. This is the architectural innovation that enables per-patch specialization: a single global INR with limited capacity cannot capture an entire image's variation, but 196 small INRs (for a 14×14 patch grid) can each specialize to their local 16×16 region.

Adding the global token further improves PSNR from 51.96 to 53.36 (+1.40 dB), with SSIM rising from 0.9974 to 0.9985 and LPIPS dropping from 0.0026 to 0.0007. The compression and decoder additions actually reduce PSNR slightly (53.36 → 48.58 → 48.44), which is expected since they introduce a bottleneck that discards information. Per the paper: "The compression and decoder are not necessary for reconstruction, but they are essential for good TinToks and well-behaved distillation, respectively" (Section 4.5). This trade-off is central to the paper's design philosophy: the architecture is optimized for unified performance, not maximal reconstruction, and the components that enable recognition (compression, decoder with attention) impose a modest reconstruction cost.

Recognition and Reconstruction Scale Together — But With Diminishing Returns

Figure 3 demonstrates that both classification accuracy and reconstruction quality improve with training time, establishing that the two objectives are not in fundamental tension. For ViT-B/16, classification rises from ~0.78 at 100k iterations to ~0.85 at 600k iterations. PSNR rises from ~26.2 to ~27.7 over the same range, but the curve flattens more quickly — the majority of reconstruction improvement occurs in the first 200k iterations, while classification continues to improve more steadily. The paper notes: "the reconstruction performance saturates more quickly than classification" (Section 4.6). This suggests that the INR reconstruction objective is "easier" to learn (the model quickly finds a good modulation strategy) while distillation requires longer training to fully align with the teacher's feature space, consistent with the crossover effect observed in Table 11.

The teacher model choice also affects both recognition and reconstruction simultaneously (Table 7). When distilling ViT-B from different teachers (DINOv3 ViT-B vs. ViT-L), using the larger teacher improves PSNR (25.33 → 26.30, with DINOv3 ViT-L teacher) and SSIM (0.7129 → 0.7439). Standard token classification improves with the larger teacher only after sufficient training (the crossover effect in Table 11), but TinTok classification and reconstruction improve from the larger teacher even at shorter training durations. When comparing across teacher types (DINOv3 vs. SigLIP 2 vs. C-RADIOv3), there is no single best teacher: SigLIP 2 (ViT-B) gives the best reconstruction (26.61 PSNR, 0.7458 SSIM) but poor segmentation (37.87 mIoU); C-RADIOv3 gives the best segmentation (43.32 mIoU) but the worst classification (78.3%); DINOv3 gives the best balance. The paper notes that "a truly optimal method would probably distill from a mixture of teachers" but considers such engineering out of scope (Section 4.7).

Dense Recognition Under Extreme Compression

Table 4 (lower sections) shows that TinToks can perform semantic segmentation and depth estimation even at 32 dimensions. For ViT-B, HUVR at 32 dimensions achieves 29.7 mIoU on ADE20K — matching DINOv3 at the same compressed dimension (29.7) and exceeding C-RADIOv3 (28.9) and SigLIP 2 (17.5). For depth estimation, HUVR achieves 0.5980 RMSE vs. 0.7056 for DINOv3 at 32 dimensions — a notably lower error. For ViT-L at 32 dimensions, HUVR achieves 30.9 mIoU vs. 29.4 for DINOv3 and 0.5726 RMSE vs. 0.6685 for DINOv3.

However, the standard-size tokens significantly outperform the compressed ones on these dense tasks: ViT-B 768-dimensional tokens achieve 52.0 mIoU (vs. 29.7 at 32 dimensions) and 0.3263 RMSE (vs. 0.5980). This ~22 mIoU gap between standard and compressed representations quantifies the cost of extreme compression for spatial tasks. The paper presents this as a strength — TinToks perform surprisingly well given the compression ratio — but the absolute gap is large enough that standard tokens would be preferred for applications where dense task accuracy is the primary concern.

Ablation Studies and Robustness Checks

Distillation token selection (Table 8): Distilling to the global token only produces the best standard-token classification (82.9%) and TinTok classification (71.3%), but poor segmentation (39.29 mIoU). Distilling to patch tokens only produces the best segmentation (45.23 mIoU) and reconstruction (26.95 PSNR, 0.7580 SSIM), but slightly worse classification (81.9% standard, 65.5% TinTok). Distilling to both (the chosen setting) gives a compromise: 83.2% classification, 43.68 mIoU segmentation, but noticeably worse reconstruction (25.33 PSNR, 0.7133 SSIM). This is a non-obvious finding: adding more distillation signal (distilling to both token types) hurts reconstruction, suggesting that the distillation objective competes with the reconstruction objective for model capacity. The paper chooses the "both" setting anyway and compensates with longer training and larger teachers — a practical compromise rather than an optimal solution.

Distillation block selection (Table 9): The choice of which encoder and decoder blocks receive distillation loss affects different metrics differently. Distilling to encoder block 12 and decoder block 1 gives best standard classification (83.2%) and best reconstruction (26.18 PSNR), but worst TinTok classification (68.8%). Distilling to encoder block 11 and decoder block 4 gives best TinTok classification (72.6%) but worse reconstruction (25.40 PSNR). The standard setting (12, 4) — the outputs of both final blocks — is a middle ground, sacrificing some TinTok accuracy and reconstruction quality for simplicity. This ablation reveals that the "optimal" block for distillation depends on which representation you care about: earlier encoder blocks and later decoder blocks favor compressed representations; later encoder blocks and earlier decoder blocks favor full-size representations.

Decoder design choices (Table 10): Several architectural decisions in the decoder have measurable trade-offs. Increasing the INR hidden dimension from 256 to 512 decreases PSNR (25.35 → 25.30) and dramatically decreases segmentation (43.85 → 23.36 mIoU), suggesting that the larger INR is harder to modulate effectively from compressed tokens. Changing the number of INR layers from 3 to 5 provides marginal benefits: 4 layers (default) achieves 25.35 PSNR, while 5 layers achieves 25.48 PSNR and better TinTok classification (70.2% vs. 69.7%). Reducing decoder layers from 4 to 1 hurts reconstruction (25.35 → 24.78 PSNR) and slightly hurts TinTok classification (69.7% → 69.6%). Removing attention from the transformer decoder (converting it to an MLP with residual connections — the non-obvious finding in this table) improves TinTok classification from 69.7% to 73.5% but significantly degrades reconstruction (25.35 → 24.90 PSNR). The paper exploits this: for TinTok-focused experiments (such as Table 1), the attention-free decoder is used, accepting the reconstruction penalty in exchange for better compressed representation quality.

Reconstruction objective necessity (Table 13): Training HUVR without the INR prediction loss (distillation only, no reconstruction) degrades TinTok classification from 68.2% to 64.2% at 32 dimensions. This is a critical ablation for the paper's central claim: it demonstrates that the reconstruction objective is not merely compatible with recognition but is synergistic — the act of learning to reconstruct improves the quality of compressed semantic representations. The model trained without reconstruction is "fundamentally incapable of reconstruction" (supplementary Section 7.3), so the 64.2% accuracy represents the best a pure-distillation approach can achieve at this dimension. The 4.0 percentage point gain from adding reconstruction validates the core hypothesis that compression-for-reconstruction produces more information-dense representations.

Distillation loss weights (Table 12): Adjusting the relative weights of the four distillation components (global-encoder, patch-encoder, global-decoder, patch-decoder) produces measurable but not dramatic changes. The default weights (α_g,enc = 2.0, α_p,enc = 2.0, α_g,dec = 0.5, α_p,dec = 0.5) give 83.0% classification, 68.2% TinTok, 43.68 mIoU, and 25.84 PSNR. Halving all weights (1.0, 1.0, 0.25, 0.25) improves reconstruction to 26.75 PSNR with only small costs to classification (82.8%) and TinToks (69.2%). Doubling decoder weights degrades reconstruction to 25.19 PSNR with minimal classification gain. The overall picture is that distillation weights primarily trade reconstruction quality against recognition quality, with fairly smooth gradients — there are no sharp cliffs, suggesting stable optimization.

Distillation teacher size crossover (Table 11): When distilling ViT-B from a DINOv3 teacher, the ViT-B teacher initially outperforms the ViT-L teacher on standard-token classification (at 10 epochs: 82.5% vs. 80.3%). The crossover point — where the ViT-L teacher becomes superior — occurs at different epochs for different metrics: standard token classification crosses over between 30–40 epochs, standard token segmentation crosses over at approximately 10–20 epochs, TinTok classification and reconstruction are better with the larger teacher even at 10 epochs. By 50 epochs, the ViT-L teacher gives 84.6% standard classification vs. 84.1% for ViT-B, 76.6% TinTok vs. 72.8%, and 26.02 PSNR vs. 25.50. This non-monotonic behavior — larger teacher initially worse but eventually better — is an important practical finding: it means experiments with insufficient training would conclude (incorrectly) that matching-size teachers are optimal.

INR hyper-network design components (Table 6): The cumulative ablation on ImageNette, discussed above, quantifies the contribution of each architectural change. The dominant factor is the patch-wise design (+24.81 PSNR), followed by the global token (+1.40 PSNR). The IPC "second layer only" modulation (+3.37 PSNR over the baseline TransINR) validates the efficiency of modulating fewer layers. The compression and decoder additions show the cost of enabling unified representation: compression reduces PSNR from 53.36 to 48.58 (a 4.78 dB loss, corresponding to the information discarded by the bottleneck), and the decoder marginally adjusts this to 48.44.

Reconstruction objective with SSIM and LPIPS (Section 4.1 and Figure 4 caption): The paper states that SSIM and LPIPS losses are optional and used primarily for generation experiments. No ablation table compares MSE-only vs. MSE+SSIM+LPIPS reconstruction quality, but Figure 4's caption notes that the diffusion samples were generated from a HUVR trained with "LPIPS and SSIM losses in addition to the pixel-wise and DINOv3 MSE losses," implying that perceptual losses help generation quality. The quantitative impact on standard reconstruction metrics is not reported.

Critical Assessment

Claim 1: HUVR matches or outperforms DINOv3 on recognition while enabling reconstruction.

The evidence for this claim is strong but includes important caveats. The headline numbers — +0.4% ImageNet classification for ViT-B (85.0% vs. 84.6%), +1.2 mIoU on ADE20K (52.0 vs. 50.8), +4.84 PSNR reconstruction — are clear and well-supported by Tables 3, 4, and 1 respectively. However, several qualifications are necessary:

First, the "match or outperform" on recognition is true for ViT-B but not uniformly true for ViT-L. On ImageNet, ViT-L HUVR achieves 86.9% vs. 87.1% for DINOv3; on ADE20K, 53.5 vs. 54.2 mIoU; on CUB, 85.7% vs. 91.1%. The ViT-L results show a consistent (if small) recognition deficit. The paper acknowledges this implicitly by attributing gaps to data curation differences — DINOv3 mines data similar to the FGVC datasets — but this means HUVR's recognition parity is conditional on having access to curated training data, which HUVR does not use.

Second, the +4.84 PSNR reconstruction advantage is somewhat misleading as a standalone claim because DINOv3 was never designed to support reconstruction. The PCA+decoder baseline in Table 1 is the fairest comparison, and there HUVR wins dramatically (27.83 vs. 17.68 PSNR at 32 dimensions), but this is a combination of architecture (HUVR has a built-in decoder pathway) and training objective (HUVR trains with reconstruction loss). A more probing comparison would be: how well does HUVR reconstruct compared to a dedicated autoencoder of similar capacity trained purely for reconstruction? The paper partially addresses this with the SD VAE comparison in Table 1 (HUVR achieves 27.83 vs. 24.99 PSNR), but the SD VAE uses a different architecture and was not optimized for the same input resolution. A direct comparison to a ViT-based autoencoder trained with the same reconstruction losses but no distillation would help isolate the effect of the unified training on reconstruction quality. This experiment is not reported.

Third, the reconstruction evaluation is limited to PSNR, SSIM, and briefly LPIPS. These metrics are known to correlate imperfectly with perceptual quality. The diffusion results in Table 2 suggest that HUVR's latents, while achieving good PSNR, produce worse FID when used for generation (24.72 vs. 23.05 for SD VAE). This implies that the PSNR advantage may not fully translate to perceptual or generative quality. A broader evaluation with perceptual metrics (LPIPS across the full validation set, not just in ablations) and human preference studies would strengthen the claim.

Claim 2: TinToks offer dramatically better compressed representations than PCA baselines.

This claim is strongly supported by Table 1, which is the paper's most compelling single table. The gaps at low dimensions are large and consistent across datasets: +48% ImageNet accuracy at 8 dimensions, +35% at 16 dimensions, +15.3% at 32 dimensions (ViT-B). The inclusion of multiple baseline models (DINOv3, C-RADIOv3, SigLIP 2) and the better-than-VAE reconstruction quality add credibility.

However, there is a genuine weakness in the PCA baseline construction. The PCA transform is fit on ImageNet-1k training set features from each model and then applied uniformly to all datasets. This means the PCA basis is optimized for ImageNet reconstruction, which may not be optimal for other datasets. A per-dataset PCA transform (which would be a stronger baseline) is not reported. Additionally, the PCA baseline compresses after feature extraction — the features were originally trained at full dimensionality with no compression pressure. It is unsurprising that they compress poorly compared to features trained with an explicit bottleneck. A stronger baseline would be training DINOv3 (or another recognition model) with a bottleneck at the target dimension — essentially, "what if you just trained a recognition model with a smaller embedding dimension?" The paper does not report such a baseline, making it unclear whether the TinTok advantage comes from the INR reconstruction objective or simply from training with a bottleneck (any bottleneck) combined with distillation.

The dense task results under compression (Table 4) also raise a question: at 32 dimensions, HUVR matches DINOv3 on ADE20K mIoU (29.7 vs. 29.7), but standard-sized DINOv3 achieves 50.8 mIoU. The ~21 mIoU gap between compressed and uncompressed is large, meaning that for applications where dense prediction accuracy matters most, compressed tokens (even HUVR's) are substantially worse than standard tokens. The paper does not quantify this compression-accuracy trade-off curve — e.g., what mIoU do TinToks achieve at 64, 128, 256 dimensions? — which would help practitioners understand where TinToks become "good enough" for dense tasks.

Claim 3: The INR hyper-network design is state-of-the-art for image INRs.

The evidence in Tables 5 and 6 strongly supports this claim. HUVR achieves the highest PSNR on all three benchmark datasets while using 10% of the training epochs. The fairness constraints (matching encoder/INR/unique parameter counts) are explicitly documented in the supplementary material. The cumulative ablation in Table 6 cleanly isolates the contribution of each design choice.

The main limitation is the narrow evaluation: reconstruction quality (PSNR/SSIM/LPIPS) on three datasets. The paper does not evaluate other properties that matter for INRs, such as: how well do the INRs generalize to super-resolution (querying at higher coordinate density than trained on)? How smooth are the reconstructions across patch boundaries? What is the inference speed for reconstruction vs. prior methods? The claim of "state-of-the-art" is thus limited to PSNR on standard benchmarks — a narrow but standard evaluation in the INR literature.

Additionally, the LSUN Churches comparison in Table 5 lacks IPC and LA-IPC baselines, comparing only to ANR (28.30 PSNR). The paper does not explain why these comparisons are missing. Given that LA-IPC achieves 46.10 on ImageNette (vs. HUVR's 48.44), the LSUN gap might be similarly large, but without the numbers this remains speculative.

Claim 4: Recognition and reconstruction improve together.

Figure 3 supports this claim for training time: both classification accuracy and PSNR increase with iterations. However, the paper's own ablations show that this is not universally true. Table 8 shows that strengthening distillation (adding more distillation targets) reduces reconstruction quality. Table 10 shows that removing decoder attention increases TinTok classification at the cost of reconstruction. The teacher choice in Table 7 shows that SigLIP 2 gives the best reconstruction but worst segmentation among the teachers. These results collectively demonstrate that recognition and reconstruction are often in tension, not always aligned. The "improve together" claim is true specifically for scaling training time and (to some extent) scaling teacher size, but architectural and loss-weighting choices reveal trade-offs. The paper is honest about this — Section 4.7 is titled "Some Design Decisions Involve Trade-offs" — but the executive summary's framing of "recognition and reconstruction can improve together" somewhat understates the pervasiveness of these trade-offs.

Missing Experiments and Analyses

Several experiments would have strengthened the paper substantially:

  • A bottleneck-only recognition baseline. Train a ViT-B with a 32-dimensional embedding layer (no reconstruction, only distillation) to test whether the TinTok advantage comes from the INR objective or simply from bottleneck training. This is the most important missing ablation for the paper's central claim about unified representation.

  • Scaling the TinTok dimension continuously. Table 1 shows 8, 16, 32 dimensions. What happens at 4, 64, 128, 256? Where does the advantage over PCA disappear? Where does reconstruction quality saturate? A scaling curve would characterize the information-density advantage more precisely.

  • Cross-dataset generalization of TinToks. All TinTok evaluations use linear probes. Does a TinTok trained on ImageNet transfer to other datasets without re-training the probe (zero-shot retrieval)? The PCA baselines would likely have an advantage here since PCA is an optimal linear compression for ImageNet, and HUVR's advantage might shrink or reverse under distribution shift.

  • Latency and throughput measurements. The paper mentions that patch-wise INR evaluation has "slightly higher time costs and a larger memory footprint" (supplementary Section 8.2) but provides no numbers. For deployment, wall-clock reconstruction time and GPU memory matter as much as PSNR.

  • Robustness to image corruptions and out-of-distribution inputs. How do TinToks perform on ImageNet-C or ImageNet-R compared to PCA baselines? The reconstruction objective might produce features that are more robust (because they must preserve pixel structure) or less robust (because they overfit to pixel-level statistics of the training distribution).

  • Alternative INR architectures. The paper uses a fixed INR design (3-layer MLP, 256 hidden dimensions, sinusoidal embeddings). Would a SIREN (Sitzmann et al., 2020) or instant-NGP-style (Müller et al., 2022) INR produce better results? The design space for the INR itself is not explored.

  • Multiple teacher distillation. The paper notes that distilling from a mixture of teachers would likely be optimal but does not implement it. Given that Table 7 shows complementary strengths (SigLIP 2 excels at reconstruction, C-RADIOv3 at segmentation, DINOv3 at classification), a multi-teacher distillation experiment would directly test whether unified representations can capture the best of all worlds.

Assessment of the Core Argument

The paper's central argument — that INR hyper-networks, properly refactored and combined with distillation, can serve as unified recognition-generation models — is substantively supported by the experiments, but the support is stronger for the "unified" claim (the model can do both) than for the "no fundamental trade-off" claim (the model doesn't sacrifice one for the other). The evidence consistently shows that HUVR achieves competitive recognition and reconstruction, which no prior single model does. This is a genuine achievement. However, the evidence also consistently shows that architectural and training choices involve navigating a Pareto frontier where improvements to one axis often degrade the other. The paper's practical contribution is thus not eliminating the trade-off but providing a single architecture with controllable knobs to navigate it — a more nuanced but equally valuable contribution that the paper's own framing occasionally underplays.

The most robust and surprising result is the TinTok compression advantage (Table 1), which is large, consistent across dimensions and baseline models, and not easily explained by teacher quality or architecture alone. The most limited result is the generation quality (Table 2, Figure 4), which lags behind dedicated autoencoders and contains visible artifacts. This suggests that while HUVR succeeds at representation-level unification (embeddings useful for recognition and reconstructible into images), it has not yet achieved generation-level unification (latents from which high-quality novel images can be sampled). The paper acknowledges this limitation transparently and positions it as future work, which is appropriate.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For in the Headline Efficiency Claims

The assumption or constraint. The paper's most distinctive result—that HUVR's compressed TinToks dramatically outperform PCA-compressed baselines—depends on the model having been trained with an explicit compression bottleneck combined with both reconstruction and distillation objectives. This requires a full pre-training pipeline: a ViT encoder, a transformer decoder, an INR prediction head, a frozen teacher model (DINOv3), and coordinated multi-objective training with carefully tuned loss weights. The paper never quantifies the total computational cost of this pipeline relative to the PCA baselines it compares against. PCA is computationally trivial: fit a covariance matrix on the training set features and apply a matrix multiplication. HUVR requires pre-training a ~136M parameter model (for ViT-B) for 50 ImageNet22k-equivalent epochs with a teacher model running in parallel for distillation.

The paper acknowledges the general issue of training scale relative to baselines: "Our pre-training does not operate at the same scale or scope as prior image representation methods. Compared to SigLIP 2, we train with less data for less time. Compared to DINOv3, we do not train with specialized curated data" (Section 4.8). But this is framed as a caveat about absolute performance, not as a limitation of the compression comparison specifically. The actual computational cost of achieving the TinTok compression advantage is never reported.

The consequence. A practitioner deciding whether to use HUVR's compressed embeddings vs. a simpler PCA baseline faces an unquantified cost-benefit trade-off. The TinToks offer substantially better accuracy at a given dimension, but they require training an entire unified model—even if the practitioner only needs compressed embeddings and does not need reconstruction. The paper provides no guidance on whether the accuracy gain justifies the training cost, because the cost itself is not measured or reported. This is particularly relevant because PCA can be applied to any pre-trained model (including off-the-shelf DINOv3, SigLIP 2, or CLIP models that the practitioner may already have), while HUVR requires training from scratch or using the authors' released checkpoints (which are trained on a specific data mixture and may not transfer to the practitioner's domain).

What evidence exists in the paper. The paper provides no FLOP count, GPU-hour estimate, or wall-clock training time for any experiment. Table 1 reports accuracy and reconstruction metrics for TinToks vs. PCA baselines, but there is no corresponding cost column. The supplementary Section 8.1 reports hyperparameters (batch size, learning rate, epochs) and model parameter counts, but not total floating-point operations or training time. Section 4.5 notes that HUVR trains ImageNette for 400 epochs instead of 4000 for prior INR hyper-networks, but this relative efficiency claim is about INR reconstruction quality specifically, not about the full unified pre-training pipeline. The diffusion experiment in Table 2 reports that DiT-XL was trained for 400k steps (vs. 7000k for the original DiT-XL/2), but this again is a relative comparison, not an absolute cost measurement.

Mitigation status. The paper does not address this limitation. It neither reports training cost nor provides a cost-aware comparison to baselines. The absence is notable because the paper explicitly motivates TinToks in terms of practical deployment: "Tasks like retrieval become very difficult as the amount of data increases at scale, and reducing the embedding size introduces massive savings" (Section 1). The inference-time savings from smaller embeddings are quantified (via accuracy at various dimensions), but the training-time cost to produce those embeddings is entirely unexamined. A cost-benefit analysis of training HUVR specifically for compressed embedding extraction—amortizing training cost over expected inference queries—would be needed for a deployment decision, and the paper provides no data to support such an analysis. The release of model checkpoints mitigates this somewhat for practitioners who can use the authors' pre-trained weights, but the checkpoint is trained on a specific data mixture and may not generalize to other domains, so domain-specific fine-tuning costs would still apply.

Hard Problems Remain Essentially Unsolved: No Generation Quality Comparable to Dedicated Autoencoders

The assumption or constraint. HUVR's unified representation is trained with a reconstruction objective (pixel-wise MSE, optionally SSIM and LPIPS) that optimizes for fidelity to the input image. The paper uses this reconstruction capability to position HUVR as a model that "enables generative capabilities with its high-quality tiny embeddings" (Abstract). However, reconstruction fidelity (measured by PSNR/SSIM) is not the same as generative capability (measured by FID/IS when training a downstream generative model on the latents). The paper implicitly assumes that latents which reconstruct well will also serve as good foundations for generative modeling, but the diffusion experiments reveal a substantial gap between HUVR and dedicated autoencoders.

The consequence. When training a DiT-XL on HUVR's compressed latents (Table 2), the resulting generative model underperforms the same DiT-XL trained on Stable Diffusion VAE latents. At 16×16×16 TinTok dimensions, HUVR achieves FID of 24.72 vs. 23.05 for SD VAE—a gap of 1.67 FID that represents a noticeable degradation in sample quality. More tellingly, spatial FID (sFID) is 76.09 vs. 68.65, indicating that HUVR latents produce samples with worse spatial coherence at multiple scales. Inception Score drops from 70.34 to 60.17, and Recall drops from 0.4775 to 0.4645, indicating reduced sample diversity. When increasing the TinTok dimension to 256 (producing 16×256×256 latents), FID improves to 24.53 and sFID to 68.37—much closer to the VAE baseline—but Inception Score remains substantially lower (66.13 vs. 70.34), and both Precision (0.4307 vs. 0.4318) and Recall (0.4367 vs. 0.4775) trail the baseline.

The paper is candid about these results: "we acknowledge these results are significantly worse than the current generative state-of-the-arts" (supplementary Section 9). But the implication is that HUVR does not actually achieve generative unification—it achieves reconstruction unification. The latents can reproduce input images faithfully (high PSNR), but they do not support high-quality sampling of novel images from the learned distribution. This is a qualitatively different kind of failure than the recognition trade-offs: it suggests that the INR-based latent space, as currently designed, is structurally unsuited to serving as a generative prior, even though it excels at compression.

What evidence exists in the paper. Table 2 and Figure 4 provide the direct evidence. Table 2 quantifies the FID/IS gap across two TinTok configurations, showing that even at 256 dimensions (a much larger latent than the SD VAE's 32×32×4 = 4096-dimensional latent when flattened), HUVR cannot match the VAE's generative quality. Figure 4 shows visual examples that the paper describes as demonstrating "many degradations and artifacts" (supplementary Section 9). The figure caption notes that this DiT was trained for 4500k steps (vs. 400k for Table 2) and that the HUVR encoder was trained with LPIPS and SSIM losses in addition to MSE—meaning these results represent the paper's best effort at generative quality, not a preliminary attempt.

The paper offers hypotheses for the gap but does not experimentally diagnose it: "we hypothesize we have optimized neither the diffusion model nor the diffusion process for our architecture. For example, we have both patch tokens and a global token, and the global token is responsible for combining with the patch token to modulate every single patch INR. In spite of this, our implementation of DiT handles the global and patch tokens equivalently" (supplementary Section 9). This suggests that the architectural features that enable unified representation (global token, patch-specific modulation) may actually be obstacles to standard generative modeling pipelines, which expect spatially homogeneous latent representations.

Mitigation status. The paper partially addresses this by referencing concurrent work: "Future work could apply techniques, such as those in RAE [117], to improve the quality" (supplementary Section 9). But it provides no experimental evidence that these techniques would close the gap, and the fundamental tension—that HUVR's latents are designed for a specific decoding pathway (INR modulation) that has no analog in standard diffusion training—remains unresolved. The paper frames the diffusion results as a "proof-of-concept to convey the promise of HUVR for generation" (supplementary Section 9), which is an appropriate characterization, but it means the "generation" half of the "unified recognition and generation" claim is aspirational rather than demonstrated at competitive quality.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. This is the prior analysis's framing, but the corresponding limitation in HUVR is the structural tension between attention and compressed representation quality revealed in the decoder design ablations. The paper's architecture relies on a transformer decoder to process TinToks before INR modulation, with the decoder providing inter-patch communication that helps resolve ambiguities in the compressed representation. However, Table 10 demonstrates that the presence or absence of attention in this decoder creates a direct trade-off between reconstruction quality and compressed representation quality.

The consequence. Removing attention from the transformer decoder (converting it to an MLP with residual connections) improves TinTok classification accuracy from 69.7% to 73.5%—a 3.8 percentage point gain at 32 dimensions. But this same change degrades reconstruction PSNR from 25.35 to 24.90 (−0.45 dB). The paper exploits this trade-off: "for our TinTok experiments (such as in Table 1), we opt to use this setting, and we mainly to counter-balance the negative effect with more training time" (Section 4.7). This means that the headline TinTok results in Table 1 use a different model configuration than the standard-token recognition results in Table 3—the attention-free decoder for compressed evaluation, the full-attention decoder for standard evaluation.

The consequence for deployment is that a single HUVR checkpoint cannot simultaneously achieve optimal standard-token recognition and optimal TinTok recognition. The practitioner must choose which configuration to deploy, and that choice will advantage one set of downstream tasks over the other. The paper does not report whether the attention-free decoder's improved TinTok classification comes at a cost to standard-token classification or segmentation (Table 10 only reports d=768 classification for the default setting, and the d=768 column is constant at 83.1% across all decoder configurations, suggesting the choice does not affect standard-token quality—but this is counterintuitive and not explained).

Reducing the decoder from 4 layers to 1 layer degrades PSNR from 25.35 to 24.78 with minimal impact on TinTok classification (69.7% vs. 69.6%). Increasing the INR hidden dimension from 256 to 512 causes a dramatic drop in segmentation mIoU from 43.85 to 23.36—nearly halving dense prediction performance—while barely affecting classification or reconstruction. This sensitivity suggests that the architecture has not converged to a configuration that simultaneously optimizes all axes; rather, the paper has found a set of configurations that each optimize a subset of metrics, and the reported results cherry-pick the best configuration per evaluation.

What evidence exists in the paper. Table 10 is the primary evidence, supported by Tables 8 and 9 which show similar trade-offs in distillation target selection and block selection. The paper is transparent about these trade-offs—Section 4.7 is explicitly titled "Some Design Decisions Involve Trade-offs"—but it does not provide guidance on how to select a single configuration for deployment, nor does it report whether the trade-offs can be mitigated through longer training or different hyperparameters.

Mitigation status. The paper acknowledges the trade-offs but does not attempt to resolve them. The approach is pragmatic: use different configurations for different evaluations, and note that "counter-balancing the negative effect with more training time" helps. But this is a mitigation, not a solution—the underlying tension remains, and practitioners inheriting HUVR for their own applications would need to navigate the same trade-offs, potentially requiring multiple rounds of experimentation to determine which decoder configuration works best for their specific task mix.

The Method Is Evaluated on a Single Visual Domain with a Narrow Range of Tasks

The assumption or constraint. All experiments in the paper use natural image datasets: ImageNet-1k, ImageNet22k, DataComp (image-only), ADE20K, NYUv2, and a few smaller benchmarks (ImageNette, CelebA, LSUN Churches). The paper implicitly assumes that findings from these datasets generalize to other visual domains and tasks, but provides no evidence for this assumption. The paper's abstract claims the model "learns representations which are simultaneously useful for recognition and generation" without qualification, and the title promises "Unified Universal Vision Encoding"—the word "universal" implying domain generality that is not tested.

The paper acknowledges the training data limitation: "Our pre-training does not operate at the same scale or scope as prior image representation methods" (Section 4.8). But this is framed in terms of data quantity, not data diversity. The more fundamental issue is domain coverage: natural images from standard computer vision datasets represent a narrow slice of "vision." Medical images (X-rays, MRI, pathology slides), satellite imagery, document scans, diagrams and charts, microscopy images, and synthetic renderings all have different statistical properties that might interact differently with the INR reconstruction objective and the distillation target.

The consequence. The reconstruction quality of HUVR depends on the base INR θ_b learning a shared representation of "what images look like" that generalizes across the training distribution. If deployed on images from a substantially different domain (e.g., monochrome medical images with different noise characteristics, or satellite images with different spatial frequency spectra), the base INR—trained on natural color images—may not provide an effective prior for modulation. The per-patch modulation mechanism partially mitigates this (patches can adapt locally), but the base INR's architecture (sinusoidal embeddings, 3-layer MLP with 256 hidden dimensions) and training distribution still constrain what kinds of signals can be faithfully reconstructed.

More specifically, the distillation target (DINOv3) is itself trained on natural images. Its feature space encodes semantic distinctions relevant to ImageNet categories (object types, breeds, models). For domains where the relevant semantic distinctions are different—e.g., medical images where the relevant categories are tissue types or abnormalities—DINOv3's feature space may not provide useful guidance, and distilling from it could actively harm domain-specific task performance by pulling features toward natural-image semantics that are irrelevant to the target domain.

The paper also does not evaluate on tasks that are standard for generative models but not for recognition models, such as image inpainting (reconstructing masked regions), super-resolution (reconstructing at higher resolution than the input), or image-to-image translation. These tasks would test whether the INR-based decoding pathway can generalize beyond exact input reconstruction—a capability that dedicated autoencoders (including the SD VAE) possess but which HUVR has not demonstrated.

What evidence exists in the paper. The paper provides no cross-domain evaluation. All classification benchmarks use natural image datasets. The reconstruction evaluation is on ImageNet-1k validation images (natural images). The dense prediction tasks (ADE20K, NYUv2) are both natural indoor/outdoor scenes. Even the diffusion experiments use ImageNet. The paper does not report results on medical imaging, satellite imagery, document understanding, or any non-natural domain. The INR hyper-network comparison (Table 5) uses ImageNette (natural objects), CelebA (faces), and LSUN Churches (indoor/outdoor scenes)—all natural images.

Mitigation status. The paper does not address this limitation. The title's claim of "universal" vision encoding is unsupported by domain-diverse evaluation. The supplementary Section 9 suggests that "future work could apply techniques, such as those in RAE, to improve the quality" for generation, but does not discuss domain generalization. For a practitioner considering HUVR for a non-natural-image domain, the paper provides no guidance on whether the approach will transfer, what adaptations might be needed, or what performance degradation to expect. This is particularly significant because the INR-based reconstruction mechanism makes stronger assumptions about signal structure (continuity, smoothness, color-space properties) than pure recognition models, so domain shift could affect reconstruction more severely than it affects classification.

The Paper Provides No Computational Cost Analysis for Any Component

The assumption or constraint. The paper evaluates HUVR's quality extensively (accuracy, mIoU, PSNR, SSIM, LPIPS, FID, IS) but provides zero measurements of computational cost: no FLOP counts, no GPU-hours for training, no inference latency, no memory usage, no throughput (images per second) for any component. The only cost-related information is parameter counts (85.9M for ViT-B encoder, 50.3M for decoder, 275k for base INR) and relative training epoch comparisons (400 vs. 4000 epochs for ImageNette in Table 5). This omission is systematic across all experiments.

The paper implicitly assumes that cost can be inferred from parameter counts and that the comparisons are fair because models are matched on total parameters. But parameter count is a poor proxy for computational cost in this architecture because the INR evaluation step—querying a per-patch MLP at strided coordinates and upsampling via PixelShuffle—has no analog in the recognition-only baselines. Comparing HUVR to DINOv3 on parameter count alone ignores that HUVR's reconstruction inference requires an additional computational step that DINOv3 does not perform at all.

The consequence. A practitioner cannot make informed deployment decisions from this paper. The headline results—HUVR matches DINOv3 on recognition while adding reconstruction—imply that the reconstruction capability comes "for free" in terms of compute, but this is almost certainly false. The INR evaluation requires forward passes through 196 small MLPs (for a 14×14 patch grid) plus a PixelShuffle upsampling operation per patch, which adds latency and memory overhead relative to a pure recognition forward pass. The paper's supplementary acknowledges this: "Our patch-wise INR formulation can result in less efficiency during forward computation, since our modulation procedure will yield p unique INRs... This has slightly higher time costs and a larger memory footprint" (supplementary Section 8.2). But "slightly higher" is never quantified.

The training cost is similarly opaque. Training HUVR requires (1) a forward pass through the encoder, (2) compression and decoder processing, (3) INR modulation and reconstruction (with coordinate queries and PixelShuffle), (4) a forward pass through the frozen teacher model for distillation targets, (5) multiple loss computations (reconstruction MSE, distillation L2, optionally SSIM/LPIPS). The teacher model forward pass alone doubles the effective model size during training (the teacher is as large as or larger than the student, per Section 4.6: ViT-B distills from ViT-L, ViT-L distills from ViT-H). None of these costs are reported.

What evidence exists in the paper. No cost measurements exist anywhere in the paper. The supplementary Section 8.1 reports batch size, learning rate, and epoch counts, but not wall-clock time. The supplementary Section 8.2 mentions efficiency in qualitative terms ("slightly higher time costs") without numbers. Table 5 reports training epochs for INR hyper-network comparisons, which provides a relative cost metric (HUVR trains for 10% of the epochs of prior work), but this is specific to the pure reconstruction setting and does not include distillation costs. For the main experiments (Tables 1, 3, 4), no cost information of any kind is provided.

Mitigation status. The paper does not address this limitation. There is no mention of computational cost as a limitation in Section 4.8, and no suggestion that future work should report such measurements. For a paper that explicitly motivates its design in terms of practical deployment—"Tasks like retrieval become very difficult as the amount of data increases at scale, and reducing the embedding size introduces massive savings" (Section 1)—the absence of cost analysis is a significant gap. The paper quantifies the savings from smaller embeddings (accuracy at various dimensions) but not the cost to produce those embeddings, making the overall efficiency picture incomplete.

The Architecture Cannot Be Directly Applied to Video or Multi-Modal Inputs Without Substantial Modification

The assumption or constraint. HUVR is designed for and evaluated exclusively on still images. The core architectural mechanisms—patch-wise INR prediction, global token × patch token outer product modulation, stride-4 coordinate queries with PixelShuffle upsampling—are all defined in terms of 2D spatial structure. The paper's title claims "Universal Vision Encoding," and the introduction references "image and video representations" as the domain of vision encoders (Section 1), but the method itself has no temporal modeling capability and no mechanism for handling video, 3D data, or multi-view inputs.

The authors implicitly acknowledge this by situating their work in the context of prior video INR research (citing the NeRV family in Section 2.1) but not claiming that HUVR extends to video. However, the "universal" in the title and the framing around "unified universal vision encoding" sets an expectation that the approach generalizes beyond still images, and the paper provides no evidence for such generalization.

The consequence. The INR hyper-network design decisions that work well for images may not transfer to video. Video INRs typically require either 3D coordinate inputs (x, y, t), frame-wise latent codes with shared base networks, or flow-based warping mechanisms to exploit temporal redundancy. HUVR's per-patch modulation—where each spatial patch gets an independent INR and the global token provides image-level context—has no natural extension to the temporal dimension. Would each spatio-temporal tube get its own INR? Would the global token summarize an entire video clip? How would the outer product modulation handle the additional dimension? These design questions are non-trivial, and the paper provides no guidance.

For multi-modal inputs (e.g., images with text, video with audio, RGB-D data), HUVR's architecture is even more constrained. The INR prediction pathway assumes RGB output (3 channels), and the base INR is trained to produce color values. Extending to depth, surface normals, or semantic segmentation maps would require either separate INR heads for each modality or a multi-channel output with a modality-agnostic reconstruction loss. The distillation pathway, which currently aligns to a single teacher's feature space (DINOv3's), would need to incorporate multiple teachers with different semantic spaces—a problem the paper acknowledges is non-trivial ("a truly optimal method would probably distill from a mixture of teachers, but we consider such engineering efforts out of scope," Section 4.7).

What evidence exists in the paper. The paper cites video INR works extensively in the related work (Section 2.1, citing NeRV, CNeRV, HNeRV, E-NeRV, etc.) and references a prior work that "redesigns INRs as per-video hyper-networks, and aligns a learnable latent for each frame with its corresponding CLIP embedding" (Latent-INR, Section 2.1). But HUVR itself is evaluated only on still images. There is no video experiment, no discussion of how the architecture would extend to video, and no acknowledgment in the limitations section that the current design is image-specific.

Mitigation status. The paper does not address this limitation. The "universal" claim is not qualified by modality, and Section 4.8 (Limitations) discusses data scale and curation but not the image-only scope. For a method positioned as "unified universal vision encoding," the restriction to still images is a significant scope limitation that should be explicitly acknowledged. The paper suggests text-aligned pre-training as future work for Vision Language Model applications (Section 4.8) but does not discuss video or 3D extensions at all. A practitioner interested in video understanding or generation—a major application domain for unified representations—would find no guidance on how to adapt HUVR beyond still images.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new loss function, a new training paradigm, or a new state-of-the-art on a single benchmark. Instead, it makes a structural argument that will shift how the field thinks about the relationship between recognition and generation architectures: the INR hyper-network, properly refactored, is not just a compression tool but a natural vehicle for unified representation learning. This is a reframing, not a paradigm shift—the underlying components (ViTs, INRs, knowledge distillation) all exist—but the reframing has practical consequences for architecture design that the paper's experiments make concrete.

The specific conceptual shift is the elimination of the assumption that recognition and generation require different representational formats. Prior work had approached unification from two directions: either start with a recognition model and try to extract generative capabilities (typically failing to produce high-quality reconstructions), or start with a generative model and try to extract recognition features post-hoc (typically underperforming dedicated recognition models). The post-hoc unification works the paper cites (Donahue and Simonyan 2019; Li et al. 2022; Mukhopadhyay et al. 2024; Zheng et al. 2025) all accept the premise that the model was optimized for one task family and must be retrofitted for the other. HUVR's argument is different: if you design the architecture correctly from the start—specifically, by ensuring every token serves both the reconstruction pathway and the recognition pathway—you do not need to retrofit. The representation is natively dual-purpose.

This claim is substantiated by the paper's two most diagnostic findings. First, removing the reconstruction objective degrades compressed representation quality (Table 13: TinTok classification drops from 68.2% to 64.2% without INR prediction). If the reconstruction and recognition objectives were merely compatible, removing reconstruction would leave recognition unchanged. The fact that it hurts recognition means the reconstruction pressure is producing information that distillation alone does not capture—confirming the paper's hypothesis that reconstruction-forces-understanding. Second, the per-patch INR design is the dominant factor in reconstruction quality (Table 6: +24.81 PSNR from patch-wise prediction alone), validating the architectural diagnosis that prior INR hyper-networks' global-weight-token design was the specific bottleneck preventing unified representation, not a fundamental limitation of the hyper-network approach.

The paper also reconciles a tension in the representation learning literature. Pure recognition models (DINOv3, SigLIP 2) compress to extremely low dimensions surprisingly poorly under PCA—the paper shows DINOv3 PCA drops from 84.6% to 16.1% accuracy at 8 dimensions (Table 1). This suggests that these models, despite their strong full-dimensional performance, distribute information across many dimensions in a way that is not amenable to linear compression. HUVR's TinToks, by contrast, survive compression because the training objective explicitly forces information density: every dimension of a TinTok must carry both semantic and pixel-level information, preventing the representation from spreading useful signal thinly across hundreds of dimensions. This finding implies that compression quality is not a byproduct of recognition quality—the strongest recognition models do not produce the most compressible representations—and that explicit compression training (via a reconstruction bottleneck) produces fundamentally different representational geometry.

Which research directions become more attractive? The paper makes architecture co-design for recognition and generation a more central research question. Previously, the two communities operated largely independently, with unification attempts treated as a niche interest. This paper demonstrates that a single model can be competitive on both axes simultaneously (HUVR matches or slightly exceeds DINOv3 on recognition benchmarks), which raises the bar for future recognition-only models: if you can get reconstruction capability for free (or at negligible recognition cost), why would you design a model without it? Conversely, the paper shows that pure reconstruction models (prior INR hyper-networks) leave substantial recognition performance on the table by neglecting semantic training signals, making the case that even compression-focused models should incorporate distillation.

Which directions become less attractive? The paper's results suggest that post-hoc unification—taking a pre-trained generative model and extracting recognition features from it—may be a dead end for achieving competitive recognition. The gap between HUVR's native recognition performance (85.0% ImageNet for ViT-B) and what post-hoc methods achieve from generative models (typically 5-15 percentage points lower, though the paper does not provide a direct comparison to specific post-hoc methods) suggests that recognition requires being an architectural first-class citizen. The information is present in generative models (as the post-hoc literature shows), but extracting it efficiently may be fundamentally harder than designing the architecture to preserve it from the start.

Follow-Up Research This Work Enables

Direct comparison of HUVR's compressed representations against bottleneck-trained recognition models. The paper's central TinTok result—that 32-dimensional HUVR features achieve 79.4% ImageNet accuracy vs. 64.1% for DINOv3 PCA—leaves open whether the advantage comes from the INR reconstruction objective specifically or from training with any bottleneck. A clean experiment: take a ViT-B, add a trainable 32-dimensional bottleneck layer between the encoder and a standard classification head, and train with the same distillation target (DINOv3 ViT-L) but no reconstruction loss. This isolates the effect of the bottleneck from the effect of the INR objective. If the bottleneck-only model achieves, say, 75% accuracy, then most of HUVR's advantage is architectural (the bottleneck itself forces information density). If it achieves 65%, then the INR objective is doing something qualitatively different—perhaps learning a representation whose geometry is fundamentally more compressible. This experiment would distinguish between "bottlenecks are good for compression" (unsurprising) and "INR-based bottlenecks produce fundamentally better compressed representations than classification bottlenecks" (surprising and HUVR-specific).

Multi-teacher distillation with controllable semantic axes. Table 7 shows that different teachers optimize different metrics: SigLIP 2 gives best reconstruction (26.61 PSNR for ViT-B) and ObjectNet accuracy (68.4%), C-RADIOv3 gives best segmentation (43.32 mIoU), DINOv3 gives best balance. The paper explicitly states that "a truly optimal method would probably distill from a mixture of teachers" but does not implement this. A follow-up would train HUVR with a weighted combination of distillation targets—say, 50% DINOv3 + 30% C-RADIOv3 + 20% SigLIP 2—and measure whether the resulting model achieves the best of all teachers or a compromised average. More ambitiously, one could train with per-block teacher selection: distill encoder outputs to a teacher that excels at dense tasks (C-RADIOv3), distill global token to a teacher that excels at classification (DINOv3), and distill decoder outputs to a teacher that produces reconstruction-friendly features (SigLIP 2). This would test whether different representational levels benefit from different semantic guidance, and whether the architecture can integrate them without destructive interference. A strong negative result (mixture distillation produces worse results than any single teacher) would be valuable for understanding the limits of multi-signal training.

Scaling the TinTok dimension to find the Pareto frontier of compression vs. accuracy. The paper evaluates TinToks at 8, 16, and 32 dimensions for ViT-B and 32 dimensions for ViT-L. The curve of accuracy vs. dimension appears roughly logarithmic (large gains from 8 to 16, smaller gains from 16 to 32), but three points cannot characterize the frontier. A follow-up would measure TinTok classification accuracy, segmentation mIoU, and reconstruction PSNR at 4, 8, 16, 32, 64, 128, 256, and 512 dimensions, producing scaling curves that reveal (a) where accuracy saturates relative to full-dimensional performance, (b) whether reconstruction saturates earlier or later than recognition, and (c) the marginal information gain per additional dimension. This would allow practitioners to make quantitative cost-benefit trade-offs: if going from 64 to 128 dimensions buys only +0.5% accuracy but doubles embedding storage, the optimal operating point is clear. Comparing these curves against PCA scaling curves for DINOv3 and SigLIP 2 would reveal whether HUVR's advantage is consistent across dimensions or concentrated at extreme compression ratios (the current data suggest the gap narrows as dimension increases, but where it disappears is unknown).

Domain transfer and robustness stress-testing of TinToks. The paper's evaluation is entirely in-distribution (ImageNet variants, ADE20K, NYUv2). A critical follow-up would test whether TinToks' information density comes at the cost of brittleness. Specific experiments: (1) Domain shift: freeze a HUVR encoder trained on ImageNet22k/DataComp, extract TinToks for images from a substantially different domain (e.g., medical images from CheXpert, satellite images from EuroSAT, document images from RVL-CDIP), and train linear probes. Compare to PCA-compressed DINOv3 and SigLIP 2 features on the same domains. If TinToks' advantage disappears or reverses under domain shift, it suggests the INR-based compression overfits to the training distribution's pixel statistics. (2) Corruption robustness: evaluate TinTok classification on ImageNet-C (corruptions) and ImageNet-R (renditions) at various compression levels. The hypothesis to test: compressed representations may be more robust (they discard high-frequency information that correlates with pixel-level corruptions) or less robust (the reconstruction pressure forces sensitivity to pixel-level details that change under corruption). (3) Adversarial robustness: perform a white-box adversarial attack on the linear probe trained on TinToks vs. PCA baselines. The reconstruction objective's sensitivity to pixel-level changes might make TinToks more vulnerable to small perturbations, which would be a significant deployment concern.

Video and 3D extensions of the per-patch INR hyper-network design. The paper's architecture is 2D-image-specific, but the design principle—spatially-grounded patch tokens serving as weight tokens for local INR prediction, with global context injected via outer product modulation—has natural extensions. For video: each spatio-temporal tube (a patch tracked across frames) could get its own INR predicting (r, g, b, t) values, with the global token providing clip-level context. The key question is whether the outer product modulation g × p^T should operate over spatial patches, temporal tubes, or both (a tensor product). A concrete starting point: replace the 2D patch embedding with a 3D tube embedding (ViT-style spatial patches + temporal position encoding), keep the global token, and have the decoder predict per-tube INR modulation matrices. Evaluate on video reconstruction (UCF-101, Kinetics-400) and video classification (linear probe on frozen features). This experiment would test whether the architectural insights generalize or are specific to the 2D statistics of still images. A negative result (patch-wise INRs fail for video because temporal coherence requires global motion information that per-tube INRs cannot capture) would clarify the limits of the approach.

Verifier-over-optimization-style analysis of the distillation-reconstruction trade-off. The paper documents trade-offs between reconstruction quality and recognition accuracy across multiple design axes (distillation targets, block selection, decoder attention), but does not characterize these as a formal multi-objective optimization problem. A follow-up could explicitly map the Pareto frontier by training multiple HUVR variants with different weights on the distillation loss (λ from 0.1 to 10.0) and measuring both PSNR and ImageNet accuracy. This would reveal the shape of the frontier: is it convex (smooth trade-off everywhere), or are there "knees" where small sacrifices in one metric buy large gains in the other? More importantly, it would test whether the Pareto frontier shifts with training duration—the paper's Figure 3 shows both metrics improving with time, but does the frontier itself improve (the entire curve shifts upward), or do the metrics converge to a fixed trade-off line? Understanding this would inform practitioners about whether longer training resolves the trade-offs or merely moves along a fixed frontier.

Practical Applications and Downstream Use Cases

Large-scale image retrieval with compressed embeddings. The paper's most directly actionable result is that 32-dimensional TinToks achieve 79.4% ImageNet accuracy (ViT-B) with a PSNR of 27.83—an embedding small enough that one billion images would occupy only 128 GB (32 dimensions × 4 bytes × 1B). For comparison, standard ViT-B embeddings (768 dimensions) would require 3 TB for the same dataset. In a production retrieval system where storage and I/O dominate cost, switching from 768-dimensional features to 32-dimensional TinToks reduces the embedding index size by 24× while retaining strong classification accuracy and, crucially, the ability to approximately reconstruct retrieved images on-the-fly from the embeddings alone. This enables a "search and preview" workflow: a user queries with text or an image, the system retrieves top-k matches using TinTok similarity, and then reconstructs thumbnails directly from the retrieved TinToks without accessing the original image store. The reconstruction quality at 32 dimensions (27.83 PSNR, 0.78 SSIM) is sufficient for visual preview—enough to recognize objects and scenes—though not for archival-quality reproduction.

On-device unified vision pipelines for mobile applications. A ViT-B HUVR model (~136M parameters total, ~86M for the encoder) fits within the capabilities of modern mobile GPUs and NPUs. The architectural design enables a single forward pass to produce three outputs simultaneously: (1) a global token for image-level classification (85.0% ImageNet accuracy), (2) patch tokens for dense tasks like portrait segmentation or depth estimation (52.0 mIoU on ADE20K, 0.3263 RMSE on NYUv2), and (3) a full reconstruction pathway for tasks like image compression or thumbnail generation. No separate models are needed. The compressed TinToks (32 dimensions, ~79% classification accuracy) could serve as a persistent on-device cache: the phone stores TinToks for all photos, enabling fast search and duplicate detection without keeping full-resolution images in active memory. The full-resolution reconstruction is available on-demand via the INR decoding pathway, trading computation (INR evaluation) for storage. This is the paper's "unified universal vision encoding" vision made concrete: one model, multiple embedding sizes, multiple task outputs, all from a single forward pass.

Efficient data curation and deduplication for large-scale training datasets. Dataset curation pipelines (like those used to create DataComp or DINOv3's curated data) require clustering and deduplicating billions of images, a task that becomes computationally prohibitive with full-dimensional embeddings. HUVR's 32-dimensional TinToks offer a 24× reduction in clustering cost relative to standard ViT-B embeddings, while the reconstruction capability provides an additional quality-control mechanism: after deduplication, a human reviewer or automated system can reconstruct candidate duplicate pairs from their TinToks and visually confirm whether they are true duplicates, without accessing the original images. The 27.83 PSNR reconstruction is sufficient for this verification task. The paper's results on FGVC datasets (80–90% accuracy at 32 dimensions on fine-grained categories like birds and cars) suggest that TinToks retain enough detail for the fine distinctions needed in curation. Moreover, the fact that TinToks achieve strong performance without being trained on DINOv3's curated data (Section 4.3) means they could be applied to curate that data in the first place, potentially breaking the circular dependency where curation requires a model trained on curated data.

Compressed video storage with semantic searchability. While the paper does not evaluate video, the per-patch INR design has an immediate extension to video compression: instead of storing keyframes as JPEGs, store them as TinToks (32 dimensions per patch, ~24× compression from pixel space), and reconstruct on-the-fly via INR evaluation. Combined with a temporal interpolation INR (trained separately or as part of the same architecture), this could provide a video codec where every frame's latent is directly searchable—you can query "find all frames containing a golden retriever" by running a linear classifier on the stored TinToks, without decoding to pixels first. The paper's reconstruction quality (27.83 PSNR at 32 dimensions) is not competitive with modern video codecs for distribution-quality video, but for surveillance, autonomous driving logs, or robotics replay buffers—applications where storage dominates and pixel-perfect reconstruction is unnecessary—the trade-off of 24× compression against ~28 PSNR reconstruction with built-in searchability is compelling. This application would require extending HUVR to handle video-specific temporal redundancy, but the architectural foundation (patch-wise INRs with global context) provides a starting point.

When to Prefer This Method

The paper does not articulate an explicit "use HUVR instead of X when Y" framework, but the results imply several decision boundaries that practitioners can operationalize:

  • Prefer HUVR (or its design principles) over a pure recognition model + separate autoencoder when you need both recognition and reconstruction from a single model, and the deployment constraint is model count or memory (one model on device vs. two), not raw generative quality. The paper demonstrates that HUVR's reconstruction PSNR (27.83 at 32 dimensions) exceeds the SD VAE (24.99), meaning for applications where reconstruction fidelity matters more than generation quality, HUVR is already state-of-the-art.

  • Prefer HUVR-style compressed embeddings over PCA-compressed recognition features when your embedding budget is below ~64 dimensions and you need both semantic discrimination and image reconstruction from the compressed representation. Table 1 shows the advantage is largest at extreme compression (48 percentage-point gap at 8 dimensions) and narrows as dimension increases, suggesting a crossover point somewhere between 64 and 256 dimensions where PCA catches up. If your application can tolerate 256-dimensional embeddings, the implementation simplicity of PCA on a pre-trained model may outweigh HUVR's advantage.

  • Prefer dedicated generative models (SD VAE, VQGAN) for the generation pathway when the primary use case is sampling novel images rather than reconstructing input images. Table 2 shows that even at 256 dimensions with extended training, HUVR's FID (24.53) trails the SD VAE (23.05), and the visual examples in Figure 4 exhibit degradations. HUVR unifies recognition and reconstruction, but generation (in the sense of high-quality novel sampling) remains better served by specialized autoencoders.

  • Prefer HUVR with attention-free decoder when compressed embedding quality (TinTok classification) is the primary metric and reconstruction is secondary. Table 10 shows this configuration gains +3.8 percentage points on 32-dimensional ImageNet classification (69.7% → 73.5%) at a cost of −0.45 PSNR. For a retrieval-only deployment, this is the strictly better configuration. Conversely, prefer the full-attention decoder when reconstruction quality matters.