Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution

ArXiv: 2409.12961

🎯 Pitch

Oryx can visually process a 30-minute documentary and a high-resolution newspaper scan with the same architecture, not by forcing them into a fixed-format pipeline, but by dynamically deciding how much resolution and compression each input actually needs. This on-demand strategy allows a 32B model to beat GPT-4o on video benchmarks by preserving native detail only when required, challenging the assumption that more uniform processing is always more general.

---

1. Executive Summary

This paper proposes Oryx, a unified multimodal architecture for spatial-temporal understanding that processes images, videos, and multi-view 3D scenes at arbitrary resolutions and temporal lengths in an on-demand manner. Built around a pre-trained OryxViT encoder (a Vision Transformer with adaptive positional embeddings and variable-length self-attention that preserves native aspect ratios) and a dynamic compressor module (a cross-attention-based downsampling mechanism supporting 1× to 16× token compression), Oryx is evaluated on a comprehensive suite of benchmarks—including VideoMME, MLVU, MMBench, DocVQA, and ScanQA—using 7B and 32B–34B language model backbones. The model establishes new state-of-the-art results among open-source models on multiple video understanding benchmarks, with the 32B variant surpassing 72B competitors on NextQA and Perception Test while achieving a 72.3% score on MLVU that beats GPT-4o by 7.7%, yet this advantage concentrates on tasks where native-resolution encoding and spatial-aware correspondence markers are directly applicable, establishing that on-demand resolution and compression improve multimodal understanding only when visual inputs are processed in their original form rather than through fixed-size partitioning.

2. Context and Motivation

The Core Problem: Uniform Visual Processing in a Diverse World

The fundamental challenge this paper addresses is that contemporary multimodal LLMs process all visual inputs the same way, despite the fact that visual data is extraordinarily heterogeneous. A document image requiring fine-grained text recognition, a 30-second sports clip needing action localization, and a 45-minute documentary requiring narrative summarization present radically different demands—yet existing systems typically force them through identical preprocessing pipelines with fixed resolutions and fixed token budgets.

This uniformity creates a tension between competing requirements. High-resolution encoding preserves the local details essential for recognizing text or small objects, but it generates prohibitively many tokens when applied to long videos (a single hour-long video at high resolution could easily exceed the context windows of even the most capable LLMs). Low-resolution encoding or aggressive frame sampling keeps token counts manageable but discards information—sometimes catastrophically, as when the text in a document becomes illegible or a brief but critical action in a video falls between sampled frames. The paper frames this as an efficiency-precision tradeoff that prior work treats as a fixed design choice rather than a runtime decision to be optimized per-input.

This problem is not merely an inconvenience—it represents a fundamental architectural limitation that prevents MLLMs from being genuinely general. A model that must choose at design time whether to optimize for document reading or video summarization cannot serve both use cases well. The paper's central claim is that this tradeoff is unnecessary: a properly designed architecture can support both extremes simultaneously by making resolution and compression on-demand decisions conditioned on the characteristics of each input.

Why This Gap Matters: Real-World Deployment and the Limits of Scaling

The practical significance of this problem becomes clear when considering how MLLMs are actually deployed. An organization building a single multimodal system might need it to handle customer service screenshots (where text legibility is paramount), surveillance footage analysis (where temporal precision matters), and automated document processing (where both high resolution and multi-page context are needed). Under existing paradigms, the organization must either deploy multiple specialized models—one optimized for images, another for video, perhaps a third for documents—or accept degraded performance on some task classes.

The paper identifies several concrete failure modes of uniform processing, illustrated in Figure 1:

• Text-heavy tasks (Case 1, newspaper description): Low-resolution encoding renders text illegible. The model literally cannot read what it needs to describe. In contrast, native-resolution processing with no compression preserves the document at its original quality, directly enabling accurate description.

• Typical object recognition (Case 2, panda counting): A thumbnail-sized image at modest resolution suffices for counting two pandas. Applying the same high-resolution encoding used for documents wastes computation on pixels that contain no additional discriminative information.

• Fine-grained temporal localization (Case 3, tennis point identification): High per-frame precision (225 tokens per frame) enables identifying which specific player wins a point from a close-up view. Aggressive compression (20 tokens per frame) might retain enough information for a coarse summary but loses the detail needed for frame-level reasoning.

• Long-form summarization (Case 4, 30-minute documentary): Processing with high per-frame token counts would exceed practical context limits. Heavy compression enables the model to ingest the full temporal span, trading spatial detail for temporal coverage—the right tradeoff for summarization tasks.

The computational economics make this gap urgent. As video understanding becomes a larger fraction of MLLM workloads, the cost of uniform high-resolution processing grows linearly with video length. A 30-minute video at 1 frame per second processed at 225 tokens per frame generates 405,000 tokens for the visual modality alone (at 1 fps)—well beyond the context windows of most deployed 7B models. The paper's on-demand approach addresses this by allowing the system to dynamically allocate resolution where it matters and compression where it doesn't, directly reducing inference costs without sacrificing task-specific accuracy.

Prior Approaches and Their Shortcomings

The paper situates its contribution against a progression of visual encoding strategies in MLLMs, each of which addresses part of the resolution-flexibility problem but introduces new limitations.

Fixed-Resolution Encoding (First Generation MLLMs)

Early MLLMs—Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023), LLaVA-1.5 (Liu et al., 2024b), Qwen-VL (Bai et al., 2023)—established the dominant paradigm: resize and crop all visual inputs to a fixed square resolution (typically $336 \times 336$ or $448 \times 448$ pixels), then encode them with a pre-trained CLIP or SigLIP vision encoder. The architectural motivation is clear: Vision Transformers conventionally require fixed-size position embeddings and fixed batch dimensions, so standardizing input sizes simplifies both training and inference.

Where this falls short: Resizing and cropping can destroy task-critical information. A $336 \times 336$ resize of a document image renders most text below the threshold of legibility—the CLIP patch size of $14 \times 14$ pixels means the encoder receives only $24 \times 24 = 576$ patches, each covering roughly $14 \times 14$ pixels of the original document, far too coarse to resolve individual characters. Similarly, a photograph with a 16:9 aspect ratio cropped to a square either loses the edges (where relevant content may reside) or introduces distortion that corrupts the spatial layout the model was trained to interpret. The paper cites prior vision recognition research (Dehghani et al., 2024; Beyer et al., 2023) that demonstrated the benefits of preserving native aspect ratios for pure vision tasks, noting that these insights had "barely been explored in the realm of MLLM" (Section 3.1.1).

Dynamic Partitioning (Second Generation MLLMs)

More recent models—LLaVA-NeXT (Liu et al., 2024c), InternVL2 (Chen et al., 2024b), MiniCPM-V (Yao et al., 2024), LLaVA-UHD (Xu et al., 2024b)—introduced dynamic partitioning as a workaround for the fixed-resolution limitation. Instead of resizing the entire image, these methods divide it into multiple sub-images, each at the native resolution of the pre-trained CLIP encoder (e.g., $336 \times 336$), encode each sub-image independently, and then concatenate the resulting visual tokens. This enables high-resolution perception without modifying the vision encoder itself—the encoder processes familiar fixed-size patches, just more of them.

Where this falls short: The paper identifies two specific deficiencies with this approach (Section 2, visual encoding discussion):

1. Information loss at partition boundaries. When an image is arbitrarily divided into a grid, objects or text that span across partition boundaries get split across separate encoding contexts. The CLIP encoder processes each sub-image independently; it has no mechanism to attend across partition boundaries. This means a word or object bisected by a partition line loses the visual continuity that the original image contained. The paper describes this as the partitioning process potentially "resulting in the loss of critical information present in the original image" (Section 2).

2. Computational inefficiency. Dynamic partitioning inflates token counts proportionally to the number of sub-images. A high-resolution document might require 9 or 16 sub-images, each producing hundreds of tokens, making the approach expensive for long videos where per-frame high-resolution encoding multiplies costs. The approach also offers no mechanism for reducing resolution when a task doesn't require it—the model processes each sub-image at full CLIP resolution regardless of whether the content warrants it.

The paper's critique is subtle but important: dynamic partitioning is fundamentally a compromise, not a solution. It accepts the constraint of a fixed-resolution encoder and works around it by slicing inputs into encoder-compatible pieces. Oryx's alternative—modifying the encoder itself to accept arbitrary resolutions natively—eliminates the need for this compromise entirely.

Fixed Visual Token Budgets and Video Processing

For video understanding, existing approaches typically address the temporal dimension through uniform frame sampling (Section 2, discussion on diverse contexts). Methods like VideoLLaMA2 (Cheng et al., 2024), LLaVA-OneVision (Li et al., 2024a), and LongVILA (Xue et al., 2024) sample a fixed number of frames (e.g., 32 or 64) from each video, encode each frame independently, and concatenate the tokens before feeding them to the LLM. For long videos, this means discarding the vast majority of frames.

Where this falls short: Uniform frame sampling creates a tension between temporal coverage and temporal precision. Sampling 32 frames from a 45-minute movie means the model sees one frame roughly every 84 seconds—it cannot possibly answer questions about events occurring on finer timescales. Increasing the frame count addresses this but linearly increases token counts, hitting context window limits. The paper notes that this approach "treats all inputs equally" and "leads to inefficient computational costs" (Section 3.1.2), since frames that contain little task-relevant information receive the same encoding budget as frames that require detailed analysis.

Fixed Compression Ratios

Most existing MLLMs use a single connector (typically an MLP) between the vision encoder and the LLM with a fixed compression ratio—usually 1:1 (no compression), meaning each visual patch produces one token for the LLM. Some recent work explores token reduction via learned queries (Q-former approaches) or convolutional downsampling, but these apply a uniform compression ratio to all inputs. The paper observes that this is "non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents" (Abstract). A 30-second video clip and a 2-hour movie currently receive the same per-frame compression, even though the former could benefit from high-fidelity encoding and the latter requires aggressive compression just to fit in context.

Unifying the Gaps: The Missing Capability

Stepping back, the three limitations—fixed resolution, fixed frame sampling, fixed compression—share a common root cause: existing MLLMs hardcode visual processing decisions at architecture design time rather than making them conditional on the input at runtime. The paper's diagnosis is that this hardcoding stems from two architectural dependencies:

1. Vision encoders trained on fixed-size inputs cannot natively handle arbitrary resolutions. Attempts to feed differently-sized images to a standard CLIP or SigLIP encoder produce significant accuracy degradation because the positional embeddings are not designed for interpolation and the self-attention implementation assumes uniform sequence lengths within a batch.

2. Connector modules designed for fixed token counts do not support dynamic compression ratios. An MLP that maps each visual patch to one LLM token has no mechanism for merging or discarding patches based on input characteristics.

Oryx addresses both dependencies simultaneously: OryxViT removes the fixed-resolution constraint by incorporating adaptive positional embeddings and variable-length self-attention, enabling the encoder itself to process images at native resolution. The dynamic compressor removes the fixed-compression constraint by introducing a cross-attention mechanism that can downsample at ratios from 1× (no compression) to 16× (aggressive compression) based on task needs.

How the Paper Positions Itself

The paper positions Oryx not as an incremental improvement to any single component, but as a unified framework that solves the resolution-flexibility problem at the architectural level rather than through post-hoc workarounds. Several elements of this positioning are explicit:

Against the partitioning paradigm. The paper directly contrasts native resolution encoding (OryxViT) with dynamic partitioning (LLaVA-NeXT et al.), arguing that the former is more principled because it preserves the original image's spatial structure and avoids boundary artifacts. The ablation in Table 5(a) provides the empirical backing: SigLIP with dynamic partitioning achieves 74.8% on DocVQA, while SigLIP with native resolution input collapses to 17.1%—demonstrating that the native-resolution approach only works when the encoder itself is designed for it (OryxViT with native resolution achieves 78.5%). This is not merely a better encoder; it enables a different processing paradigm.

Against modality-specific architectures. The paper emphasizes that Oryx handles images, videos, and 3D multi-view data within a single architecture, contrasting with approaches that train separate video-specific components (e.g., separate temporal encoders, separate projectors for video tokens). The unified design is enabled by the fact that native-resolution encoding and dynamic compression apply uniformly across modalities—a video is simply a sequence of images, each processed at its native resolution with compression applied as needed. The paper's ablation on shared vs. separate MLP adapters (Table 8) further supports this: a shared MLP projector outperforms separate image and video projectors (55.4 vs. 54.0 on VideoMME), suggesting that the unified design enables beneficial cross-modal transfer.

Against static compression. The dynamic compressor is explicitly contrasted with fixed downsampling strategies (average pooling, convolutional downsampling, Q-former). The paper shows (Table 9a) that the dynamic compressor's cross-attention mechanism outperforms a pure average-pooling baseline (55.4 vs. 54.6 on VideoMME) and dramatically outperforms Q-former approaches (42.7), which the paper argues are fundamentally limited because "the information capacity of a visual token is directly proportional to its length" (Appendix C.5)—fixed-length compression cannot adapt to varying input complexity.

The "on-demand" framing. The paper's central conceptual contribution is reframing the resolution-token tradeoff as an on-demand resource allocation problem rather than an architecture design constraint. Figure 1 visually encodes this philosophy: different tasks demand different points on the resolution-compression Pareto frontier, and an ideal MLLM should be able to operate anywhere on this frontier depending on the input and task. Oryx implements this by decoupling the visual encoding quality (controlled by input resolution) from the visual token budget (controlled by compression ratio), making both axes independently controllable at inference time.

This positioning has a broader implication the paper only hints at: if MLLMs can dynamically adjust their visual processing resolution and compression, the traditional distinction between "image models," "video models," and "document models" becomes obsolete. A single deployment can handle all visual modalities by adapting its processing strategy per-input, which is exactly the capability Oryx aims to demonstrate through its simultaneous strong performance on image, video, and 3D benchmarks.

3. Technical Approach

3.1 Reader Orientation

Oryx is a vision-language architecture that takes images, videos, or multi-view 3D scenes of any size and any length, encodes them into language-model-compatible token sequences, and produces text answers—all without forcing the visual inputs through a fixed-size bottleneck at any stage. The core problem it solves is that existing multimodal LLMs standardize diverse visual data (a document screenshot, a 45-minute movie, a 3D room scan) to identical resolutions and token budgets, which either wastes computation on simple inputs or destroys information in complex ones; Oryx's solution is to make both the resolution at which visual data is encoded and the compression ratio that determines how many tokens reach the LLM into runtime decisions that adapt to each input's characteristics.

3.2 Big-Picture Architecture (Diagram in Words)

Oryx consists of five major components connected in a pipeline:

1. Raw visual input — an image of arbitrary aspect ratio, a video composed of sampled frames, or a set of multi-view 3D images, each at its original pixel dimensions (no forced resize to squares).

2. OryxViT (native-resolution vision encoder) — a Vision Transformer modified with adaptive positional embeddings and variable-length self-attention that encodes each visual input at its native aspect ratio and resolution, producing a feature map whose spatial dimensions are proportional to the original image size rather than fixed to a square grid.

3. Dynamic Compressor (on-demand token compression) — a cross-attention-based downsampling module that receives the OryxViT feature map and produces a compressed feature map at a user-specified ratio (1×, 2×, or 4× pooling), allowing the system to reduce the number of visual tokens by up to 16× for long videos while preserving full resolution for single images.

4. Shared MLP projector — a single multi-layer perceptron that maps compressed visual features from any modality (image, video frame, 3D view) into the embedding space of the language model, with the same weights used across all visual input types to maintain a unified token distribution.

5. Large Language Model — a decoder-only transformer (Qwen-2-7B, Qwen-2.5-32B, or Yi-1.5-34B depending on the variant) that receives the concatenated sequence of visual tokens and text tokens, performs autoregressive generation, and produces the text response.

Information flows sequentially: raw visual input → OryxViT encodes at native resolution → Dynamic Compressor downsamples (ratio chosen per-input) → shared MLP projects to LLM embedding space → LLM generates text conditioned on the combined visual-text token sequence. Crucially, the resolution decision (how many pixels OryxViT sees) and the compression decision (how aggressively the dynamic compressor pools) are independent and per-input: a document image might use 1536-pixel native resolution with 1× compression, while each frame of a 45-minute documentary might use 384-pixel resolution with 4× compression, producing vastly different token counts optimized for each task.

3.3 Roadmap for the Deep Dive

• First, OryxViT's native-resolution encoding — how the vision encoder handles arbitrary input sizes through adaptive positional embeddings and variable-length self-attention, and why existing CLIP/SigLIP encoders fail at this task.
• Second, the Dynamic Compressor — the cross-attention mechanism that enables variable downsampling ratios, how it fuses high-resolution and low-resolution feature maps, and why average pooling outperforms learned downsampling within the compressor.
• Third, the shared MLP projector and modality unification — how a single projection layer processes all visual modalities, the evidence for shared vs. separate projectors, and how this enables joint image-video-3D training.
• Fourth, the data curation and training pipeline — the two-stage training strategy, the long-form needle-in-a-haystack data generation procedure, and the coarse correspondence approach for 3D spatial awareness.
• Fifth, inference-time configuration — how resolution, compression ratio, and frame sampling are chosen per-input at deployment, including the specific pixel and frame limits used in practice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that visual processing in MLLMs should be input-conditional rather than fixed at design time, implemented through two architectural innovations (native-resolution encoding and dynamic compression) supported by targeted data curation for long-form temporal retrieval and 3D spatial reasoning.

---

3.4.1 OryxViT: Native-Resolution Visual Encoding

The foundational problem OryxViT solves is that standard Vision Transformer encoders used in MLLMs—particularly CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023)—are trained on fixed-size square images (typically $N \times N$ pixels) and consequently have two baked-in assumptions that break when presented with arbitrary-resolution inputs: a fixed-size position embedding matrix and uniform sequence lengths across a batch.

Standard ViT processing and its constraints. In a conventional ViT, an input image is first resized to a fixed square resolution $\{x\} \in \mathbb{R}^{N \times N}$, then partitioned into patches of size $p \times p$, yielding a sequence of $\frac{N}{p} \times \frac{N}{p}$ patches. Each patch is linearly projected to a $C$-dimensional embedding, and a learned position embedding $P \in \mathbb{R}^{(N/p) \times (N/p)}$ is added element-wise to encode spatial location. The self-attention layers then process all patches as a uniform-length sequence. When an image at its native resolution $\{x\} \in \mathbb{R}^{H \times W}$ (where $H \neq W$ in general) is presented without resizing, the patch grid becomes $\lfloor H/p \rfloor \times \lfloor W/p \rfloor$, which differs from the $(N/p) \times (N/p)$ grid that $P$ was designed for. The paper notes that "directly resizing $P$ to $\lfloor H/p \rfloor \times \lfloor W/p \rfloor$ can lead to a significant drop in accuracy, as demonstrated in previous works (Dehghani et al., 2024; Beyer et al., 2023)" (Section 3.1.1). This accuracy drop occurs because the interpolated position embeddings no longer correspond to the spatial relationships the model learned during pre-training—a position embedding that originally encoded "center of the image" might, after interpolation, encode a location near the top edge.

OryxViT's architectural modifications. OryxViT introduces two changes to the standard ViT to support native-resolution inputs:

Adaptive positional embeddings. Instead of a fixed-size position embedding matrix, OryxViT incorporates "a sufficiently large position embedding matrix $P$ that accommodates the maximum target input sizes" (Section 3.1.1). For each visual input, the system rescales this large reference embedding to match the actual patch grid dimensions using bilinear interpolation:

$P_{\text{adapted}} = \text{BilinearInterpolate}(P_{\text{large}}, \lfloor H/p \rfloor, \lfloor W/p \rfloor)$

where $P_{\text{large}}$ is a position embedding matrix pre-initialized for the maximum supported resolution, and the interpolation produces a grid of size $\lfloor H/p \rfloor \times \lfloor W/p \rfloor$ that matches the current input's patch layout.

What it computes: the bilinear interpolation takes the reference position embedding defined on a dense grid and samples values at fractional grid positions corresponding to the actual patch centers of the native-resolution image. This produces a position embedding whose spatial layout is proportional to the original image—a patch in the upper-left corner of a $1920 \times 1080$ image gets a position embedding that reflects being in the upper-left, just as it would in a resized $336 \times 336$ image, but with the correct aspect ratio.

Why this form: bilinear interpolation is a standard technique from computer vision for resampling spatial feature maps, and it has the property of being continuous and differentiable with respect to the target grid coordinates. This means the position embedding smoothly varies across the image plane rather than jumping at discrete patch boundaries. The alternative—training separate position embeddings for every possible resolution—would be combinatorially infeasible. The paper builds on prior work (Dehghani et al., 2024; Beyer et al., 2023) that established interpolation as effective for vision-only ViTs, extending it to the MLLM context where the vision encoder's outputs must remain compatible with a separately trained language model.

Variable-length self-attention. The second modification addresses a batch-processing challenge. When images in a training batch have different native resolutions, they produce patch sequences of different lengths $\{N_1, N_2, \ldots, N_b\}$. Standard transformer implementations expect all sequences in a batch to have the same length so that attention can be computed as a single batched matrix multiplication. OryxViT solves this by concatenating all patch sequences along the sequence dimension into a single tensor of shape $[1, \sum_{i=1}^{b} N_i, C]$ and then applying the variable-length attention operator from FlashAttention (Dao et al., 2022). This operator computes self-attention independently within each original sequence by using a cumulative sequence length array to track where each sample's patches begin and end, ensuring patches from different images never attend to each other.

What this enables operationally: OryxViT can process a batch containing a square $1024 \times 1024$ image (producing 4096 patches with $p = 16$), a portrait $480 \times 1280$ document (producing 2400 patches), and a panoramic $2048 \times 512$ image (producing 4096 patches) simultaneously, with each image's self-attention computed correctly within its own patch set. The paper reports that the variable-length attention throughput is comparable to fixed-size processing: "the inference throughput of our arbitrary-resolution visual encoder to remain comparable to the dynamic partition approach used in previous methods" (Appendix C.2), with OryxViT at $1280 \times 1280$ achieving 146.5 images/second vs. 157.7 for SigLIP with dynamic partitioning on an NVIDIA A100 GPU—a 7% slowdown that the paper deems acceptable for the resolution flexibility gained.

The catastrophic failure of SigLIP at native resolution. Table 5(a) contains one of the paper's most revealing results. When the standard SigLIP encoder (trained on fixed-size images) is presented with native-resolution inputs:

• DocVQA accuracy drops from 74.8% (with dynamic partitioning) to 17.1% (with native resolution)
• OCRBench score collapses from 531 to 67
• MMBench falls from 68.0% to 15.8%

This demonstrates that native-resolution encoding is not simply a matter of "not resizing the image"—the encoder itself must be adapted to handle variable input sizes. OryxViT, which was explicitly pre-trained for this capability (details in Section 3.4.4), achieves 78.5% on DocVQA with native resolution, surpassing both SigLIP with partitioning (74.8%) and OryxViT itself with partitioning (76.3%). The paper concludes that "the previous mainstream multi-modal encoder SigLIP fails to process native visual input and only works on fixed resolution with the dynamic partition trick. On the contrary, the OryxViT benefits from the visual inputs at native resolution, which is superior to the partition approach" (Section 4.4).

OryxViT pre-training details. The paper pre-trains OryxViT using a relatively lightweight language model (Qwen2-0.5B) as the language interface, keeping the vision encoder unfrozen while applying LoRA fine-tuning to the language model (Appendix D.2). The total trainable parameters are 0.6B, making this stage "significantly faster than supervised fine-tuning in the main stage (approximately 10 times faster)." The pre-training dataset consists of 400M image-text pairs focusing on image captioning (using CapsFusion; Yu et al., 2024) and OCR tasks (using synthesized OCR data pairs). The training uses a batch size of 2048 with standard cross-entropy loss.

A critical architectural detail: the OryxViT model is built on top of the advanced SigLIP (Zhai et al., 2023) backbone, inheriting its strong vision-language alignment. The modifications (adaptive positional embeddings, variable-length attention) are additions to this base rather than a from-scratch reimplementation. The SigLIP training objective—which uses a sigmoid loss for image-text matching rather than the softmax-based contrastive loss of CLIP—is preserved, giving OryxViT a strong starting point for vision-language alignment.

---

3.4.2 Dynamic Compressor: On-Demand Token Compression

Once OryxViT produces a feature map at native resolution, the system faces a second decision: how many tokens to pass to the language model. A $1536 \times 1536$ image with a $16 \times 16$ patch size produces 9216 visual tokens—a substantial fraction of a 7B model's context window. For a 30-minute video at 1 fps, this becomes $9216 \times 1800$ tokens, which is infeasible. Conversely, a simple $288 \times 288$ image might need only 324 tokens. The Dynamic Compressor allows the system to choose a compression ratio per-input to match the task's precision requirements with the available token budget.

Architecture walk-through. The Dynamic Compressor takes as input the high-resolution feature map $f_H \in \mathbb{R}^{N \times C}$ produced by OryxViT, where $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ is the number of patches and $C$ is the feature dimension. It consists of two stages (Section 3.1.2):

Stage 1: Spatial downsampling. A pooling layer with a configurable stride reduces the spatial dimensions of the feature map:

$f_L = \text{Pool}_{r \times r}(f_H)$

where $r$ is the downsampling ratio (1, 2, or 4 in the paper's implementation) and $f_L \in \mathbb{R}^{N/r^2 \times C}$ is the low-resolution feature map. Three separate pooling configurations are maintained:

• $d_1$: 1×1 pooling (no compression) for pure images, preserving all spatial detail
• $d_2$: 2×2 pooling for short video frames, reducing tokens by 4×
• $d_3$: 4×4 pooling for long video frames, reducing tokens by 16×

The paper refers to these as "downsample layers $d_1$, $d_2$, $d_3$ respectively" and specifies that "the downsampling ratio $r_1 < r_2 < r_3$" is maintained (Section 3.1.2).

Why pooling rather than learned downsampling? Table 9(b) compares average pooling, depthwise convolution (DWConv), and Conv-MLP within the compressor. Average pooling achieves 55.4 on VideoMME and 59.3 on MLVU, outperforming DWConv (55.0, 58.9) and Conv-MLP (54.7, 58.5). The paper hypothesizes that "the parameter-free nature of average pooling preserves the distribution of visual features, and more complex downsampling layers may not be effectively trained through the current training pipeline" (Section 4.4). This is a practical tradeoff: learned downsampling could theoretically adapt to different content types, but the training data and optimization may not be sufficient to learn better pooling strategies than uniform averaging.

Stage 2: Cross-attention fusion. The key innovation of the Dynamic Compressor is that it does not simply discard the high-resolution information after pooling. Instead, it uses a cross-attention mechanism to let the compressed representation attend back to the original high-resolution features:

$f_L = f_L + \text{Softmax}\left(\frac{\phi_q(Q)\phi_k(K^T)}{\sqrt{d_k}}\right)V$

where:

• $Q = f_L \in \mathbb{R}^{N/r^2 \times C}$ is the query tensor derived from the low-resolution feature map
• $K = f_H \in \mathbb{R}^{N \times C}$ is the key tensor from the high-resolution feature map
• $V = f_H \in \mathbb{R}^{N \times C}$ is the value tensor, also from the high-resolution feature map
• $\phi_q$ and $\phi_k$ are linear projection layers that map queries and keys to a lower dimension $d_k$
• The softmax is computed over the spatial dimension of the key tensor

What it computes: each patch in the compressed feature map $f_L$ computes attention weights over all $r^2$ patches in the corresponding spatial region of the high-resolution feature map $f_H$. For a 2×2 downsampling, each compressed patch attends to the 4 original patches in its 2×2 neighborhood. The attention-weighted sum of the high-resolution value vectors is added back to the pooled representation through a residual connection. This can be understood as the compressed patch asking: "Given my summarized view of this region, which fine-grained details are most relevant to retain?"

Why this form and not other fusion strategies? The paper makes several deliberate design choices in this attention formulation:

1. Residual connection. The attention output is added to $f_L$ rather than replacing it, preserving the pooled representation as a base and using attention to inject high-resolution detail. This prevents the attention mechanism from "forgetting" the pooled information if the attention weights are poorly calibrated.

2. No value projection, no output projection. Standard attention modules project queries, keys, and values into a common space and project the attention output back to the original dimension. The paper "omits the value and output projection layers commonly used in attention modules" (Section 3.1.2) to "maintain the original features from the visual encoder and limit the number of linear projection layers." This means the value vectors used in the attention computation are the raw high-resolution features, not a learned transformation of them.

3. Query and key projection to lower dimension. The projection layers $\phi_q$ and $\phi_k$ reduce the dimensionality before computing attention, making the computation cheaper. The reduced dimension $d_k$ is not explicitly stated in the paper, but the design follows standard practice for efficient attention.

The paper reports that this cross-attention mechanism provides a modest but consistent improvement over pure average pooling: 55.4 vs. 54.6 on VideoMME and 59.3 vs. 57.5 on MLVU (Table 9a). The improvement is larger relative to other learned approaches: Q-former-based compression achieves only 42.7 on VideoMME and 35.3 on MLVU, which the paper attributes to the fact that "Q-former-based methods are not suitable for handling long visual content with fixed lengths of visual tokens. This limitation arises because the information capacity of a visual token is directly proportional to its length."

After compression: shared MLP projection. The compressed feature map $f_L$ is then passed through a shared MLP that projects it into the embedding space of the language model. This MLP is identical for all visual modalities (images, video frames, 3D views) and for all compression ratios. The paper explicitly compares shared vs. separate MLPs in Table 8: using separate MLPs for images and videos (both initialized from the same pre-trained image weights) degrades video performance (VideoMME drops from 55.4 to 54.0, MLVU drops from 59.3 to 54.2) while image performance is largely unaffected (MMBench: 81.4 vs. 81.2; MMMU: 43.9 vs. 43.1). The paper interprets this as evidence that "using separate MLPs negatively impacts video benchmarks, as the dual-projector design can lead to differing distributions for similar data" (Appendix C.4). The shared projector ensures that image and video tokens occupy a common embedding space, enabling the LLM to process them with the same attention mechanisms and transfer knowledge across modalities.

---

3.4.3 Inference-Time Resolution and Compression Configuration

The "on-demand" nature of Oryx is realized through a set of heuristics that map input characteristics to processing decisions at inference time. These are not learned policies but fixed rules based on the paper's analysis of resolution effects (Figure 3) and the task-dependent tradeoffs identified in the introduction (Figure 1).

Resolution configuration for images. Single images are processed at their native resolution with a maximum size constraint: "Image data are maintained at their native resolution, with a maximum size of 1536 pixels" (Appendix D.1). An image whose longer side exceeds 1536 pixels is scaled down to fit within this bound while preserving aspect ratio. The paper's resolution ablation (Figure 3) justifies this: increasing from 768² to native resolution provides substantial gains on text-heavy benchmarks (DocVQA, OCRBench) but further increasing to 2× native area yields minimal additional benefit, suggesting that 1536 pixels on the longer side captures most of the useful detail while keeping token counts manageable.

For Stage 1 image training, a lower limit of 1280² pixels was used "for efficiency," and pre-training used 768² pixels since the goal was connector alignment rather than full visual understanding. Stage 2 increases the limit to 1536² to handle the longer sequence lengths encountered with temporal and 3D data.

Resolution configuration for video frames. Video frames are processed at a lower resolution than still images: "we normalize each frame to the minimum size of 288² pixels and the maximum size of 480² pixels" (Appendix D.1). This results in per-frame token counts (before compression) ranging from $324 = (288/16)^2$ to $900 = (480/16)^2 = 900$. The lower resolution acknowledges that for most video tasks—action recognition, event understanding, temporal reasoning—individual frame fidelity is less critical than in document analysis or OCR tasks.

Compression ratio selection. The compression ratio is chosen based on the type of visual input (Section 3.1.2, Appendix D.1):

• Pure images: 1× (no compression) — the $d_1$ path, preserving all spatial detail for the highest-fidelity tasks like document reading.
• Short videos and multi-frame 3D data: 2×2 average pooling ($r = 2$) — the $d_2$ path, reducing per-frame token counts by 4× (e.g., a $384 \times 384$ frame goes from 576 tokens to 144 tokens).
• Long videos (needle-in-a-haystack retrieval data): 4×4 average pooling ($r = 4$) — the $d_3$ path, reducing per-frame token counts by 16× (e.g., a $384 \times 384$ frame goes from 576 tokens to 36 tokens).

Frame sampling strategy. For video inputs, the paper uses uniform temporal sampling at 1 frame per second (1 fps) with a maximum of 64 frames for standard videos and 256 frames for long videos requiring high compression (Appendix D.1): "we restrict the frame number to 64 for standard videos of low compression ratio and 256 for long videos of high compression ratio." When a video has more frames than the limit at the chosen sampling rate, the system "uniformly samples the frames among all the frames if the number exceeds the upper bound." This means a 30-minute video (1800 seconds at 1 fps = 1800 frames) would be uniformly subsampled to 256 frames for long-form processing, yielding roughly one frame every 7 seconds.

Token budget analysis. The combination of resolution constraints and compression ratios determines the total visual token budget per input:

• A single document image at 1536² with 1× compression produces up to $96 \times 96 = 9216$ visual tokens (with $p = 16$).
• A 30-second video at 1 fps with 64 frames, each at 480² and 2× compression, produces $64 \times (30 \times 30) / 4 = 64 \times 225 = 14,400$ tokens.
• A 45-minute movie at 1 fps downsampled to 256 frames, each at 384² and 4× compression, produces $256 \times (24 \times 24) / 16 = 256 \times 36 = 9216$ tokens.

Remarkably, Oryx's design produces comparable token counts for a single high-resolution image and a heavily compressed 45-minute movie, demonstrating the flexibility of the on-demand approach. The maximum sequence length during training is set to 8192 tokens for Stage 1 and 16384 for Stage 2, accommodating these visual token budgets alongside the text.

---

3.4.4 Data Curation and Training Pipeline

Oryx employs a two-stage training strategy that first establishes image understanding capabilities and then extends to video and 3D modalities through joint training. The training data is entirely sourced from open-source datasets, and the paper emphasizes that no proprietary or exclusive data is used.

Stage 1: Image-only training. Stage 1 consists of two sub-phases:

Pre-training (connector alignment). The Dynamic Compressor and shared MLP projector are trained on 558K image caption pairs from the BLIP model, following the LLaVA-1.5 recipe (Liu et al., 2024b). Only the connector parameters are unfrozen; OryxViT and the LLM remain frozen. The training configuration uses: total batch size 256, learning rate $1 \times 10^{-3}$, image resolution fixed at 768² pixels for efficiency. This phase is described as "lightweight thanks to the small number of parameters for the connector and the relatively lower image-text data pairs" (Appendix D.3).

Supervised fine-tuning (SFT). The full model (minus the frozen vision encoder) is trained on 4.1M curated image-text pairs sourced from LLaVA-NeXT (Liu et al., 2024c), Cauldron (Laurençon et al., 2024), and Cambrian-1 (Tong et al., 2024). The paper emphasizes that "we do not incorporate large-scale pre-training stages as described in Li et al. (2024a) or employ exclusive supervised fine-tuning data such as those in Lin et al. (2023b); Bai et al. (2023), as our primary objective is to validate the effectiveness of our unified Oryx architecture" (Section 3.2.2). This is an important caveat: Oryx's performance should be evaluated relative to models trained on comparable (open-source) data volumes, and the paper's results likely represent a lower bound on what the architecture could achieve with larger-scale proprietary data.

Training configuration: OryxViT frozen, connector and LLM unfrozen. Learning rate $2 \times 10^{-5}$ for 7B models, $1 \times 10^{-5}$ for 34B models. Total batch size 128. Images at native resolution with maximum 1280² pixels. Training conducted on 64 NVIDIA A100-40G GPUs (7B) or 64 NVIDIA A800-80G GPUs (34B). Maximum sequence length 8192 tokens.

Stage 2: Joint image-video-3D training. Stage 2 extends the model to handle temporal and spatial modalities by training on a mixed dataset of approximately 1.2M samples:

• ~600K image samples randomly sampled from the Stage 1 SFT dataset
• ~650K video and 3D samples (described below)
• Total: ~1.2M samples

Training configuration: most hyperparameters identical to Stage 1 SFT. Maximum image pixels increased to 1536². Maximum sequence length increased to 16384. Video frames restricted to 64 (standard) or 256 (long-form), uniformly sampled at 1 fps. The learning rate is $2 \times 10^{-5}$ for 7B models and $1 \times 10^{-5}$ for 34B/32B models.

Video data sources. The video training data is assembled from multiple open-source repositories:

• Comprehensive QA and captioning: VideoChatGPT-Plus (Maaz et al., 2024), ShareGPT4Video (Chen et al., 2024a), LLaVA-Hound (Zhang et al., 2024b)
• Multiple-choice benchmarks: Cinepile (Rawal et al., 2024), NextQA (Xiao et al., 2021), PerceptionTest (Pătrăucean et al., 2024)
• Long-form needle-in-a-haystack: custom-generated data from MovieNet (described below)

Long-form temporal training with needle-in-a-haystack data. The paper recognizes that standard video QA datasets rarely require the model to locate specific information within very long contexts—the defining challenge of long-form video understanding. To address this, the authors design a synthetic data generation procedure inspired by the "needle-in-a-haystack" evaluation paradigm from NLP (Section 3.2.1):

1. Source material. Video samples are drawn from MovieNet (Huang et al., 2020), a dataset of full-length movies averaging 1000 frames per movie and 45 minutes of duration. These provide natural long-form visual contexts with coherent narrative structure.

2. Frame annotation. Individual frames (or pairs of frames for the "differing" task) are extracted from these movies and captioned using a state-of-the-art LLM (GPT-4o; OpenAI, 2024). The captions describe the visual content at those specific temporal indices.

3. Reinsertion into context. The annotated frames are reinserted into their original positions within the full movie sequence. The model is then trained to answer questions about these specific frames given the complete movie context. Two task types are generated:

   • Captioning task: "generate captions for frames at specific indices" — the model must locate the target frame within the long sequence and describe it
   • Differing task: "identifying differences between two frames given their indices" — the model must locate two specific frames, compare them, and articulate the differences

4. Training integration. These generated samples are included in the Stage 2 video training mixture, providing explicit supervision for long-range retrieval. The paper reports that this data is processed through the 4×4 pooling path ($d_3$), enabling the model to handle the extreme token counts of full-length movies.

Why this approach matters. The needle-in-a-haystack training directly addresses a capability that standard video QA datasets do not test: the ability to maintain attention over very long temporal distances. Standard video benchmarks like NextQA or PerceptionTest involve videos of 30–180 seconds, where all frames are typically within the model's effective attention span. A 45-minute movie at 1 fps with 256 frames requires the model to attend across 256 visual contexts—far more than typical training distributions. The synthetic data teaches the model that information can be arbitrarily positioned within the visual sequence and that it must scan the entire input to answer questions.

Learning spatial-aware knowledge via coarse correspondences. For 3D understanding, the paper builds on prior work (Liu et al., 2024a) that introduced coarse correspondence markers into multi-view image training. The core concept is to assign a consistent label to the same physical object across different camera views, enabling the model to learn that "object A in view 1" and "object A in view 2" refer to the same entity in 3D space:

1. Tracking model. The authors use Track-Anything (Yang et al., 2023b) as the tracking model to generate coarse correspondences across multi-view images from the ScanQA (Azuma et al., 2022) training set.

2. Label assignment. For each object that appears in multiple views, Track-Anything identifies the corresponding image regions and assigns a consistent numeric label across views. These labeled images form the training data.

3. Training integration. The labeled multi-view samples are included in Stage 2 training. The model learns to use these labels to reason about 3D spatial relationships—for example, determining that "the chair to the left of the door in the first image" corresponds to "the chair visible through the doorway in the second image."

Table 7(b) ablates the contribution of each component: adding 3D data without coarse correspondences improves ScanQA METEOR from 11.7 to 12.8, while adding coarse correspondences further improves it to 14.0. Similarly, ROUGE-L improves from 28.1 (no 3D data) to 32.7 (3D data without correspondences) to 35.1 (with correspondences). The paper notes that "both components contribute to an improved understanding results" (Appendix C.3).

Why coarse correspondences rather than explicit 3D coordinates? The paper does not provide 3D point cloud data or depth maps to the model. Instead, the correspondence markers operate purely in the 2D image space—they are visual labels that indicate which pixels correspond to the same object across views. This approach has the practical advantage that it requires no specialized 3D data formats: the input remains a set of 2D images, just with consistent object labels across views. The LLM is expected to learn 3D spatial reasoning indirectly, through the linguistic supervision that connects the labeled views to questions about spatial relationships.

---

3.4.5 Summary of Design Choices and Their Justifications

• Native-resolution encoding over fixed-size partitioning: eliminates information loss at partition boundaries and produces a continuous feature map that respects the original image's spatial layout. The ablation (Table 5a) shows that native resolution with OryxViT (78.5% DocVQA) outperforms partitioning with the same encoder (76.3%), confirming that the partition boundaries themselves cause measurable degradation.

• Bilinear interpolation for positional embeddings over learned per-resolution embeddings: avoids a combinatorial explosion of embedding matrices while providing smooth, spatially consistent position signals across arbitrary resolutions. The alternative (training on multiple fixed resolutions, as in FlexiViT) would require enumerating and training on a set of target sizes, which is less general.

• Variable-length attention over padding to maximum length: avoids wasting computation on padding tokens and enables genuinely heterogeneous batches. The FlashAttention implementation makes this efficient enough to be practical (only 7% slower than fixed-size encoding).

• Average pooling with cross-attention over learned downsampling: the parameter-free average pooling preserves the feature distribution and is robust to limited training data, while the cross-attention provides a lightweight mechanism for recovering fine-grained detail. The ablation (Table 9b) shows that learned alternatives (DWConv, Conv-MLP) underperform, likely due to training data insufficiency.

• Shared MLP projector over separate per-modality projectors: ensures a unified token embedding space that enables cross-modal knowledge transfer. Separate projectors (Table 8) hurt video benchmarks without improving image benchmarks, suggesting the shared projector benefits from the larger effective training set.

• Needle-in-a-haystack training over uniform video QA data alone: provides explicit supervision for the long-range retrieval capability that distinguishes long-form video understanding from standard video QA. Standard datasets rarely require attending to information separated by thousands of frames.

• Coarse correspondence markers over explicit 3D coordinates: enables 3D spatial reasoning through purely 2D supervision, avoiding the complexity of 3D data formats while providing the key signal (object identity across views) needed for spatial inference.

• Two-stage training over end-to-end joint training: allows the image understanding foundation to be established before introducing the complexity of temporal and 3D modalities. The Stage 1 image SFT uses 4.1M samples, providing a strong visual-language base that Stage 2 builds upon with only 650K additional video/3D samples.

4. Key Insights and Innovations

Innovation 1: Reframing Resolution Flexibility as an Encoder Architecture Problem Rather Than a Preprocessing Workaround

The paper's most conceptually significant move is diagnosing why prior MLLMs struggle with arbitrary-resolution inputs and identifying the root cause as a mismatch between the vision encoder's architectural assumptions and the diversity of visual data. Before Oryx, the field had largely converged on a specific workaround: dynamic partitioning, where high-resolution images are sliced into fixed-size sub-images that a standard CLIP or SigLIP encoder can process. This approach, used by LLaVA-NeXT (Liu et al., 2024c), InternVL2 (Chen et al., 2024b), MiniCPM-V (Yao et al., 2024), and others, implicitly accepts that modifying the vision encoder itself is too difficult or unnecessary—instead, the problem is solved at the preprocessing level by decomposing images into encoder-friendly pieces.

Oryx makes the opposite bet: that the vision encoder should be modified to natively accept arbitrary resolutions, and that doing so eliminates the fundamental limitations of the partitioning approach. This is a conceptual reframing rather than merely an architectural improvement. The paper identifies that dynamic partitioning introduces boundary artifacts (objects bisected by partition lines lose visual continuity across independently encoded sub-images) and offers no mechanism for reducing resolution when a task does not require it. These are not implementation bugs but inherent limitations of solving the resolution problem at the preprocessing level rather than the encoder level.

The empirical evidence for why this reframing matters is starkest in the ablation that tests SigLIP—the standard encoder used in partitioning-based MLLMs—at native resolution (Table 5a). SigLIP collapses from 74.8% to 17.1% on DocVQA when presented with native-resolution inputs instead of fixed-size partitioned inputs. This is not a modest degradation; it is catastrophic failure. The implication is that the partitioning workaround is not merely a convenience for efficiency—it is necessary for standard encoders to function at all on variable-resolution inputs. The dominant assumption in the field—that CLIP/SigLIP encoders can be used as-is with preprocessing tricks—is revealed as a constraint that fundamentally limits what MLLMs can do. OryxViT, by incorporating adaptive positional embeddings and variable-length self-attention directly into the encoder architecture, achieves 78.5% at native resolution, demonstrating that the resolution-flexibility problem is solvable at the architectural level and that doing so yields better performance than working around the limitation.

The significance of this reframing extends beyond the specific OryxViT implementation. If the field accepts that vision encoders for MLLMs should natively support arbitrary resolutions—rather than relying on partitioning tricks—it changes the direction of encoder research and pre-training. Future encoders would be designed from the start with variable-length inputs, adaptive position embeddings, and multi-resolution training objectives, rather than being retrofitted after pre-training. The paper's demonstration that OryxViT pre-training (with a small 0.5B language model and 400M image-text pairs) is sufficient to achieve this capability provides a template for how to build such encoders at scale.

Innovation 2: Separating Resolution Quality from Token Budget as Independent Inference-Time Decisions

A second conceptual contribution is the architectural decoupling of how much visual detail the encoder sees (determined by input resolution) from how many tokens the LLM receives (determined by the compression ratio). In virtually all prior MLLMs, these two quantities are rigidly linked: each visual patch produces exactly one visual token, so increasing resolution necessarily increases the token count proportionally. This creates an uncomfortable zero-sum tradeoff where improving visual fidelity comes at the direct cost of consuming more of the LLM's limited context window—forcing designers to choose between high-resolution encoding that exhausts the context budget and low-resolution encoding that misses critical details.

Oryx breaks this coupling through the Dynamic Compressor. An image can be encoded at high resolution (e.g., 1536² pixels producing 9216 patches) and then compressed before reaching the LLM (e.g., 4×4 pooling reducing it to 576 tokens). Conversely, a video frame can be encoded at modest resolution (e.g., 384² pixels, 576 patches) with no compression, or at even lower resolution with aggressive compression for long videos. The resolution and compression decisions become independent axes: resolution determines the quality of visual information extracted by OryxViT, while compression determines the communication bandwidth to the LLM.

This separation matters because different visual inputs demand different points on the resolution-compression plane (Figure 1). A document image needs high resolution to render text legible but can tolerate compression if the text regions survive the pooling operation. A 30-second sports clip might need moderate resolution with low compression for frame-level action recognition. A 45-minute documentary needs aggressive compression to fit within context limits, with resolution set just high enough to preserve scene-level semantics. Prior approaches cannot make these distinctions—a model designed with fixed resolution and fixed compression must choose a single operating point that compromises across all input types.

The evidence that this decoupling is practically significant comes from the needle-in-a-haystack experiment (Figure 10). At 1.6K frames, the baseline LLaVA-NeXT-Video-7B—which uses fixed per-frame token counts—fails to retrieve inserted images entirely, suffering from "severe information loss." Oryx-7B, using aggressive 4× compression on video frames, successfully locates and answers questions about these inserted images even at 1.6K frames. This is possible specifically because the compression ratio was chosen independently of the resolution: the model sees many frames at adequate (not maximal) resolution, rather than being forced to choose between fewer high-resolution frames or many frames at degraded quality. The ability to independently control these two parameters—which prior architectures conflate into a single fixed design choice—is what enables the model to operate across the full range from document-level precision to movie-length temporal coverage.

Innovation 3: Identifying and Addressing the Training Data Gap for Long-Form Retrieval via Synthetic Needle-in-a-Haystack Supervision

The paper makes a diagnostic contribution to long-form video understanding that is conceptually distinct from its architectural innovations: it identifies that standard video training datasets do not teach models the specific skill of locating information at arbitrary positions within very long visual contexts. Most video QA datasets (NextQA, PerceptionTest, VideoChatGPT) feature videos of 30–180 seconds, where all frames are within the model's effective attention span. A model trained exclusively on such data may develop strong short-range temporal reasoning but has no exposure to the needle-in-a-haystack problem—finding a single relevant frame within thousands of frames of irrelevant content. This is a training data gap, not an architectural one, and it persists even if the architecture can technically handle long inputs.

The paper diagnoses this gap explicitly in Section 3.2.1: "The key ability for processing long-form video inputs is the identification of specific information within an extensive context, akin to the 'needle-in-a-haystack' task in the NLP field." Prior long-video models (LongVA, LongVILA, LLaVA-NeXT-Video) had extended architectural support for longer contexts but relied on training data that did not explicitly teach long-range retrieval. The result is models that can process long videos in principle but degrade in practice because they have never been trained to attend across large temporal gaps.

The synthetic data generation procedure—extracting and captioning individual frames from MovieNet movies, then reinserting them into the full movie context with questions targeting those specific frames—is a targeted intervention that directly addresses this gap. It is inspired by the needle-in-a-haystack evaluation paradigm from NLP but adapted for the visual domain. The training tasks (captioning frames at specific indices, identifying differences between specified frame pairs) force the model to develop the capability to scan long visual sequences for position-specific information, rather than relying on the global summarization strategies that short-video training encourages.

Table 7(a) provides the empirical validation: including the long-form needle-in-a-haystack data improves MLVU from 58.1 to 59.3 and VideoMME from 55.2 to 55.4. These are modest absolute gains, but they are measured on benchmarks (MLVU features videos from 3 minutes to 2 hours; VideoMME-Long features 30–60 minute videos) that specifically test the capability the synthetic data targets. The fact that the improvement is consistent rather than dramatic is itself informative: it suggests that the architectural capacity for long-context processing (OryxViT + Dynamic Compressor) provides the foundation, but targeted retrieval training is needed to realize it for localization tasks.

The broader significance of this innovation is methodological: it identifies a class of capabilities that standard training distributions do not cover and shows how synthetic data generation can fill the gap. This is not a one-time fix but a template—future work on long-form video understanding can apply similar needle-in-a-haystack generation to other long-form video sources and task types, systematically expanding the temporal retrieval capabilities of MLLMs.

Innovation 4: Enabling 3D Spatial Reasoning Through 2D Coarse Correspondence Markers Without Explicit 3D Representations

The paper's approach to 3D understanding is conceptually notable for what it does not do: it does not provide the model with point clouds, depth maps, voxel grids, or any explicit 3D representation. Instead, it introduces coarse correspondence markers—consistent numeric labels assigned to the same physical object across multiple 2D camera views—and relies on the LLM to infer 3D spatial relationships from these 2D annotations combined with linguistic supervision.

This represents a fundamentally different bet from prior work on 3D-capable MLLMs. The dominant approach, exemplified by 3D-LLM (Hong et al., 2023), has been to develop specialized 3D encoders that process point cloud data and produce features that can be aligned with language. This requires maintaining separate 3D processing pipelines, handling the complexity of 3D data formats, and bridging the modality gap between 2D pre-trained encoders and 3D inputs. The alternative approach used by general MLLMs adapted to 3D tasks (LLaVA-OneVision, Mantis) treats multi-view images as independent inputs without explicit cross-view correspondence, relying entirely on the LLM's reasoning to connect views.

Oryx's coarse correspondence approach occupies a middle ground that is theoretically motivated and practically lightweight. The theoretical motivation is that the key missing information for 3D reasoning from 2D multi-view images is object identity across views—knowing that "the chair in view 1" is the same chair as "the chair in view 2" enables spatial reasoning about that chair's position relative to other objects. Depth information and 3D coordinates are useful but may not be necessary if the model can track object identity and reason about spatial relationships linguistically. The coarse correspondence markers provide exactly this identity signal in a format (numeric labels overlaid on 2D images) that requires no specialized 3D processing—the input remains a set of 2D images, just with consistent object labels.

Table 7(b) demonstrates the value of this approach incrementally: adding 3D training data without correspondences improves ScanQA METEOR from 11.7 to 12.8, and adding coarse correspondences further improves it to 14.0. The fact that correspondences help beyond simply including 3D data confirms that the identity signal across views is the bottleneck, not just exposure to multi-view imagery. The final Oryx-1.5-32B model achieves a METEOR of 15.3, ROUGE-L of 38.4, and CIDEr of 74.3 on ScanQA, surpassing both specialized 3D models (3D-LLM: 14.5 METEOR, 35.7 ROUGE-L, 69.4 CIDEr) and general MLLMs adapted to 3D (LLaVA-OneVision-72B: 35.8 ROUGE-L).

The broader insight is that spatial reasoning in MLLMs may be achievable through correspondence-level supervision rather than explicit 3D geometry. This matters practically because 2D correspondence data is far easier to acquire than 3D ground truth (Track-Anything can generate it automatically from multi-view videos), and it matters theoretically because it suggests that LLMs can perform spatial inference when given the right relational signals, even without metric depth or coordinate information. Whether this approach generalizes to tasks requiring precise metric spatial reasoning (e.g., "how far is the chair from the door in meters?") remains an open question, but for the qualitative spatial reasoning tested by ScanQA, the correspondence-based approach is sufficient and avoids the complexity of full 3D representations.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a broad set of standard multimodal benchmarks. For general video understanding: VideoMME (Fu et al., 2024a), NextQA (Xiao et al., 2021), MVBench (Li et al., 2024c), Perception Test (Pătrăucean et al., 2024), MMBench-Video (Fang et al., 2024), Video-ChatGPT (Maaz et al., 2023), and Video Detailed Caption (VDC). For long-form video: MLVU (Zhou et al., 2024), LongVideoBench (Wu et al., 2024), and the VideoMME-Long subset (videos from 30 to 60 minutes). For 2D image understanding: MMBench (Liu et al., 2023a), MMMU (Yue et al., 2024), DocVQA (Mathew et al., 2021), OCRBench (Liu et al., 2023b), AI2D (Kembhavi et al., 2016), and TextVQA (Singh et al., 2019). For 3D spatial understanding: the ScanQA validation set (Azuma et al., 2022). All are publicly available, standard benchmarks.

• Base model(s). The Oryx series uses two language model families: Qwen-2-7B and Yi-1.5-34B for the original Oryx (released first), and Qwen-2.5-7B and Qwen-2.5-32B for Oryx-1.5 (the updated variant). The vision encoder is the pre-trained OryxViT (built on SigLIP; Zhai et al., 2023) in all variants. The choice of multiple LLM families and scales (7B, 32B–34B) is deliberate: it demonstrates the architecture generalizes beyond a single LLM and allows competitive comparison against both small-scale models (the 7B tier) and large-scale models (the 30–40B and 72B tiers). The paper states it uses Qwen-2-7B and Yi-1.5-34B for Oryx, and Qwen-2.5-7B and Qwen-2.5-32B for Oryx-1.5 (Appendix D.1).

• Metrics. Each benchmark has its own standard metric, and the paper adopts these without modification. For multiple-choice video benchmarks (VideoMME, NextQA, MVBench, Perception Test): accuracy (%), the fraction of questions answered correctly. For generation-based video benchmarks scored by proprietary models: MMBench-Video uses a GPT-4-1106 evaluator (OpenAI, 2024) producing a score on a 1–2 scale (higher is better); Video-ChatGPT uses a GPT-3.5-0613 evaluator producing scores on several sub-dimensions, with the reported "score" being the average across dimensions; Video Detailed Caption similarly uses GPT-3.5-0613 evaluation. For image benchmarks: MMBench and MMMU use accuracy (%); DocVQA uses exact match accuracy (%); OCRBench uses a composite score; AI2D and TextVQA use accuracy (%). For 3D: ScanQA uses standard captioning metrics—METEOR, ROUGE-L, CIDEr, BLEU-1, and BLEU-2. The paper reports "the main score for each dataset" (Section 4.1), meaning the single number typically highlighted in prior work for that benchmark.

• Baselines. The paper compares against a wide set of prior work, grouped by domain. For video: VideoChat2-HD (Li et al., 2024c, 7B), VideoLLaMA2 (Cheng et al., 2024, 7B and 72B), LLaVA-OneVision (Li et al., 2024a, 7B and 72B), Kangaroo (Liu et al., 2024e, 8B), VideoCCAM (Fei et al., 2024, 9B and 14B), LLaVA-NeXT-Video (Zhang et al., 2024c, 34B), PLLaVA (Xu et al., 2024a, 34B), VILA-1.5 (Lin et al., 2023b, 40B), LongVA (Zhang et al., 2024a, 7B), LongVILA (Xue et al., 2024, 8B). Proprietary models serve as upper-bound references: GPT-4V (OpenAI, 2023b), GPT-4o (OpenAI, 2024), Gemini-1.5-Pro (GeminiTeam, 2024). For image: Deepseek-VL (Lu et al., 2024, 7B), Monkey (Li et al., 2024d, 7B), LLaVA-NeXT (Liu et al., 2024c, 8B and 34B), Bunny-LLama3 (He et al., 2024, 8B), Cambrian-1 (Tong et al., 2024, 8B and 34B), VILA-1.5 (Lin et al., 2023b, 8B and 40B), Idefics2 (Laurençon et al., 2024, 8B), Yi-VL (Young et al., 2024, 34B), VITA (Fu et al., 2024b, 8×7B), LLaVA-OneVision (Li et al., 2024a, 72B). For 3D: 3D-specific models (VoteNet+MCAN, ScanQA, ScanRefer+MCAN, 3D-LLM) and general MLLMs (BLIP2, Flamingo-7B, Mantis-7B, LLaVA-NeXT-Interleave-14B, LLaVA-OneVision-72B). All baselines use their published or reported scores; the paper does not re-evaluate them.

• Generation budget / compute accounting. The paper does not frame its comparisons in terms of a unified generation budget (unlike the prior example paper, which uses "number of generations" as the universal cost metric). Instead, cost is implicitly measured through model size (7B vs. 32B vs. 72B parameters) and, for the speed benchmark in Appendix C.2, throughput (images/second) and GPU memory. The throughput test compares OryxViT (146.5 images/second at 1280 × 1280 on one NVIDIA A100) against SigLIP with dynamic partitioning (157.7 images/second), with OryxViT showing a 7% slowdown. Memory cost is comparable: 49.1 GB for OryxViT vs. 48.7 GB for the SigLIP baseline, both with batch size 4. The paper notes that visual token counts are the main determinant of inference cost, and the dynamic compressor can reduce these by 4× or 16× for long videos—a cost reduction that prior partitioning-based methods cannot achieve because they lack a compression mechanism. However, no end-to-end FLOPs comparison or wall-clock benchmark is reported.

• Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, or confidence intervals. Results are reported as single-point estimates from a single evaluation run on each benchmark's standard test set. The paper also does not describe any hyperparameter search over the evaluation protocol—the inference-time configuration (resolution limits, compression ratios, frame sampling rates) is fixed by the heuristics described in Section 3.4.3 and applied uniformly across all benchmarks. This is a notable methodological gap: the reported scores could be sensitive to random seed, data ordering during training, or specific checkpoint selection, but no robustness analysis is provided.

Main Quantitative Results

General Temporal Understanding

The headline results are in Table 1, which reports scores across seven video benchmarks. The paper emphasizes that Oryx establishes "new state-of-the-art results among open-source models on several benchmarks" (Section 4.1), with the 32B Oryx-1.5 variant outperforming models up to 72B on specific datasets.

Oryx-1.5-32B performance. On the four multiple-choice benchmarks (Table 1, bottom row):

• VideoMME: 67.3 without subtitles, 74.9 with subtitles. This surpasses LLaVA-OneVision-72B (66.2/69.5) by 1.1 and 5.4 points respectively, and beats GPT-4V (59.9/63.3) by substantial margins. It trails Gemini-1.5-Pro (75.0/81.3), a proprietary model with likely far larger scale.
• NextQA: 85.0, surpassing the previous best open-source result (LLaVA-OneVision-72B at 80.2) by 4.8 points.
• MVBench: 70.1, surpassing LLaVA-OneVision-72B (59.4) by 10.7 points.
• Perception Test: 74.0, surpassing LLaVA-OneVision-72B (66.9) by 7.1 points.

On the three generation-based benchmarks, Oryx-1.5-32B scores 1.52 on MMBench-Video, 3.66 on Video-ChatGPT, and 3.63 on Video Detailed Caption. The MMBench-Video score of 1.52 surpasses GPT-4V (1.53) and Gemini-1.5-Pro (1.30). The Video-ChatGPT score of 3.66 exceeds LLaVA-OneVision-72B (3.62) and GPT-4V (4.06). Note that Video-ChatGPT scores are on a scale where higher is better and GPT-4V achieves 4.06—here Oryx is actually below the proprietary baseline.

Comparing Oryx variants. The Oryx-1.5 models (using Qwen-2.5 LLMs) consistently outperform the original Oryx models (using Qwen-2 and Yi-1.5 LLMs) at comparable parameter counts. For example, Oryx-1.5-7B achieves 67.6 on MVBench vs. 63.9 for Oryx-7B, and Oryx-1.5-32B achieves 70.1 vs. Oryx-34B's 64.7. This improvement likely reflects the stronger base LLMs (Qwen-2.5 vs. Qwen-2/Yi-1.5) rather than architectural changes, but the paper does not isolate this factor.

Scale analysis. Moving from 7B to 32B yields consistent but diminishing gains. On VideoMME (without subtitles), Oryx-1.5 improves from 58.8 (7B) to 67.3 (32B), a gain of 8.5 points. On NextQA, the improvement is from 81.8 to 85.0 (3.2 points). On Perception Test, from 70.0 to 74.0 (4.0 points). The gains are more modest on the generation-based benchmarks: MMBench-Video improves from 1.49 to 1.52, and Video-ChatGPT from 3.62 to 3.66—near the ceiling for these metrics. This pattern is consistent with diminishing returns from model scaling on tasks that are already near-saturated.

Comparison to proprietary models. The paper highlights that Oryx "outperforms advanced proprietary models such as GPT-4V and Gemini-1.5-Pro on several of the most challenging benchmarks" (Section 4.1). This claim is specifically supported by:

• MMBench-Video: Oryx-1.5-32B (1.52) > GPT-4V (1.53) > Gemini-1.5-Pro (1.30). Note that on this metric, higher is better, so 1.52 does indeed beat 1.53—but the margin is tiny (0.01).
• NextQA and Perception Test: GPT-4V and Gemini-1.5-Pro are listed with dashes ("–") in Table 1, indicating results were not available or not reported for these benchmarks. The paper's claim of surpassing proprietary models on these benchmarks cannot be verified from the table—it may be that the proprietary baselines were simply not evaluated on these datasets.

The paper does not make a blanket claim of exceeding proprietary models; it specifically says "on several of the most challenging benchmarks." The evidence for this claim is present but limited to the specific benchmarks where comparison was possible, and the margins are often small.

Long-Form Temporal Understanding

Table 2 reports results on three long-form video benchmarks—MLVU, LongVideoBench, and VideoMME-Long—each featuring videos of tens of minutes to hours. The paper's headline claim is that Oryx "surpasses several proprietary models across various benchmarks" (Section 4.2).

MLVU. This is the paper's strongest long-form result. Oryx-1.5-32B achieves 72.3%, which surpasses GPT-4o (64.6%) by 7.7 points and surpasses the best open-source baseline (LLaVA-OneVision-72B at 66.4%) by 5.9 points. Oryx-1.5-7B achieves 67.5%, matching LLaVA-OneVision-72B (66.4%) and exceeding LLaVA-OneVision-7B (64.7%) by 2.8 points. The improvement from 7B to 32B is 4.8 points (67.5 to 72.3), a meaningful gain suggesting that long-form tasks benefit from larger model capacity.

LongVideoBench. Oryx-1.5-32B achieves 62.0%, which surpasses LLaVA-OneVision-72B (61.3%) by 0.7 points and exceeds GPT-4V (60.7%) by 1.3 points. However, it trails GPT-4o (66.7%) by 4.7 points and Gemini-1.5-Pro (64.4%) by 2.4 points. The Oryx-1.5-7B achieves 56.3%, surpassing Kangaroo-8B (54.8%) and LongVA-7B (not listed for this benchmark but likely lower based on its VideoMME performance). The scale improvement from 7B to 32B is 5.7 points (56.3 to 62.0), again indicating that long-form understanding benefits from scale.

VideoMME-Long. Oryx-1.5-32B achieves 59.1 (without subtitles) and 69.7 (with subtitles). This compares to GPT-4V at 53.5/56.9, GPT-4o at 65.3/72.1, and Gemini-1.5-Pro at 67.4/77.4. Oryx-1.5-32B beats GPT-4V by 5.6 and 12.8 points but trails GPT-4o by 6.2 and 2.4 points, and trails Gemini-1.5-Pro by 8.3 and 7.7 points. The Oryx-1.5-7B achieves 51.2/58.3, surpassing the best 7B baselines (LLaVA-OneVision-7B is not listed; LongVA-7B achieves 46.2/47.6).

Comparison across variants. The original Oryx and Oryx-1.5 track each other closely on MLVU (Oryx-7B: 67.5 vs. Oryx-1.5-7B: 67.5, identical; Oryx-34B: 70.8 vs. Oryx-1.5-32B: 72.3, a 1.5-point gain). On LongVideoBench, Oryx-1.5-32B (62.0) slightly trails Oryx-34B (62.2). The consistency suggests the architectural improvements dominate over the LLM upgrade for long-form tasks.

Interpreting the long-form results. The needle-in-a-haystack training data (Section 3.2.1) was specifically designed to improve performance on benchmarks like MLVU and LongVideoBench, which require locating specific information within lengthy visual contexts. The strong MLVU result—beating GPT-4o by 7.7 points—is the clearest evidence that this targeted data curation addresses a genuine capability gap. However, the more modest LongVideoBench and VideoMME-Long results (trailing GPT-4o and Gemini-1.5-Pro) suggest that proprietary models may have other advantages (larger context windows, more extensive long-form training data, stronger base LLMs) that partially compensate for the lack of specialized needle-in-a-haystack training.

2D Spatial Understanding (Image Benchmarks)

Table 3 reports results on six image benchmarks. The headline finding is that Oryx "maintains pioneering results on image benchmarks" (Section 4.3) while using a simpler training pipeline than competing models.

MMBench. Oryx-1.5-32B scores 86.3%, competing directly with LLaVA-OneVision-72B (85.6%) and exceeding Cambrian-1-34B (81.4%) and VILA-1.5-40B (82.4%). Oryx-1.5-7B scores 81.3%, a strong result for a 7B model but trailing LLaVA-OneVision-72B (85.6%) by a substantial margin.

MMMU. Oryx-1.5-32B scores 56.1%, closely matching LLaVA-OneVision-72B (56.8%) and exceeding VILA-1.5-40B (51.9%) and Cambrian-1-34B (49.7%). This is a significant result because MMMU tests reasoning across multiple disciplines and is considered a challenging benchmark. The scale gain from Oryx-1.5-7B (47.1%) to 32B (56.1%) is 9.0 points, one of the larger scale improvements reported.

DocVQA. Oryx-1.5-32B scores 92.7%, which is very strong but slightly below LLaVA-OneVision-72B (93.1%). Oryx-1.5-7B scores 90.1%, surpassing LLaVA-NeXT-8B (78.2%) by a dramatic 11.9 points. This is the benchmark where native-resolution encoding is expected to provide the greatest advantage, since document reading depends on preserving fine text detail. The large gap between Oryx-7B and similarly-sized baselines (Cambrian-1-8B: 77.8%, LLaVA-NeXT-8B: 78.2%) provides evidence for OryxViT's contribution, though part of the gap may also reflect differences in training data and the underlying LLM.

OCRBench. Oryx-1.5-32B scores 746, surpassing Cambrian-1-34B (600) by 146 points and LLaVA-NeXT-34B (574) by 172 points. Oryx-1.5-7B scores 713, surpassing Cambrian-1-8B (624) and Monkey-7B (534). The OCRBench results are particularly strong relative to the baselines, consistent with the paper's claim that native-resolution encoding preserves text detail that is lost in fixed-resolution or partitioning-based approaches.

AI2D. Oryx-1.5-32B scores 83.2%, trailing LLaVA-OneVision-72B (85.6%) by 2.4 points but exceeding Cambrian-1-34B (79.7%). Oryx-1.5-7B scores 79.7%, a strong result exceeding most 7–8B baselines.

TextVQA. Oryx-1.5-32B scores 78.3%, exceeding Cambrian-1-34B (76.7%) but trailing LLaVA-OneVision-72B (which does not report a number). TextVQA, like DocVQA, tests text reading ability, so the strong relative performance is expected given OryxViT's native-resolution encoding.

Overall image assessment. Oryx achieves tier-1 performance among open-source models across the image benchmarks, but it does not universally dominate. LLaVA-OneVision-72B scores higher on MMBench (85.6 vs. 86.3 for Oryx-1.5-32B—Oryx wins by 0.7), MMMU (56.8 vs. 56.1—Oryx loses by 0.7), DocVQA (93.1 vs. 92.7—Oryx loses by 0.4), and AI2D (85.6 vs. 83.2—Oryx loses by 2.4). These margins are often small, and Oryx uses a smaller model (32B vs. 72B) and, the paper emphasizes, a simpler training pipeline without large-scale pre-training or proprietary SFT data. The competitive but not dominant image results are consistent with the paper's positioning: Oryx's main contribution is the unified architecture that handles images, videos, and 3D simultaneously, not necessarily beating specialized image models at their own game.

3D Spatial Understanding

Table 4 reports results on the ScanQA validation set. The Oryx-1.5-32B model scores 15.3 METEOR, 38.4 ROUGE-L, 74.3 CIDEr, 38.8 BLEU-1, and 24.4 BLEU-2. This surpasses all baselines listed, including:

• Specialized 3D models: 3D-LLM (Hong et al., 2023) scores 14.5 METEOR, 35.7 ROUGE-L, 69.4 CIDEr. Oryx-1.5-32B surpasses 3D-LLM by 0.8 METEOR and 2.7 ROUGE-L.
• General MLLMs: LLaVA-OneVision-72B scores 35.8 ROUGE-L (METEOR not reported). Oryx-1.5-32B surpasses this by 2.6 ROUGE-L. LLaVA-NeXT-Interleave-14B scores 34.5 ROUGE-L (METEOR not reported), which Oryx-1.5-32B surpasses by 3.9.
• Oryx variants: The original Oryx-34B scores 15.0 METEOR, 37.3 ROUGE-L. Oryx-1.5-32B improves by 0.3 METEOR and 1.1 ROUGE-L—a small but consistent gain.

The scale comparison within Oryx shows a modest improvement from 7B to 32B: METEOR increases from 15.2 to 15.3 (0.1), ROUGE-L from 38.4 to 38.4 (identical), and CIDEr from 73.5 to 74.3 (0.8). The near-identical ROUGE-L scores at 7B and 32B are unusual and may indicate that 3D understanding, as measured by these captioning metrics, is not bottlenecked by LLM capacity—perhaps the coarse correspondence mechanism already provides a strong enough spatial signal that additional LLM parameters yield marginal benefit. Alternatively, the ScanQA test set may be too small or too easy to differentiate between 7B and 32B models.

The paper attributes the strong 3D results to the coarse correspondence training (Section 3.2.1) combined with the unified architecture, but it is worth noting that ScanQA is a relatively small and specific benchmark. Whether these results generalize to other 3D reasoning tasks (navigation, object manipulation, spatial QA in dynamic environments) is untested.

Analysis: Effects of Resolution and Resize Strategy

Figure 3 presents an ablation that is central to the paper's core claim about native-resolution encoding. The experiment compares four input configurations: images at fixed 768² pixels, images at fixed 1024² pixels, images at native resolution, and images at 2× native resolution area. The results span three benchmarks (MMBench, DocVQA, OCRBench), and the key pattern is:

• Native resolution consistently outperforms fixed sizes. For example, on OCRBench, native resolution achieves approximately 570 (exact values are read from the bar chart in Figure 3, left panel), while 768² and 1024² achieve approximately 510 and 540 respectively. On DocVQA, native resolution achieves roughly 79%, while 768² achieves roughly 50% and 1024² achieves roughly 65%.

• The text-heavy benchmarks show the largest sensitivity to resolution. DocVQA and OCRBench show much larger gaps between low-resolution (768²) and native resolution than does MMBench, which tests general visual understanding. This is consistent with the paper's claim that text recognition specifically benefits from preserving the original image resolution—lower resolution renders text illegible.

• Beyond native resolution, further enlargement yields minimal gain. The "2× native" configuration (rightmost set of bars in Figure 3, left panel) shows performance roughly comparable to native resolution, with no meaningful improvement on most benchmarks. This is the evidence behind the paper's claim that "further enlarging the resolution does not yield significant gains in most benchmarks" (Section 4.4).

• The resolution-performance curve shows a saturation effect. Figure 3, right panel, plots performance against total pixel count for MMBench and OCRBench. Both show an initial steep increase from low to moderate resolution, followed by a plateau. The native resolution point (marked for reference) sits near the knee of the curve—higher than the moderate fixed sizes but not meaningfully lower than the maximum tested resolution.

The paper also tests a search over "optimal anchor resolution" (Table 5a, last row), finding that selecting the ideal resolution per image yields an additional 0.7 points on DocVQA (79.2% vs. 78.5% for native) and 0.6 points on MMBench (69.9% vs. 69.3%). This suggests that native resolution is close to but not quite the optimal configuration—a learned resolution selection policy might provide small additional gains. The paper explicitly does not use this optimization in primary evaluations "for the sake of fairness and efficiency" (Section 4.4).

Ablation Studies and Robustness Checks

Visual encoder and input resolution (Table 5a): The ablation compares four configurations: SigLIP with dynamic partitioning, SigLIP with native resolution, OryxViT with dynamic partitioning, and OryxViT with native resolution. The key results on DocVQA: SigLIP+partitioning: 74.8%; SigLIP+native: 17.1% (catastrophic failure); OryxViT+partitioning: 76.3%; OryxViT+native: 78.5%. This demonstrates that native-resolution encoding requires an encoder designed for it—standard SigLIP collapses—and that OryxViT with native resolution outperforms OryxViT with partitioning, confirming that partitioning boundaries cause measurable information loss (1.2 percentage points on DocVQA). The same pattern holds on OCRBench (SigLIP+native: 67; OryxViT+native: 572) and MMBench (SigLIP+native: 15.8%; OryxViT+native: 69.3%). This ablation is the single strongest piece of evidence supporting the paper's central architectural claim.

Dynamic compressor design (Table 5b): The ablation compares the Dynamic Compressor against a simple MLP with average pooling, and tests three downsampling functions within the compressor. On VideoMME and MLVU: MLP+AvgPool gives 54.6 and 57.5; Dynamic Compressor+AvgPool gives 55.4 and 59.3; Dynamic Compressor+DWConv gives 55.0 and 58.9; Dynamic Compressor+Conv-MLP gives 54.7 and 58.5. The Dynamic Compressor provides a 0.8-point gain on VideoMME and 1.8-point gain on MLVU over the simple MLP baseline. Within the compressor, average pooling outperforms learned alternatives, which the paper attributes to preservation of feature distributions and insufficient training data for learning effective downsampling. This ablation is conducted on a subset of video training data "to maintain training efficiency," which may limit the generalizability of the learned downsampling results—with more training data or longer training, DWConv or Conv-MLP might eventually surpass average pooling.

Overall downsampling architecture (Table 9a): Compared against average pooling alone (54.6 VideoMME, 57.5 MLVU), convolution alone (54.2, 56.8), and Q-former (42.7, 35.3), the Dynamic Compressor achieves 55.4 and 59.3. The Q-former result is a strong negative finding: fixed-length token compression via learned queries performs dramatically worse than spatial downsampling for long visual contexts, confirming the paper's claim that "the information capacity of a visual token is directly proportional to its length" (Appendix C.5).

MLP adapter design (Table 8): Shared MLP projector vs. separate MLPs for images and videos. Shared: 55.4 VideoMME, 59.3 MLVU, 81.4 MMBench, 43.9 MMMU. Separated: 54.0, 54.2, 81.2, 43.1. The shared projector outperforms on video benchmarks by 1.4 (VideoMME) and 5.1 (MLVU) points, while image benchmarks are nearly identical. This supports the claim that a single projector benefits from joint training and prevents distribution shift between modalities.

Training data ablations (Table 7): Two ablations examine the contribution of data curation choices. (a) Long-form data: including needle-in-a-haystack data improves VideoMME from 55.2 to 55.4 and MLVU from 58.1 to 59.3. The MLVU improvement (1.2 points) is larger than the VideoMME improvement (0.2 points), which makes sense because MLVU features longer videos on average than standard VideoMME. (b) 3D data: adding 3D training data without coarse correspondences improves ScanQA METEOR from 11.7 to 12.8 and ROUGE-L from 28.1 to 32.7; adding coarse correspondences further improves METEOR to 14.0 and ROUGE-L to 35.1. The incremental benefit of correspondences (+1.2 METEOR, +2.4 ROUGE-L over 3D data alone) confirms that cross-view object identity is the key missing signal, not just exposure to multi-view imagery.

Negative Results and Limitations in Ablations

ReST-trained revision model degrades performance (Appendix K, Figure 16 of the paper—though this belongs to the prior example, not Oryx; there is no such experiment in the Oryx paper). The Oryx paper does not report any negative results from attempted architectural variants—all ablations show small but consistent improvements for the proposed components. This uniformity of positive results is somewhat unusual and may reflect selective reporting or the fact that the paper tested only a modest range of alternatives.

Q-former performs dramatically worse than spatial pooling (Table 9a). This is the clearest negative result: Q-former achieves 42.7 on VideoMME and 35.3 on MLVU, compared to 54.6/57.5 for average pooling alone. The paper argues Q-former's fixed token length is fundamentally inadequate for long videos, and this result provides empirical support for that claim. However, the paper does not explore whether Q-former with larger token budgets would close the gap—the comparison may be somewhat unfair if Q-former was constrained to a token budget far smaller than the pooling approaches.

Memory-resolution curve (Figure 11). The paper shows that memory cost grows approximately linearly with image token length (and thus with resolution) on an NVIDIA A100 GPU. At 1000 tokens, memory is roughly 29 GB; at 6000 tokens, memory is roughly 35 GB. The paper notes that the primary memory cost comes from the LLM, not the vision encoder, so even substantial increases in visual resolution have modest memory impact. This is a robustness check rather than an ablation: it confirms that native-resolution encoding does not cause memory explosions that would make the approach impractical.

Needle-in-a-haystack visualization (Figure 10). This experiment tests the extreme end of the compression pipeline by inserting a single relevant frame into a video of up to 1.6K frames and asking the model to answer a question about that frame. Oryx-7B's performance is visualized as a heatmap showing accuracy across frame depths (0.0 to 1.0, x-axis) and total frame counts (0.1K to 1.6K, y-axis). The Oryx heatmap (right subfigure) shows mostly green (correct retrieval) across all depths and frame counts. The LLaVA-NeXT-Video-7B baseline (left subfigure) shows predominantly red (failed retrieval), particularly at deeper positions and higher frame counts. This is a qualitative visualization rather than a quantitative benchmark, but it dramatically demonstrates the information loss that occurs without dynamic compression—the baseline model, trained on 32 frames, simply cannot retain information from a single frame embedded within 1.6K frames of context.

Critical Assessment

The paper makes three central claims that can be evaluated against the experimental evidence:

Claim 1: Native-resolution encoding (OryxViT) enables better visual understanding than fixed-size partitioning. The evidence is strong but narrower than the framing. Table 5a directly demonstrates that OryxViT with native resolution (78.5% DocVQA) outperforms OryxViT with dynamic partitioning (76.3%), which in turn outperforms SigLIP with dynamic partitioning (74.8%). This is a clean, well-controlled comparison that isolates the effect of native resolution vs. partitioning within the same encoder architecture. The result is consistent across DocVQA, OCRBench, and MMBench. The Figure 3 resolution analysis further shows that native resolution consistently outperforms fixed-size resizing and that the gain is largest on text-heavy benchmarks—exactly where the paper's motivation predicts it should be.

However, the evidence has significant limitations:

• The comparison is against a single partitioning strategy. The paper implements its own dynamic partitioning (dividing images into fixed-size sub-images) but does not compare against the specific partitioning strategies used by LLaVA-NeXT, InternVL2, or other models. It is possible that more sophisticated partitioning strategies (e.g., with overlap between sub-images, or with learned partition boundaries) could narrow or close the gap.
• The ablation uses a subset of training data for efficiency. Table 5 states that the image benchmark ablations "use a subset of image training data for efficient training." The absolute numbers in this ablation (SigLIP+partitioning at 74.8% DocVQA) are lower than the numbers reported in the main results (Oryx-7B at 89.0% DocVQA in Table 3), indicating the ablation models are significantly undertrained. The relative pattern (native > partition > SigLIP) may hold at full scale, but the magnitude of the gap at full training is unknown.
• The OryxViT is pre-trained with an MLLM objective (400M image-text pairs with Qwen2-0.5B), while the SigLIP baseline was trained with a contrastive objective. The comparison is therefore not purely about the encoder architecture—it also reflects different pre-training objectives. The paper does not control for this. A fairer comparison would involve training SigLIP (or fine-tuning it) with the same MLLM objective as OryxViT.
• No comparison against other native-resolution encoders. FlexiViT (Beyer et al., 2023), NaViT (Dehghani et al., 2024), and ViTAR (Fan et al., 2024) are cited as prior work that also handles variable resolutions. The paper does not compare OryxViT against these alternatives, making it impossible to determine whether OryxViT's specific implementation (bilinear interpolation, FlashAttention variable-length attention) is superior to other native-resolution approaches. The claim that OryxViT is necessary for native-resolution MLLM performance stands only relative to the (unsurprising) failure of standard SigLIP.

Claim 2: The dynamic compressor enables efficient processing of long visual contexts. The evidence is partially supporting but incomplete.

What is demonstrated:

• Table 5b and Table 9a show that the Dynamic Compressor outperforms simple average pooling and dramatically outperforms Q-former on VideoMME and MLVU. These are standard benchmarks that include videos of varying lengths.
• Figure 10 shows that Oryx-7B can successfully retrieve information from a single frame embedded within 1.6K frames, while a fixed-32-frame baseline fails entirely. This qualitatively demonstrates the value of dynamic compression for extremely long contexts.

What is NOT demonstrated:

• No end-to-end cost-benefit analysis. The paper argues that dynamic compression enables handling longer videos, but it does not show a controlled comparison where Oryx with 4× compression is compared against a model that sees the same frames at a lower resolution or fewer frames at a higher resolution, matched for token budget. Without this, it is unclear whether the compression ratio or the resolution choice (or their interaction) drives the performance.
• No benchmark that specifically isolates compression quality. The video benchmarks (VideoMME, MLVU, LongVideoBench) test overall video understanding, which depends on resolution, frame sampling, compression, the LLM's reasoning ability, and training data. None of these benchmarks is designed to isolate the effect of visual token compression specifically. The improvement from Dynamic Compressor over average pooling (0.8 points on VideoMME, 1.8 on MLVU) could be due to the cross-attention mechanism preserving useful detail, or it could be noise.
• The compression ratio is not swept as an independent variable. The paper uses 1× for images, 2× for standard videos, and 4× for long videos, but there is no experiment showing how performance varies as compression increases for a fixed input type. A curve of accuracy vs. compression ratio (e.g., 1×, 2×, 4×, 8× on the same video benchmark) would be highly informative but is absent.
• No comparison against adaptive frame sampling as an alternative. If the goal is to reduce token count, an alternative to compression is simply sampling fewer frames at higher per-frame quality. The paper does not compare "many compressed frames" against "fewer uncompressed frames" at equal token budgets, leaving open the question of whether compression preserves more useful information than temporal subsampling.

Claim 3: Oryx achieves state-of-the-art open-source performance across image, video, and 3D benchmarks simultaneously within a single unified architecture. This claim is supported, but the specific nature of the SOTA designation requires careful qualification.

What the experiments show:

• Video benchmarks: Oryx-1.5-32B achieves top open-source scores on NextQA (85.0, beating LLaVA-OneVision-72B at 80.2), MVBench (70.1, beating 59.4), Perception Test (74.0, beating 66.9), and MLVU (72.3, beating 66.4). On VideoMME, it is competitive (67.3/74.9 vs. 66.2/69.5 for LLaVA-OneVision-72B). These are clear SOTA results among open-source models on these specific benchmarks.
• Image benchmarks: Oryx-1.5-32B is competitive but not dominant. On MMBench (86.3% vs. 85.6% for LLaVA-OneVision-72B), MMMU (56.1% vs. 56.8%), DocVQA (92.7% vs. 93.1%), and AI2D (83.2% vs. 85.6%), it hovers around parity with or slightly behind the best open-source models, despite using a smaller LLM backbone (32B vs. 72B). The paper's claim of "pioneering results" is accurate for DocVQA and OCRBench but overstated as a general claim about image understanding.
• 3D benchmarks: Oryx-1.5-32B achieves the highest reported scores on ScanQA among all compared models (both specialized 3D models and general MLLMs). The margins over 3D-LLM are modest (+0.8 METEOR) but consistent.

What should give the reader pause:

• Lack of error bars or statistical testing. All results are single-point estimates. On benchmarks with 500 test questions (like the image benchmarks) or similarly small test sets, differences of 1–2 percentage points may not be statistically significant. The paper's claims of "outperforming" should be interpreted with appropriate uncertainty, particularly for comparisons where margins are small (e.g., MMBench: 86.3% vs. 85.6%).
• Different training data across compared models. The paper emphasizes that Oryx uses only open-source data and avoids large-scale pre-training or proprietary SFT data. This is a strength for reproducibility, but it means the comparison against models trained on larger or proprietary datasets (LLaVA-OneVision, Cambrian-1, InternVL2) is not purely architectural—Oryx may be achieving competitive results with less data, or it may be that the specific open-source data mixture used happens to align well with certain benchmarks. The paper cannot disentangle data effects from architectural effects without controlled data-equivalent training runs.
• The "unified architecture" claim is difficult to evaluate directly. Oryx demonstrates strong performance on image, video, and 3D benchmarks, but there is no experiment showing that a comparable model trained separately on each modality (e.g., with modality-specific encoders or projectors) underperforms Oryx on the same data. Table 8 shows that shared MLP outperforms separate MLPs, which is evidence for a benefit to unification, but the comparison is limited to the projector level—it does not test whether modality-specific vision encoders would help.
• No evaluation of computational cost relative to baselines. The paper argues that dynamic compression enables efficiency, and it reports a 7% slowdown for OryxViT relative to SigLIP (Appendix C.2), but it does not report the total inference cost (in FLOPs, tokens, or wall-clock time) of processing a benchmark video or image relative to competing models. The claim of "on-demand efficiency" is therefore more of a design philosophy than an empirically demonstrated operational advantage.
• Limited model family and LLM backbone testing. All experiments use Qwen-2, Qwen-2.5, or Yi-1.5 LLMs. The paper does not test with LLaMA, Mistral, or other popular open-source LLM families, leaving open the possibility that the architectural benefits depend on the specific LLM used. The two variants (Oryx and Oryx-1.5) show that results are robust to the Qwen version upgrade, but this is a narrow form of robustness testing.

Additional methodological concerns:

• The video needle-in-a-haystack experiment (Figure 10) is a visualization, not a quantitative benchmark. The heatmap shows green vs. red regions, but the paper does not report an accuracy score or specify how many test cases were run. This is a qualitative demonstration rather than a rigorous evaluation of long-form retrieval capability.

• The resolution ablation (Figure 3) does not control for total pixel count. The native-resolution point and the fixed-size points have different total pixel counts, making it impossible to determine whether the benefit comes from the resolution being "native" (matching the original image's pixel layout) or simply from having more pixels. The 2× native condition partially addresses this, showing that further increasing pixel count beyond native provides diminishing returns, but a controlled comparison at equal pixel counts (native vs. resized to same pixel count but different aspect ratio) is absent.

• Benchmark selection may favor the proposed method. The paper evaluates on benchmarks that emphasize text recognition (DocVQA, OCRBench, TextVQA) and long-form temporal understanding (MLVU, LongVideoBench), which are exactly the capabilities that native-resolution encoding and dynamic compression are designed to improve. There is no evaluation on benchmarks that might penalize variable-resolution processing, such as tasks requiring fine-grained spatial reasoning about object relationships where inconsistent patch layouts might interfere with spatial understanding. The absence of negative or neutral results on standard benchmarks is conspicuous.

Missing experiments that would strengthen the paper:

1. A controlled comparison at equal visual token budgets: Give Oryx and a baseline model the same total number of visual tokens for a long video, with Oryx using high compression on many frames and the baseline using low compression on fewer frames. This would directly test whether dynamic compression preserves more useful information than temporal subsampling.

2. A sweep of compression ratios on a single benchmark: Run VideoMME or MLVU with 1×, 2×, 4×, 8×, and 16× compression and plot the accuracy-compression curve. This would characterize the tradeoff and show where compression starts to hurt.

3. A comparison against FlexiViT or NaViT as the vision encoder: Replace OryxViT with another native-resolution ViT variant, keeping all else equal, to determine whether OryxViT's specific design is necessary or whether any native-resolution encoder would work.

4. Statistical significance testing on the key ablations: Report confidence intervals or run multiple seeds for the Table 5 ablations, given that the gains are modest (0.8–1.8 points) and the ablation used a data subset.

5. A fully controlled data-equivalent comparison: Train a baseline model (e.g., LLaVA-NeXT architecture) on the exact same data mixture as Oryx, to isolate architectural effects from data effects.

6. Evaluation on tasks that might be harmed by variable resolution: For example, visual grounding or referring expression comprehension, where the spatial correspondence between visual tokens and text tokens might be disrupted by variable patch layouts.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost: The 2048-Sample Overhead Is Not Accounted For

The compute-optimal framework requires estimating each prompt's difficulty before deciding how to allocate the test-time compute budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive relative to the budgets being optimized. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency gains (4× relative to best-of-N) are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. Generating 2048 samples per question is comparable to or greater than the largest test-time budgets studied (256–512 generations), meaning the true cost of the adaptive approach could exceed that of simply running a larger fixed strategy. For any deployment where prompts are seen only once (rather than batch-processing the same questions repeatedly, which would amortize the difficulty estimation), the reported 4× efficiency figure is best understood as an upper bound on achievable gains rather than a realized deployment improvement.

What evidence exists. The paper does not measure or report the cost of difficulty estimation anywhere. The 2048-sample procedure is described in Section 3.2, and the experiments in Figures 4 and 8 use both oracle and predicted difficulty bins, but neither curve includes the cost of producing those bins. The predicted (non-oracle) bins are computed from the PRM's average final-answer score across the same 2048 samples, substituting the ground-truth correctness check but retaining the 2048-sample generation requirement. No experiment tests whether difficulty estimated from fewer samples (e.g., 16 or 64) would produce sufficiently accurate bin assignments to preserve the gains.

Mitigation status. The paper acknowledges this as a key avenue for future work in Section 8, suggesting pretraining or fine-tuning models to directly predict difficulty from the question text. No such model is developed or evaluated, and no adaptive difficulty estimation scheme (e.g., using initial samples to estimate difficulty and then reallocating the remaining budget) is tested. The limitation remains entirely unaddressed in the current work.

---

2. Hard Problems Remain Fundamentally Unsolved: Test-Time Compute Cannot Create Capability

The paper's difficulty-bin analysis reveals a hard boundary on the effectiveness of test-time compute: for questions in the hardest quintile (bin 5), no method—search, revisions, or their compute-optimal combination—achieves meaningful improvement regardless of budget.

The consequence. Test-time compute amplifies existing capability but does not create it. If the base model's pass@1 is near zero on a problem class, there are essentially no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path, as the FLOPs-matched analysis in Section 7 confirms: on hard questions at high inference-to-pretraining ratios (R ≫ 1), test-time compute shows a −52.9% relative disadvantage compared to the ~14× larger model. This boundary is fundamental—it is not an artifact of insufficient budget or suboptimal strategy selection, but a consequence of the base model's capability ceiling.

What evidence exists. The evidence is consistent across multiple independent analyses:

• Figure 3 (right): Bin 5 accuracy hovers at 1–3% for all search methods and all budgets from 4 to 256 generations. The curves are essentially flat.
• Figure 7 (right): Bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at a fixed budget of 128 generations. The curve is flat, with no ratio providing meaningful improvement.
• Figure 9 (FLOPs-matched comparison): The bin 5 scaling line in both revisions (left) and PRM search (right) is essentially flat near 0–5%, lying below the larger model's greedy performance stars for all R values.

Mitigation status. The paper is transparent about this limitation. The Section 7 takeaway box explicitly states: "test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help." The compute-optimal policy implicitly acknowledges this by allocating minimal or no test-time budget to bin-5 problems (since no strategy helps), but this is a recognition of failure rather than a solution. The paper does not propose any method for addressing fundamentally unsolvable problems.

---

3. The FLOPs-Matched Baseline Uses a Non-Compute-Optimal Larger Model

The central training-inference tradeoff experiment in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. However, this larger model is not trained in a compute-optimal manner: the paper scales parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), where both data and parameters scale equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining—particularly the headline numbers in Figure 1 (e.g., +27.8% on easy questions at R ≪ 1)—may shrink or reverse against a properly compute-optimal larger model. This inflates the apparent value of test-time compute relative to pretraining. Additionally, the larger model is evaluated using only greedy decoding, with no test-time compute augmentation of its own (no majority voting, no best-of-N). Even a modest test-time compute budget (e.g., best-of-8) applied to the larger model would create a much stronger baseline that is never tested. The comparison therefore measures "small model with optimized inference vs. large model with minimal inference" rather than "optimized inference vs. optimized pretraining under equal total FLOPs."

What evidence exists. The paper specifies the FLOP accounting and the 14× scaling factor in Section 7 and Appendix, and explicitly notes the parameter-only scaling choice. No comparison against a Chinchilla-optimal larger model is provided. No experiment tests whether giving the larger model a modest test-time compute budget would reverse the findings.

Mitigation status. The paper flags this as future work in Section 7, but does not provide even a sensitivity analysis showing how the results would change under different pretraining scaling assumptions. The limitation is acknowledged but its quantitative impact on the main claims is unknown.

---

4. Single Benchmark and Single Model Family: Generality Is Unproven

All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper argues that PaLM 2-S* is "representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The findings could be specific to the interaction between PaLM 2's output distribution, the MATH problem distribution, and the specific verifier and revision models trained.

The consequence. Several aspects of the paper's core findings may not transfer to other settings:

• PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves, potentially changing which strategies are optimal at which difficulty levels.
• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The paper's finding that revisions work well on easy problems but not hard ones could be specific to PaLM 2's revision behavior.
• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems due to verifier over-optimization, revisions helping easy problems, neither helping hard problems) generalize to other reasoning domains such as code generation, logical reasoning, or scientific question answering, or to tasks requiring factual knowledge rather than multi-step inference.
• The test set size (500 questions) split into five difficulty quintiles of roughly 100 each, further split by two-fold cross-validation, means the compute-optimal policy is selected based on roughly 50 questions per fold per bin. This is a small sample for strategy selection, and the selected strategies may not be robust to different test sets or different base models.

What evidence exists. The paper provides no experiments on any benchmark other than MATH, nor with any base model other than PaLM 2-S*. No cross-model validation (e.g., replicating key findings with a different model family) is reported. The difficulty bins are computed relative to PaLM 2-S*'s pass@1 rates specifically; there is no analysis of whether difficulty bins would be similar for a different base model.

Mitigation status. The authors do not claim broader generality beyond noting their belief that PaLM 2-S* is representative. The limitation is not addressed experimentally, and no cross-benchmark or cross-model validation is suggested as future work in Section 8.

---

5. Sequential Revisions Introduce Latency That Parallel Strategies Avoid

The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revision strategies—which the compute-optimal policy favors for easy and medium problems—are inherently serial: each revision depends on the previous one and cannot be parallelized.

The consequence. A strategy that allocates 128 generations as 64 sequential revisions followed by 2 parallel chains (totaling roughly 64 serial steps) takes roughly 64× longer in wall-clock time than a fully parallel strategy that runs all 128 samples simultaneously on sufficient hardware. For latency-sensitive applications—interactive assistants, real-time decision-making, any deployment where users wait for responses—the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. The "4× efficiency gain" in generation budget may translate to negative efficiency gains in latency, since a strategy using 4× fewer total generations could take much longer to produce a result if those generations are sequential rather than parallel.

What evidence exists. The paper does not discuss latency or wall-clock time anywhere. The generation budget is the sole cost metric. Figure 7 (left) shows that at low budgets (8–32 generations), fully sequential is optimal, and at higher budgets, moderate sequential-to-parallel ratios are optimal. Neither the paper nor the compute-optimal policy considers the latency implications of these choices.

Mitigation status. Not addressed. The paper does not mention latency as a consideration, does not propose latency-aware allocation strategies, and does not suggest this as an avenue for future work. This is a practical deployment consideration that the paper's framework does not accommodate.

---

6. Verifier Over-Optimization Is a Hard Ceiling: The Compute-Optimal Policy Mitigates but Does Not Solve It

The paper documents verifier over-optimization as the central limiting factor for test-time compute scaling: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search—the strongest optimizer—paradoxically performs worst overall (Figure 3, left), and qualitative examples show search producing degenerate outputs (repetitive steps, overly short solutions; Appendix M) that score highly under the PRM. The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead), but it does not eliminate the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted.

The consequence. The compute-optimal approach is fundamentally bounded by verifier quality. Improving the PRM—through better training data, adversarial robustness, ensemble methods, or architectural improvements—would likely shift the difficulty thresholds and change the optimal policy, potentially enabling larger gains at higher budgets. The current results are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D. Without a more robust verifier, the scaling curves in Figures 4 and 8 will eventually plateau regardless of how intelligently the budget is allocated. This means the paper's approach does not provide a path to unbounded improvement from test-time compute—it provides a way to extract more value from a fixed verifier budget, but the ceiling is still the verifier's reliability.

What evidence exists. The evidence for verifier over-optimization as the bottleneck is distributed across several results:

• Figure 3 (right): Beam search accuracy on bin 1 (easiest) decreases from roughly 78% to 77% as the budget goes from 4 to 256 generations, while best-of-N weighted increases from 68% to 88%. This shows the verifier is being exploited—search finds solutions that score highly but are wrong.
• Figure 3 (left): Lookahead search, which uses the PRM more aggressively (simulating k steps forward per decision), underperforms all other methods at the same budget because its higher per-step cost reduces the number of beams explored and it over-optimizes the available verifier signal.
• Appendix M: Qualitative examples show degenerate outputs (low-information repetitive steps, overly short 1–2 step solutions) that score highly under the PRM.
• Figure 4: Even the compute-optimal search scaling curve shows signs of saturation at the highest budgets (256–512 generations), indicating that the verifier ceiling is being approached.

Mitigation status. The paper identifies verifier over-optimization as a key phenomenon and the compute-optimal policy is explicitly designed to work around it (by not using aggressive search where the verifier is unreliable). Section 8 identifies improving verifier robustness as an important direction for future work, specifically mentioning adversarial training and ensemble methods. However, the paper does not explore any verifier improvements beyond the Monte Carlo rollout training procedure, and the ceiling imposed by the current verifier limits the practical scaling gains achievable by any allocation policy.

7. Implications and Future Directions

How This Work Changes the Landscape

Oryx reframes a set of problems that the multimodal LLM field had largely accepted as fixed design constraints—how to handle images at different resolutions, how to allocate token budgets across video frames, how to balance spatial fidelity against temporal coverage—as a single, solvable architectural problem: make visual processing decisions input-conditional rather than hardcoded at design time. This is not a paradigm shift in the sense of introducing a new learning objective or a fundamentally different model family. It is, rather, a diagnostic reframing with architectural consequences: the paper identifies that the root cause of the resolution-flexibility gap is not insufficient training data or inadequate LLM reasoning, but rather the mismatch between standard vision encoders (which assume fixed-size square inputs) and the inherent diversity of visual data. Once this diagnosis is accepted, the solution direction becomes clear—modify the encoder to natively accept arbitrary inputs rather than working around the encoder's limitations with preprocessing tricks.

The practical significance of this reframing is that it redirects research attention from preprocessing workarounds to encoder design. Before Oryx, the dominant approach to high-resolution perception in MLLMs was dynamic partitioning (LLaVA-NeXT, InternVL2, MiniCPM-V): slice images into fixed-size sub-images, encode each independently, and concatenate the tokens. This approach implicitly accepted that the vision encoder itself cannot handle variable resolutions, so the problem must be solved at the preprocessing level. Oryx demonstrates that this acceptance was premature—with adaptive positional embeddings and variable-length self-attention, a vision encoder can handle native resolutions directly, and doing so avoids the boundary artifacts and computational inefficiency of partitioning. Table 5(a) provides the critical empirical evidence: SigLIP with native resolution collapses to 17.1% on DocVQA, showing that the partitioning workaround is genuinely necessary for standard encoders, but OryxViT with native resolution achieves 78.5%, demonstrating that the limitation is encoder-specific and addressable.

This finding has a delegitimizing effect on the partitioning paradigm for future work. If native-resolution encoding is architecturally feasible and empirically superior, then continuing to use partitioning-based approaches can be seen as a concession to legacy encoder constraints rather than a principled design choice. The paper does not argue that partitioning should be abandoned—it acknowledges that partitioning "remains a compromise due to the lack of high-quality multi-modal encoders that support native resolution inputs" (Section 1)—but it provides a concrete alternative that future work can adopt or improve upon.

The paper also establishes a decomposition of the visual processing pipeline into independent, controllable axes: resolution (how many pixels the encoder sees), compression (how many tokens the LLM receives), and temporal sampling (how many frames are extracted from a video). Prior work conflated these—increasing resolution implicitly increased tokens, increasing frame count implicitly increased tokens proportionally—creating zero-sum tradeoffs where improving one dimension necessarily sacrificed another. Oryx shows that these dimensions can be controlled independently, enabling per-input optimization that adapts to task requirements. The needle-in-a-haystack experiment (Figure 10) demonstrates the practical value of this decoupling: at 1.6K frames, Oryx successfully retrieves information that a baseline model loses entirely, specifically because compression allows many frames at adequate resolution rather than forcing a choice between few high-quality frames or many degraded ones.

However, the paper's resolution of prior contradictions is more limited than it might appear. The paper does not resolve conflicting findings about which visual encoding strategy is optimal—it introduces a new strategy and demonstrates its effectiveness, but does not systematically compare against all alternatives under controlled conditions. Prior work using partitioning (LLaVA-NeXT, InternVL2) and prior work using fixed-resolution encoding (LLaVA-1.5, Qwen-VL) are not shown to be definitively worse when matched for training data, model scale, and compute budget. The paper's contribution is more accurately described as offering a new point in the design space that was previously unexplored—native-resolution encoding for MLLMs—rather than proving that this point dominates all others.

The paper's most concrete landscape-changing contribution is demonstrating that strong multi-domain performance (image, video, 3D) is achievable within a single unified architecture, without modality-specific encoders, separate projection layers, or specialized 3D processing pipelines. The ablation on shared vs. separate MLP projectors (Table 8) shows that a single projector benefits video understanding (VideoMME improves from 54.0 to 55.4, MLVU from 54.2 to 59.3) without hurting image performance. This suggests that the common practice of building separate video-specific components on top of image foundation models may be unnecessary—a properly designed unified architecture can transfer visual understanding across modalities through shared representations. This finding makes unified multimodal architectures a more attractive research direction and modality-specific architectures correspondingly less attractive, at least for the image-video-3D combination that Oryx handles.

Follow-Up Research This Work Enables

Adaptive compression policies learned from task context. The paper uses fixed heuristics to select compression ratios: 1× for images, 2× for standard videos, 4× for long videos. The compression ratio is chosen based on the input type, not the specific content or the question being asked. A natural extension is to train a lightweight policy network—either as a separate module or integrated into the dynamic compressor—that predicts the optimal compression ratio per-input based on the visual features and the text query. For example, a question like "What color is the car?" on a video might require lower compression (more spatial detail) than "Summarize the events in this video." The paper provides the architectural hooks for this: the dynamic compressor already supports multiple compression paths (d₁, d₂, d₃), and the cross-attention mechanism could produce a difficulty or importance signal that feeds into a compression decision. A strong follow-up would sweep compression ratios on a fixed benchmark (e.g., VideoMME) to establish the accuracy-compression Pareto curve, then train a policy that selects compression ratios per-question and demonstrate that adaptive selection outperforms the fixed heuristics used in Oryx.

Native-resolution pre-training objectives, not just architectures. OryxViT is pre-trained with a standard MLLM objective (image captioning and OCR using a small LLM interface), just with native-resolution inputs rather than fixed-size crops. But the objective itself does not explicitly encourage the encoder to produce resolution-invariant representations—it simply trains on whatever resolution the input happens to have. A follow-up could design pre-training objectives that explicitly encourage OryxViT to produce consistent representations across multiple resolutions of the same image, for example through a contrastive loss that pulls together features from the same image at different native resolutions (e.g., the original and a downsampled version). This could produce an encoder that is more robust to resolution variation and that transfers better to downstream tasks where the ideal resolution is unknown or varies across instances. The paper's existing resolution ablation (Figure 3) shows that performance saturates near native resolution, but a resolution-invariant pre-training objective might shift the saturation point or improve performance at lower resolutions—a practically valuable outcome for bandwidth-constrained deployments.

Stress-testing the compression-quality tradeoff systematically. The paper demonstrates that dynamic compression can work (Table 5b shows Dynamic Compressor > average pooling; Figure 10 shows successful retrieval at 1.6K frames), but it never characterizes when compression starts to hurt or for which types of visual content it is most damaging. A direct follow-up experiment would take a fixed benchmark (MLVU would be ideal, since it covers videos from 3 minutes to 2 hours and includes 9 distinct task types) and evaluate Oryx at every compression ratio from 1× to 16×, plotting per-task accuracy curves. The hypothesis is that compression will hurt more on tasks requiring fine-grained spatial detail (e.g., object counting, text reading within video frames) than on tasks requiring global scene understanding or temporal reasoning. Characterizing this breakdown would provide practical guidance for deployment: if certain task types are known to be common in a given application, the compression ratio can be set accordingly. It would also reveal whether there are task types for which even 4× compression causes unacceptable degradation—a finding that would motivate better compression mechanisms.

Combining OryxViT-style native resolution with test-time compute scaling along the temporal dimension. The paper treats video length as a fixed property of the input and selects a single compression ratio for all frames. But within a long video, some segments may contain critical information requiring high-fidelity encoding while others are filler that can be aggressively compressed. A natural extension—bridging this work with the test-time compute scaling literature—is to develop an adaptive temporal allocation strategy: sample frames at a high initial rate with aggressive compression, use a lightweight importance scorer (perhaps the cross-attention weights from the dynamic compressor, or a small learned head on OryxViT features) to identify high-information segments, and then re-encode those segments at lower compression (or higher resolution) within a fixed total token budget. This connects directly to the needle-in-a-haystack capability that Oryx's training data targets: the model would learn not just to retrieve information from long contexts, but to decide where to spend its encoding budget to maximize retrieval accuracy. The MLVU benchmark, with its mix of global and local understanding tasks, would be an ideal testbed—global tasks should benefit from uniform compression across the full video, while local tasks should benefit from targeted high-fidelity encoding of relevant segments.

Replacing coarse correspondence markers with learned cross-view attention. The coarse correspondence approach for 3D understanding (Section 3.2.1) relies on an external tracking model (Track-Anything) to assign consistent object labels across views. This is a strong dependency: if the tracker fails or is unavailable, the 3D training pipeline breaks. A follow-up could replace the explicit correspondence markers with an architectural mechanism that learns cross-view correspondence implicitly. Specifically, one could add cross-attention layers between the visual tokens of different views before they enter the LLM, allowing the model to learn which patches in view A correspond to which patches in view B from the 3D QA supervision alone, without explicit object identity labels. The ScanQA benchmark could be used to evaluate whether the learned correspondence matches or exceeds the explicit marker approach. A negative result (learned correspondence underperforms explicit markers) would confirm that the identity signal is genuinely the bottleneck, not something the LLM can infer from multi-view geometry alone. A positive result would eliminate the dependency on external trackers and make the approach applicable to any multi-view dataset, not just those where tracking is feasible.

Evaluating Oryx on tasks that may penalize variable-resolution encoding. The paper evaluates on benchmarks that align with Oryx's strengths: text-heavy documents (DocVQA, OCRBench) where native resolution helps, and long videos (MLVU, LongVideoBench) where dynamic compression helps. There is no evaluation on tasks that might be harmed by variable-resolution processing. A critical stress-test would evaluate Oryx on visual grounding tasks (e.g., Referring Expression Comprehension on RefCOCO, or phrase grounding on Flickr30k Entities), where the spatial correspondence between visual tokens and text tokens is crucial. The hypothesis is that variable patch layouts—where the same object appears at different positions in the patch grid depending on the image's native resolution—might disrupt the learned alignment between visual regions and textual referring expressions, since the position embeddings that the LLM learns to interpret during training are no longer consistent across inputs. A negative result on these tasks would establish a boundary condition: native-resolution encoding benefits recognition and description but may interfere with precise spatial localization, suggesting that different visual processing strategies are needed for different task families even within a single input type.

Practical Applications and Downstream Use Cases

Cost-efficient video understanding for media archives. Organizations with large video archives—news agencies, sports broadcasters, educational content platforms, surveillance operations—need to answer content-based queries across thousands of hours of footage. The standard approach of processing all content at high resolution and frame rate is economically prohibitive. Oryx's on-demand compression enables a tiered processing pipeline: ingest all content at aggressive compression (4×, e.g., 36 tokens per frame at 384² resolution) to enable temporal coverage across the full archive, then selectively re-process segments identified as relevant at lower compression or higher resolution. The needle-in-a-haystack capability (Figure 10, successful retrieval at 1.6K frames) means the initial high-compression pass can reliably identify where relevant content occurs, even when the target is a single frame within hours of footage. The 32B Oryx-1.5 variant achieves 72.3% on MLVU (beating GPT-4o by 7.7 points, Table 2), demonstrating that the compressed representations retain sufficient information for accurate question answering on videos up to 2 hours. For a news archive with 10,000 hours of content, processing at 1 fps with 4× compression (36 tokens per frame) produces roughly 1.3 billion visual tokens—manageable with batched processing—compared to the 5.2 billion tokens that 2× compression would require or the 20.8 billion tokens for uncompressed frames, representing a 4× to 16× reduction in inference cost.

Unified multimodal document assistants with variable precision. An organization deploying a single MLLM to handle diverse document types—scanned contracts, presentation slides, handwritten notes, architectural diagrams, and accompanying explainer videos—faces the precise problem Oryx solves: different inputs demand different visual processing. A scanned contract requires high resolution (1536² pixels, 1× compression) to render fine print legible; an explainer video benefits from modest resolution (384²–480² per frame) with 2× compression to cover temporal content within the context window. Oryx's native-resolution encoding (78.5% DocVQA at 7B, Table 5a; 92.7% at 32B, Table 3) ensures that document-heavy queries receive high-fidelity visual encoding when needed, while the dynamic compressor ensures that video queries do not exhaust the context budget. The shared MLP projector (Table 8: shared outperforms separated by 1.4 points on VideoMME, 5.1 points on MLVU) means the system can be deployed as a single model serving all content types, simplifying deployment and maintenance relative to maintaining separate image-optimized and video-optimized models. This is particularly valuable for enterprise deployments where the mix of content types is unpredictable and maintaining multiple specialized models would increase operational complexity.

3D scene understanding from commodity RGB captures without specialized sensors. The coarse correspondence approach (Section 3.2.1, Table 4: Oryx-1.5-32B achieves 15.3 METEOR on ScanQA, surpassing specialized 3D-LLM at 14.5) enables 3D spatial reasoning from multi-view 2D images without requiring depth sensors, point cloud processing, or explicit 3D reconstruction. This has direct applications in real estate (answering spatial questions about properties from photo tours), retail (understanding product layouts from shelf photos), and augmented reality (spatial reasoning about a user's environment from multiple camera views). The practical benefit is eliminating the need for specialized 3D capture hardware or separate 3D processing pipelines—the same Oryx model that handles standard images and videos can answer questions like "Is there a window to the left of the door?" or "Which chair is closest to the table?" directly from a set of 2D photographs. The incremental benefit of coarse correspondences (+1.2 METEOR over 3D data alone, Table 7b) suggests that even automatic tracking-based correspondence generation provides a meaningful spatial reasoning signal, though the absolute METEOR score of 15.3 (BLEU-1 of 38.8, BLEU-2 of 24.4) indicates that generated captions remain far from human-level spatial descriptions, limiting current applications to qualitative reasoning rather than precise metric understanding.

When to Prefer This Method

The paper positions Oryx against two broad alternatives: MLLMs using dynamic partitioning for high-resolution perception (LLaVA-NeXT, InternVL2, MiniCPM-V) and MLLMs using fixed-resolution encoding with uniform compression (earlier LLaVA variants, Qwen-VL). The paper's explicit tradeoff criteria, grounded in the experimental evidence, are:

• Prefer Oryx over dynamic-partitioning MLLMs when: (1) preserving continuous spatial structure matters—for document images, diagrams, or scenes where objects span across what would be partition boundaries in a tiling approach (Table 5a: OryxViT native at 78.5% DocVQA vs. OryxViT partition at 76.3%, a 1.2-point gap from boundary artifacts); (2) token budget per input must be variable and compressible—for long videos where partitioning would generate prohibitive token counts (Figure 10: Oryx at 1.6K frames with 4× compression succeeds where fixed-token baselines fail); (3) a single model must handle multiple modalities (images, videos, 3D) without modality-specific components (Table 8: shared MLP outperforms separate MLPs on video benchmarks).

• Prefer Oryx over fixed-resolution MLLMs when: (1) text-heavy understanding tasks constitute a significant fraction of the workload—DocVQA shows a 74.8% → 17.1% collapse for SigLIP at native resolution (Table 5a), establishing that fixed-resolution encoding is fundamentally lossy for document tasks; (2) the input distribution spans widely varying aspect ratios where forced square resizing would distort spatial layouts; (3) the deployment requires adjusting resolution-resolution tradeoffs at inference time based on task or latency constraints.

• Prefer alternatives to Oryx when: (1) the deployment is exclusively image-based with modest resolution requirements (e.g., 336²–448² pixels), where the additional complexity of native-resolution encoding and dynamic compression provides marginal benefit—the paper's ablation (Figure 3) shows that resolution effects are largest on text-heavy benchmarks and smaller on general understanding; (2) the application demands pixel-precise spatial localization (visual grounding, referring expressions) where Oryx has not been evaluated and variable patch layouts might disrupt spatial alignment; (3) extremely low-latency inference is the primary constraint—OryxViT shows a 7% throughput reduction relative to SigLIP (Appendix C.2: 146.5 vs. 157.7 images/second), and the dynamic compressor adds additional computation, which may be unacceptable for real-time applications where every millisecond matters; (4) the training infrastructure or budget does not support pre-training a custom vision encoder—OryxViT requires 400M image-text pairs with a small LLM interface (Appendix D.2), which, while faster than full SFT, is a non-trivial pre-training cost that partitioning-based approaches avoid entirely.

The paper does not position Oryx against modality-specific architectures (separate image, video, and 3D models) as a practical deployment choice, since the unified architecture is a core contribution, not one option among several. The implicit claim—supported by Tables 1–4 showing competitive or SOTA performance across all three modalities—is that the unified approach is Pareto-dominant: it achieves at least as good performance as specialized alternatives while reducing deployment complexity. However, this claim is not tested through direct comparison against modality-specific models matched for training data and compute, so it remains a supported hypothesis rather than a demonstrated fact.