Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities

ArXiv: 2311.05698

🎯 Pitch

A 3B-parameter model surpasses the 80B Flamingo on video QA by giving video and audio their own dedicated autoregressive processing stream, rather than starving them of parameters. The key is a Combiner that fuses and heavily compresses synchronized audio-visual snippets, allowing the model to scale to 512 frames without a parameter explosion.

---

1. Executive Summary

This paper proposes an autoregressive multimodal model that decouples learning into two separate autoregressive components — one for time-aligned modalities (video and audio, processed in synchronized chunks through time) and one for non-time-aligned contextual modalities (text, processed sequentially with cross-attention to the media representations). The core architectural innovation is the Combiner mechanism (a transformer or Token Turing Machine module that fuses video and audio features within each time snippet and compresses them into compact representations, reducing the token count from hundreds of features per chunk to just 32), which enables the model to scale to 512 input frames without increasing parameter count. On MSRVTT-QA, the 3B-parameter model achieves 50.42% accuracy, outperforming the 80B Flamingo model — establishing that decoupling autoregressive modeling by modality alignment allows smaller models to surpass much larger ones on video understanding tasks only when the media inputs receive adequate computational capacity through dedicated autoregressive processing and joint feature compression.

2. Context and Motivation

The Core Challenge: Multimodal Learning With Radically Asymmetric Modalities

The fundamental problem this paper tackles is how to design a neural architecture that can effectively learn from three modalities — video, audio, and text — which arrive at the model with fundamentally different temporal characteristics, information densities, and volumes. This is not merely an engineering inconvenience; it is a structural mismatch that forces difficult trade-offs in model design.

To understand the asymmetry, consider a typical 15-second video clip. The video stream might produce 450 frames (at 30 fps), each containing rich spatial information across thousands of pixels. The accompanying audio waveform generates tens of thousands of samples, carrying complementary information about speech, environmental sounds, and music that is roughly synchronized with the visual stream. Meanwhile, the text modality — perhaps a title, a description, or a question-answer pair — arrives as a single global annotation, a few dozen tokens, applicable to the entire video without any frame-level or second-level temporal alignment. The text is sparse, abstract, and decoupled from the moment-to-moment flow of the audio-visual experience.

The information volume imbalance is staggering. As the authors note in Section 1, video and audio inputs are "of much larger volumes, and grow as the video length increases." A 512-frame video produces orders of magnitude more raw data than a 50-token text description. Yet in many prior multimodal architectures, these modalities are funneled through a unified processing pipeline — often a single autoregressive transformer — where they compete for representational capacity. The result, as we will see, is that media modalities get systematically starved of parameters.

Why This Problem Matters (Real-World Impact)

The ability to jointly understand video, audio, and text is critical for a wide range of practical applications: video question answering (answering natural language questions about visual events), content indexing and retrieval (finding "the moment when the dog barks" in hours of footage), accessibility tools (generating rich descriptions for visually or hearing-impaired users), autonomous systems (understanding dynamic environments from multiple sensory streams), and human-AI interaction (conversational agents that can reference, explain, and reason about audiovisual content).

These applications are not speculative. Video content dominates internet traffic, and multimodal models that can reason about it have direct commercial, scientific, and social value. However, the deployment of such models is constrained by two realities: (1) video processing is computationally expensive, and the cost grows with video length, making long-form video understanding particularly challenging; and (2) naive approaches to handling this compute burden — such as aggressive subsampling of frames — degrade the model's ability to capture fine-grained temporal dynamics and long-range dependencies.

The theoretical significance is equally important. Multimodal learning is fundamentally about discovering the shared structure across different sensory signals. Video and audio share a temporal dimension — events that happen together in time create a rich self-supervisory signal that is more frequent and more granular than the global supervision from text. The authors argue in the introduction that "this co-occurrence in time can contribute to their joint learning and serve as a rich self-supervisory learning signal, applied more frequently than global text signals." A model architecture that properly exploits this temporal alignment could learn better representations from less data, but only if the architecture is designed to leverage it. This paper's core argument is that most prior architectures fail to do so.

Prior Approaches and Where They Fall Short

The paper identifies three broad families of prior work, each with specific shortcomings that motivate the proposed design.

---

Approach 1: Unified Autoregressive Models With Tokenized Media Inputs

Many recent multimodal models extend the autoregressive language modeling paradigm to visual inputs by tokenizing images or videos — mapping visual content to discrete tokens that can be interleaved with text tokens and processed by a single autoregressive transformer [1, 41, 53, 69, 72]. The appeal is architectural simplicity: one model, one training objective, one sequence. For video, this typically means processing individual frames through an image encoder or tokenizer and concatenating the resulting tokens.

The problem is scale. A single video frame, when tokenized into patches (as is standard for Vision Transformers), might produce hundreds of tokens. At 32 or 64 frames — which prior video-language models like VideoCoca [64] or the dynamic pretraining approach of Piergiovanni et al. [40] typically handle — this produces thousands to tens of thousands of visual tokens for a single video. These tokens must share the transformer's self-attention context with the text tokens, and since self-attention complexity is quadratic in sequence length, the compute cost grows rapidly. The practical result is that these models process only a limited number of frames — the paper explicitly notes that "methods that process the video, running each frame independently through an encoder or a tokenizer, can process only a limited number of frames [40, 64]."

This matters because temporal coverage directly affects understanding. A model limited to 8 or 32 frames from a 160-second ActivityNet video is sampling roughly 0.5–2 frames per second — it will miss rapid actions, subtle transitions, and causal sequences that unfold across time. The model sees a sparse slideshow rather than a continuous video, and its ability to answer questions about dynamic events is fundamentally limited by this subsampling.

---

Approach 2: Encoder-Decoder Models With Compressed Visual Features

A second family of approaches, exemplified by Flamingo [2], avoids tokenizing every frame into the transformer's input sequence. Instead, a visual encoder produces frame-level feature embeddings, which are then compressed (via a Perceiver resampler or similar attention-pooling mechanism) into a smaller fixed-size set of visual tokens. These compressed tokens are injected into the language model via cross-attention layers.

This approach is computationally more efficient and allows for more frames to be processed. Flamingo can handle videos with many frames because the Perceiver compresses the per-frame features before they enter the autoregressive text model. However, the paper identifies a critical shortcoming: parameter allocation is heavily skewed toward text processing. As the authors note, Flamingo "dedicates only about 1% of the parameters to the image and video inputs, leaving the rest for text processing." In a system that must understand detailed visual events, fine-grained motion, and subtle audiovisual correspondences, this parameter starvation of the visual pathway means the model has limited capacity to learn rich video representations. The visual features are compressed aggressively, and their temporal relationships are primarily handled by the text model's cross-attention — which was designed for language understanding, not spatiotemporal reasoning.

More broadly, the paper argues that encoder-decoder approaches that embed video into a compressed representation before the main model miss the opportunity to model temporal dependencies explicitly within the media modalities themselves. The video is treated as a bag of features to attend to, rather than a dynamic sequence whose internal temporal structure carries essential information.

---

Approach 3: Joint Audio-Visual Models (Without Text Integration)

For audio-video learning specifically, several prior works have explored joint modeling of the synchronized modalities [16, 17, 18, 22, 42, 72]. UAVM [16] proposes a unified transformer that can process either modality, leveraging their temporal alignment through shared architecture. MAViL [18] uses masked autoencoders to learn jointly from visual and audio inputs. Contrastive learning between synchronized audio-video pairs has been explored by Gong et al. [17] and others.

These approaches exploit the temporal co-occurrence of audio and video, which is a genuine strength. However, the paper identifies two limitations. First, many of these works tokenize audio and video independently — applying 2D patch-based tokenization to both spectrograms and video frames separately [18] — which does not jointly model the shared spatiotemporal structure. Second, and most critically for this paper's agenda, these models do not integrate text as a full peer modality. They are audio-visual models, not audio-video-text models, and they don't address the challenge of combining temporally-aligned multimodal streams with global, non-aligned language context in a single coherent architecture.

The paper acknowledges work that does combine all three modalities [47, 72], but notes that MERLOT Reserve [72] aligns only text and audio (not video with both simultaneously), and Multimodal Transformer [47] processes all pairs via cross-attention but doesn't address the sequence length scaling challenge for long videos.

---

Summary of What's Missing

Across all three families, the paper identifies a consistent set of gaps:

How This Paper Positions Itself

The paper's positioning is clear and deliberate: it proposes a principled architectural decoupling where the model structure mirrors the data structure. Rather than forcing all modalities through a single autoregressive pipeline — which inevitably compromises between the high-frequency, high-volume, time-aligned nature of video/audio and the low-frequency, sparse, non-aligned nature of text — the architecture is split into two separate autoregressive components, each with its own representational capacity and processing characteristics.

Specifically, the paper argues (Section 1):

"the modalities need to be processed by differently-synchronized model components, which process more adequately inputs of different frequencies and allocate more parameters to the more abundant modalities."

This is the central design philosophy. It is not merely an engineering optimization; it is a hypothesis about how multimodal architectures should be structured. The hypothesis is that time-aligned modalities benefit from being modeled autoregressively in time together — not compressed from above by a text model that attends to them as a flat set of features, but rather processed by a dedicated autoregressive model that learns the sequential dynamics within the media stream itself. The text model then cross-attends to the learned latent representations of this media model, rather than to raw or lightly compressed features.

This positioning has several important implications:

1. It is a critique of the Flamingo paradigm (and similar cross-attention-based architectures). While Flamingo showed that cross-attention is an effective fusion mechanism, it demonstrably underinvested in the visual representation. This paper argues that by giving the visual pathway its own autoregressive model with substantial capacity, you get better results — even from a much smaller overall model. The headline result (3B parameters outperforming 80B Flamingo on MSRVTT-QA, Table 1) is a direct empirical argument for this architectural choice.

2. It extends the autoregressive modeling idea from text to time-aligned media. Autoregressive modeling has been spectacularly successful for language because language is sequential. The paper argues that video and audio are also sequential — they unfold in time — and therefore autoregressive modeling is natural for them too. But crucially, the paper autoregressively models the compressed feature representations (from the Combiner) rather than raw pixels or low-level tokens. This is a middle ground between pixel-by-pixel autoregressive generation [48, 55] (which is too fine-grained and captures only short-term dependencies) and treating the entire video as a single encoding (which loses temporal structure entirely). The paper explicitly argues this point:

"While autoregressive modeling has been used for videos and images, it is often done on a pixel-by-pixel basis [55] which is highly inefficient and captures only short-term dependencies. With our approach, with autoregressive modeling and the Combiner, we address both shortcomings." (Section 3.3)

3. It introduces the Combiner as the mechanism that makes the decoupling work. The Combiner is not just a compression module — it is the component that learns the joint representation of audio and video within each time chunk before the autoregressive model processes the sequence of these representations. This means the autoregressive model operates on features that already encode cross-modal interactions (audio-visual correspondences within a snippet), while the autoregressive modeling itself captures cross-time interactions (how events evolve across snippets). This hierarchical decomposition — joint per-snippet fusion followed by sequential cross-snippet modeling — is the paper's key insight for handling long video sequences efficiently.

4. It frames long-video understanding as an architectural scaling problem, not just a data problem. Prior work on long-form video understanding [13, 46, 58, 59] has proposed techniques like hierarchical attention, temporal windows, and memory mechanisms to handle longer sequences. The paper's position is complementary but distinct: rather than retrofitting long-sequence handling onto a flat video representation, it argues that the representation itself should be structured — partitioned into chunks, compressed by the Combiner, and modeled autoregressively. This makes scaling to 512 frames (Section 4, Tables 2-3) a natural consequence of the architecture rather than requiring special long-sequence handling.

5. It treats audio-video synchronization as a primary self-supervision signal, not an afterthought. Many prior multimodal models add audio as an auxiliary input (if at all). This paper places audio and video on equal footing within the time-aligned autoregressive component, processing them jointly through the Combiner and modeling their combined temporal dynamics. The paper's results on audio-video benchmarks (Table 4, Kinetics-Sound, VGG-Sound, Epic-Sound) with substantial margins over prior work validate that this joint temporal modeling is genuinely beneficial, not just architecturally convenient.

The Core Conceptual Argument

Synthesizing the above, the paper's intellectual contribution can be understood as an argument with three linked claims:

1. Claim about architecture and data structure: The temporal structure of the data should be reflected in the architecture. Modalities that are time-aligned and high-frequency (video, audio) should have their own autoregressive model. Modalities that are sparse and non-time-aligned (text) should have their own autoregressive model. Cross-attention bridges them.

2. Claim about feature representation: Learning joint audio-visual features within time chunks (via the Combiner) before modeling temporal dynamics across chunks is more effective than either (a) learning temporal dynamics from modality-specific features, or (b) compressing all temporal information into a single representation without explicit autoregressive modeling. The ablation in Table 5a supports this: the Combiner alone helps, autoregressive modeling alone helps, but the combination helps more (44.7% vs. 43.2% and 42.1%, on the 32/4 frame/chunk configuration).

3. Claim about parameter efficiency: Allocating substantial capacity to media processing (over half the model's parameters in the 3B version) pays off — a smaller model with properly allocated parameters can outperform much larger models that starve their visual pathways. This is not just about total FLOPs or total parameters; it's about distributing capacity where the information volume demands it.

The paper does not claim that autoregressive modeling of video is itself novel, nor that cross-attention fusion is novel. The novelty is in the combination — the specific way these mechanisms are assembled around the alignment characteristics of the modalities, with the Combiner as the critical enabling module that bridges per-snippet fusion and cross-snippet autoregressive modeling while controlling sequence length for scalability.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

What is being built: Mirasol3B is a 3-billion-parameter neural network that takes a video (with its accompanying audio track) and a text question or description as input, and produces a text answer as output — for example, answering "What is the person doing?" after watching a video of someone playing guitar. The system is a multimodal autoregressive model that breaks the learning problem into two separate processing pipelines: one dedicated to the video and audio (which are synchronized in time and arrive at high rates), and another dedicated to the text (which is sparse, global, and not temporally aligned with specific video frames). The core design philosophy is that the architecture should mirror the data's structure: time-aligned modalities get their own autoregressive model that learns how events unfold sequentially through the video, while the text model cross-attends to the learned media representations to produce answers.

3.2 Big-Picture Architecture (Diagram in Words)

The Mirasol3B architecture consists of six major processing stages connected in sequence:

1. Input Partitioning: The video (up to 512 frames) and audio are split into T non-overlapping time chunks, each covering a short temporal window (e.g., 16 chunks of 8 frames each). This turns one long video into a sequence of short video snippets with synchronized audio segments.

2. Feature Extraction: Each video chunk passes through a Vision Transformer (ViT) that extracts sparse 3D spatiotemporal features (not just per-frame 2D patches). Each audio chunk passes through a spectrogram converter followed by the same ViT backbone, producing time-aligned audio features. The outputs are $\hat{v}_t$ (video features per chunk $t$) and $\hat{a}_t$ (audio features per chunk $t$).

3. The Combiner: For each time chunk, the concatenated video and audio features ($u_t = [\hat{v}_t, \hat{a}_t]$) are fed into the Combiner — a transformer or Token Turing Machine module — which fuses them into a compact joint representation $x_t$ of only 32 features per chunk (down from potentially hundreds). This is the critical compression and fusion step.

4. Autoregressive Latent Causal Model: The sequence of combined features $x_1, x_2, ..., x_T$ is processed by an autoregressive transformer that models how these representations evolve through time. It predicts $x_{t+1}$ from $x_1, ..., x_t$, producing latent representations $\hat{h}_t$ that encode the temporal dynamics of the audiovisual stream.

5. Text Autoregressive Model: The text input (question, description, etc.) is processed by a separate autoregressive transformer that generates the output answer token by token. At each generation step, it cross-attends to the full sequence of latent representations $\hat{h}_1, ..., \hat{h}_T$ from the media model.

6. Loss Computation: Two losses drive training — a latent reconstruction loss that penalizes the temporal model for poor next-step predictions (encouraging learning of video dynamics), and a standard text cross-entropy loss that penalizes incorrect answer generation.

The key division of labor: the Combiner and latent causal model handle what happens in the video and how it changes over time, while the text model handles how to produce language that describes or answers questions about it. The Combiner output dimension of 32 features per chunk (Section 3.6) means that a 16-chunk video contributes only 512 features to the cross-attention context, regardless of how many raw frames were in each chunk — this is what enables scaling to 512 frames without increasing the parameter count of the text model.

3.3 Roadmap for the Deep Dive

• First, the input partitioning and feature extraction for video and audio (Section 3.1), because these produce the raw feature streams that every subsequent component operates on. Understanding what features the model has access to is prerequisite to understanding how they are combined.
• Second, the Combiner mechanism (Section 3.2), which is the central architectural contribution. We'll examine both transformer-based and TTM-based variants, their causality constraints, and why compression from hundreds of features to 32 matters for scaling.
• Third, the time-aligned autoregressive modeling (Section 3.3), including the latent causal model that processes the Combiner outputs sequentially in time, the modified attention mechanism that preserves both intra-chunk and inter-chunk interactions, and the modality reconstruction objective.
• Fourth, the combination with text modeling (Section 3.4), where the text autoregressive model cross-attends to the learned media representations. We'll cover the decoupled autoregressive design and why it allocates capacity differently from prior approaches.
• Fifth, the loss functions and training procedure (Sections 3.5–3.6), including the weighting of latent reconstruction vs. text generation losses and the pretraining-to-finetuning transition.
• Sixth, the implementation details (Section 3.6 and Appendix D), including parameter counts, model dimensions, training hyperparameters, and the specific configurations used in the main experiments and ablations.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural design paper whose core idea is that multimodal models should decouple autoregressive processing according to the temporal alignment characteristics of the input modalities. The paper proposes a specific instantiation of this principle: a two-stream autoregressive architecture with a Combiner module that jointly fuses and compresses time-aligned media features before they are modeled sequentially in time.

---

Input Partitioning: Turning a Long Video Into a Sequence of Chunks

The video arrives as $N$ frames $v = \{v^f_1, v^f_2, ..., v^f_N\}$ and the audio as $M$ waveform timesteps $a = \{a^f_1, a^f_2, ..., a^f_M\}$, where the audio was recorded during the same time interval as the video. The text input $t = \{t^f_1, t^f_2, ..., t^f_P\}$ arrives separately and is not temporally aligned — it could be a title, a descriptive caption, or a question about the content.

Rather than processing all $N$ frames and $M$ audio timesteps as a single massive input, the paper partitions the media into $T$ consecutive, non-overlapping time chunks (Section 3, "Partitioning of the media inputs"). Let $K = N/T$ be the number of frames per chunk. The video is divided as:

$v^f_1, ..., v^f_K, v^f_{K+1}, ..., v^f_{2K}, ..., v^f_{(T-1)K+1}, ..., v^f_N$

where the first chunk $v_1$ contains frames $1$ through $K$, the second chunk $v_2$ contains frames $K+1$ through $2K$, and so on. The audio is partitioned identically into $T$ chunks $a = \{a_1, a_2, ..., a_T\}$ that are roughly time-synchronized with the corresponding video chunks — the audio within chunk $t$ was recorded during the same time window as video chunk $t$.

What this computes: A long continuous video is converted into a sequence of $T$ short video clips, each of length $K = N/T$ frames, with synchronized audio segments. This turns the video from a single high-dimensional input into a temporal sequence of lower-dimensional inputs.

Why this form: Processing the entire video at once in a transformer would require quadratic attention across all frame features, making the computation scale as $O(N^2)$ in the number of frames — infeasible for 128 or 512 frames. Partitioning into chunks does three things. First, it makes the per-step computation constant with respect to total video length (each chunk has fixed size $K$). Second, it creates a natural temporal sequence that can be fed to an autoregressive model — the model learns to predict features of chunk $t+1$ from chunks $1$ through $t$, which is the standard autoregressive formulation. Third, it allows the Combiner to work on a manageable number of features per chunk rather than all features from the entire video simultaneously. The paper uses $T = 16$ chunks customarily, with each chunk containing 8 frames for the standard 128-frame setting, or 32 frames per chunk for the 512-frame setting (Section 3.6 and Table 2). Overlap between chunks is possible and mentioned but not used in the reported experiments.

---

Video Feature Extraction: Sparse 3D Tubes for Spatiotemporal Representation

Prior video-language models typically process individually sampled frames through a 2D image encoder (e.g., a ViT applied to each frame independently). This approach, the paper argues, "lacks the temporal information essential to video understanding and which might miss dynamic events" (Section 3.1). A single frame cannot capture motion, and per-frame features must be aggregated by later layers to recover any temporal signal — but this aggregation happens after the lossy compression of each frame into a fixed-size embedding vector.

Instead, the paper extracts sparse 3D tubes from the video input, building on prior work by Piergiovanni et al. [39]. A 3D tube is a spatiotemporal patch of pixels that spans both spatial dimensions (height and width) and the temporal dimension (across consecutive frames within a chunk). Unlike a standard 3D convolution that applies dense filters across all spatiotemporal locations, sparse tubes are applied at a subset of starting locations, reducing computation while still capturing motion and dynamics.

The tubes, together with standard 2D patches, are processed through a ViT encoder — specifically, the ViT-Huge variant with 32 layers, model dimension 1280, hidden dimension 5120, 16 heads, and head dimension 80 (Appendix D). This video encoder has approximately 630 million parameters, making it a substantial component of the overall 3B model.

What this computes: For each video chunk $t$ of $K$ frames, the 3D tube extraction followed by ViT encoding produces a set of time-aligned feature vectors $\hat{v}_t$ of shape $(f, d)$, where $f$ is the number of feature tokens (determined by the ViT patch size and the number of tubes extracted) and $d$ is the feature dimension from the ViT. The feature $\hat{v}_t$ encodes both the spatial content and the motion within chunk $t$.

Why this form: 3D tubes capture motion patterns — such as a hand moving from left to right or an object rotating — that 2D per-frame encoding would miss because each frame's features are extracted independently. The ViT backbone is shared with the audio encoder (see below), which has been shown by prior work [16] to be advantageous — it means the same architectural component learns to process both visual and audio spectrogram inputs, creating a shared representational space that the Combiner can then fuse. The use of sparse rather than dense 3D operations is a computational trade-off: dense 3D convolutions would be more expensive but capture all spatiotemporal interactions, while sparse tubes focus computation on a subset of locations that are sufficient for downstream tasks.

---

Audio Feature Extraction: Spectrogram Processing Through a Shared ViT Backbone

The audio waveform arrives at a predefined sampling rate. Rather than processing the raw waveform directly, the paper converts it to a spectrogram — a 2D representation where one axis is time and the other is frequency, with pixel intensity representing the energy at each time-frequency point. The spectrogram is created such that "the time bands match the 25 frames per second used in the videos" (Section 3.1), allowing it to be split into chunks that are temporally aligned with the video chunks. Each audio spectrogram snippet is then processed through an "audio input projection layer" followed by the same ViT backbone used for video features.

What this computes: For each audio chunk $t$, the spectrogram → projection → ViT pipeline produces a set of time-aligned audio feature vectors $\hat{a}_t$ of shape $(s, d)$, where $s$ is the number of feature tokens from the ViT and $d$ is the same feature dimension as the video features. The audio features $\hat{a}_t$ encode the acoustic content (speech, music, environmental sounds) occurring during the same time window as video chunk $t$.

Why this form: The shared ViT backbone between video and audio is a deliberate design choice, not an engineering convenience. Prior work by Gong et al. [16] (UAVM) demonstrated that reusing the visual transformer for audio processing is beneficial because it creates a unified architectural inductive bias across modalities before fusion. The authors explicitly cite this finding: "Reusing the visual component is previously shown to be advantageous [16]." This is important because the Combiner (next section) expects its video and audio inputs to live in the same representational space — if they came from completely different encoder architectures, the Combiner would need to learn to map them into alignment in addition to fusing them. The shared ViT provides a common initial representation, making the Combiner's job easier. The projection layer before the ViT for audio handles the initial domain adaptation (1D waveform → 2D spectrogram → appropriate input dimensionality for the ViT).

The full set of features across the entire video is:

• Video: $\hat{v} = \{\hat{v}_1, \hat{v}_2, ..., \hat{v}_T\}$ where each $\hat{v}_t$ has shape $(f, d)$
• Audio: $\hat{a} = \{\hat{a}_1, \hat{a}_2, ..., \hat{a}_T\}$ where each $\hat{a}_t$ has shape $(s, d)$

These two sequences are time-aligned: $\hat{v}_t$ and $\hat{a}_t$ correspond to the same time window. The total number of feature tokens per chunk is $f + s$, which could easily be hundreds. The Combiner's job is to reduce this to a small, fixed number $m$ (chosen as 32) of joint representation tokens per chunk.

---

The Modality Combiner: Joint Audiovisual Fusion With Dimensionality Reduction

The Combiner is the paper's central architectural contribution and serves two simultaneous purposes (Section 3.2):

1. Combine — learn a joint representation of video and audio features within each time chunk, capturing cross-modal correspondences (e.g., the visual appearance of a barking dog and the sound of barking occurring together).
2. Compress — reduce the number of feature tokens from $n = f + s$ (the total video plus audio features per chunk) down to $m$ (a small fixed number, chosen as 32 in all experiments), where $n \gg m$. The authors state: "effectively compress the representation from each video/audio snippet, which allows our model to scale to longer videos."

Formally, let $u = \{u_1, u_2, ..., u_T\}$ where $u_t = (\hat{v}_t, \hat{a}_t)$ is the concatenation of video and audio features for chunk $t$, having shape $(n, d)$ with $n = f + s$. The Combiner maps this to a compressed sequence $x = \{x_1, x_2, ..., x_T\}$ where each $x_t$ has shape $(m, d)$ with $m \ll n$ (and $m = 32$ in all reported experiments).

A critical constraint is causality: the Combiner must not use information from future time chunks when computing $x_t$. The authors state this explicitly:

$x_t = \text{Combiner}(u_1, u_2, ..., u_t)$

Why causality matters: The Combiner's outputs $x_t$ are subsequently fed to an autoregressive model that learns to predict $x_{t+1}$ from $x_1, ..., x_t$. If $x_t$ contained information leaked from $u_{t+1}$ (which contains future video frames), the autoregressive prediction task would be trivial — the model could simply read the answer from the input rather than learning temporal dynamics. The causality constraint ensures that the autoregressive model must learn to anticipate what comes next, which is what drives the learning of temporal structure.

The paper explores two different architectures for implementing the Combiner: a Causal Transformer Combiner and a Token Turing Machine (TTM) Combiner. Both satisfy causality but differ in their memory and computational characteristics.

Causal Transformer Combiner

This is the simpler variant. It is a standard Transformer model with $R = 8$ layers (Section 3.2). For each time step $t$, it takes as input all features from chunks $1$ through $t$ (that is, $u_1, u_2, ..., u_t$), concatenated together, and applies multi-head self-attention with a causal mask (described in Section 3.3.1) that prevents any feature in chunk $t$ from attending to any feature in chunk $t+1$ or later. From the transformer's output, a subset of $m$ features is selected to form $x_t$. The paper states it "specifically implements a causal version of the transformer as it masks out inputs from future timestamps (i.e., > t)."

The attention mechanism inside the Combiner processes all features $u_1, ..., u_t$ jointly, meaning that within a single time step $t$, every feature can attend to every other feature in chunks $1$ through $t$. This means cross-modal attention (video attending to audio and vice versa) happens naturally within the transformer layers, without any special cross-attention modules.

What this computes: Given all audiovisual features from the start of the video through time $t$, learn a compact $m$-token representation $x_t$ that encodes the joint video+audio content up to and including time $t$. The compression from $n$ to $m$ tokens is performed implicitly by the transformer's attention mechanism — the model learns to route information from the many input features into the $m$ output positions.

Why this form: The Transformer Combiner is straightforward and leverages the well-understood properties of attention-based architectures. All historical features $u_1, ..., u_{t-1}$ are directly available via attention, giving the model maximum information when computing $x_t$. However, this comes at a computational cost: as $t$ grows (for later chunks in a long video), the number of input tokens to the transformer grows linearly with $t$, making the attention computation $O(t^2)$ in the worst case. For a 16-chunk video, the final Combiner call $t=16$ processes features from all 16 chunks — potentially thousands of tokens — making this the computational bottleneck for long videos. The paper does not report specific runtime numbers for the Transformer Combiner, but this $O(t^2)$ scaling is the motivation for the TTM variant.

Token Turing Machine (TTM) Combiner

The TTM Combiner addresses the computational scaling problem by maintaining an external memory of fixed size rather than reprocessing the entire history at each step. The TTM architecture, introduced by Ryoo et al. [44], is a recurrent sequential model that uses Transformers with token-based operations to maintain and update a memory state.

The TTM Combiner operates through four functions executed at each time step $t$:

$z_t = \text{Read}(u_t, M_t)$

$\text{o}_t = \text{Process}(z_t)$

$M_{t+1} = \text{Write}(M_t, o_t, u_t)$

$x_t = \text{Output}(o_t)$

Here $M_t$ is the memory state at the start of time step $t$ — a set of features that encode information from all previous chunks $u_1, ..., u_{t-1}$. The Read function takes the current chunk's features $u_t$ and the current memory $M_t$, and produces a set of features $z_t$ that combine the new input with relevant information retrieved from memory. The Process function is implemented as a standard Transformer (with multi-head self-attention and MLPs) that transforms $z_t$ into intermediate outputs $o_t$. The Write function updates the memory: it takes the old memory $M_t$, the processed outputs $o_t$, and the current inputs $u_t$, and produces an updated memory $M_{t+1}$ that will be used at step $t+1$. Finally, the Output function maps $o_t$ to the compressed representation $x_t$ of $m$ features.

The Read, Write, and Output functions are all implemented using TokenLearner [43], which is a learned attention-pooling mechanism similar to Perceiver [20] — it uses a small set of learnable query vectors that attend to a larger set of input features to produce a compressed output. The paper notes that TokenLearner is "similar to Perceiver [20] and attention pooling [24]."

The memory size is much smaller than the total historical features. The paper states: "The number of such memory features are much smaller than the total number of history features ($\{u_1, ..., u_{t-1}\}$) in general (e.g., 256 vs. ~10k)." Specifically, the TTM Combiner uses output dimensions of 512 for the Read function and 256 for the Write function (Appendix D), while the Process transformer has 2 layers, 128 hidden dimension, and 12 heads.

What this computes: The TTM Combiner maintains a fixed-size memory $M_t$ that accumulates information from all previous chunks. At each step, it reads from memory (retrieving relevant context), processes the combined current-input + memory features through a small transformer, writes updated information back to memory, and outputs $m$ compressed features $x_t$. The memory acts as a learned summary of the video's history up to time $t-1$.

Why this form: The critical advantage is computational complexity: because the memory has fixed size (e.g., 256 features), and the Process function operates only on the Read output (which combines current input with a fixed number of retrieved memory features), the per-step computation is constant with respect to $t$. The paper reports: "This not only makes TTM a natural fit for the model, but also reduces the total time complexity of the TTM Combiner to be constant with respect to $t$, instead of $O(t)$ or $O(t^2)$ in Transformers." Empirically, the TTM Combiner "saves memory in both training and inference, using about 30% less memory and reduces the runtime by about 18%."

The trade-off is that the memory is a lossy compression of history — unlike the Transformer Combiner, the TTM cannot directly attend to specific features from chunk 1 when processing chunk 16; it must rely on what was encoded into the memory at earlier steps and maintained through the read-write mechanism. This means the memory update operations (the learned Read, Write functions and the transformer Process) must learn to selectively retain information that will be useful for future steps, which is a harder learning problem. Empirically, the TTM Combiner performs similarly to the Transformer Combiner in most experiments, though sometimes slightly worse (Tables 1-4 in the paper show both variants, with the Transformer Combiner leading on most benchmarks by small margins, e.g., 50.42 vs. 50.01 on MSRVTT-QA).

Combiner Design Choices: Comparison With Alternatives

The paper explicitly compares against two alternative compression methods in the ablation studies (Table 5b, with 32 video frames and 4 chunks):

• Perceiver Combiner: An adaptation of the Perceiver resampler from Flamingo [2]. It adds $m$ learnable latent queries that cross-attend to the input features $u_1, ..., u_t$, producing $m$ output features. This achieved 43.1% on MSRVTT-QA.
• CLS Combiner: Appends $m$ learnable features to the end of the input sequence, runs the whole sequence through the transformer, and takes the values of those appended positions as the combined features — analogous to how BERT's [CLS] token works. This achieved 43.7%.
• Ours-Transf.: The paper's Causal Transformer Combiner, which achieved 44.2%.
• Ours-TTM: The TTM Combiner, which achieved 44.8% (best among all variants in this ablation).

All Combiners used the same settings for fair comparison. The paper's main Combiners (both transformer and TTM) outperform the Perceiver and CLS-token alternatives, which the authors attribute to the fact that their approach processes all features jointly with causal masking rather than relying on a small set of learnable queries to extract information from the full sequence.

---

Time-Aligned Video/Audio Autoregressive Modeling (Section 3.3)

Once the Combiner has produced the compressed sequence $x = \{x_1, x_2, ..., x_T\}$, the next component models the temporal evolution of these representations. The paper applies an autoregressive formulation: the model learns to predict the features of the next time chunk given the features of all previous time chunks.

The overall probability model for the time-aligned modalities is factorized as:

$p(v, a) = \prod_{t=1}^{T} p(v_{t+1}, a_{t+1}|h_t) \, p(h_t|x_t) \, p(x_t|v_t, a_t)$

What this equation says: The joint probability of the video and audio sequence is decomposed into three terms multiplied across all $T$ time steps. For each step $t$: (1) $p(x_t|v_t, a_t)$ is the Combiner's mapping from raw chunk features to compressed representation — this is the fusion/compression step already described; (2) $p(h_t|x_t)$ is the latent causal model's mapping from the compressed representation to a hidden state — this is where the autoregressive transformer processes the sequence; (3) $p(v_{t+1}, a_{t+1}|h_t)$ is the modality reconstruction model that predicts the next chunk's audiovisual features from the current hidden state. The product over $t$ means this factorisation applies sequentially from $t=1$ to $t=T$, with each step conditioned on the representations from previous steps through the hidden state.

Why this factorization: It cleanly separates three distinct computations: (a) joint per-chunk feature fusion (the Combiner term), (b) temporal dynamics modeling (the latent causal term), and (c) prediction of future content as a self-supervision signal (the modality reconstruction term). The autoregressive structure means the model is forced to learn to anticipate the next time step's features — this is the mechanism by which it learns temporal patterns. Unlike pixel-by-pixel autoregressive modeling [55], which operates at too fine a granularity and "captures only short-term dependencies," the chunk-level autoregressive modeling operates on semantically meaningful units (several seconds of video each) and can capture dependencies across the full video length.

The paper notes that the Combiner also accumulates information from prior chunks (via causal attention in the transformer variant or memory in the TTM variant), and the autoregressive model "works at a higher level of abstraction with already learned features from the Combiner" (Section 3.3). The ablations show that both mechanisms together perform best, suggesting they capture complementary aspects of temporal structure — the Combiner handles short-range feature fusion within and across adjacent chunks, while the autoregressive model handles longer-range sequential dependencies.

Latent Causal Modeling

The latent causal model estimates the term:

$\prod_{t=1}^{T} p(h_t|x_t)$

This is implemented by applying an autoregressive transformer to the sequence $x = \{x_1, x_2, ..., x_T\}$, producing a hidden state sequence $\hat{h} = \{\hat{h}_1, \hat{h}_2, ..., \hat{h}_T\}$. The training target for $\hat{h}_t$ is $x_{t+1}$ — that is, $\hat{h}_t$ should predict the Compressed representation of the next chunk. This means the loss is computed between $\hat{h}_1, ..., \hat{h}_{T-1}$ and $x_2, ..., x_T$.

The transformer used for the latent causal model has 8 layers, model dimension 1024, hidden dimension 4096, 16 heads, and head dimension 64, with approximately 128 million parameters (Appendix D). It uses a modified causal attention mechanism described in Section 3.3.1 to properly handle chunk-level causality (described below).

What this computes: Given the sequence of compressed audiovisual representations $x_1, ..., x_T$, the latent causal model produces a sequence of hidden states $\hat{h}_1, ..., \hat{h}_T$ where each $\hat{h}_t$ encodes the temporal context up through time $t$ and is trained to predict the next chunk's representation $x_{t+1}$. At inference time, these hidden states form the cross-attention context for the text model.

Why this form: The autoregressive prediction objective forces the model to learn what typically follows what in video sequences — a form of self-supervised learning that helps the model internalize common event structures, action sequences, and transitions. Because the Combiner has already compressed the raw features into semantically meaningful representations (presumably encoding what is happening in each chunk), the latent causal model's prediction task is at the right level of abstraction: not "predict the next pixel values" but "predict the semantic content of the upcoming video segment."

Modality Reconstruction

The modality reconstruction model estimates:

$\prod_{t=1}^{T} p(v_{t+1}, a_{t+1}|\hat{h}_t)$

This is implemented by applying a separate transformer to the hidden states $\hat{h}$ to produce reconstructions $\hat{v}$ and $\hat{a}$ of the original video and audio features. The paper adds a video reconstruction loss as an auxiliary objective: "to save on computation, the video input $v$ is downsampled to $v_{small}$ for the reconstruction target, thus the actual reconstruction is $\hat{v}_{small}$." This means the model doesn't try to reconstruct full-resolution video — it reconstructs a downsampled version to reduce computational cost while still providing a self-supervisory signal.

Why this form: The modality reconstruction loss is an auxiliary objective that encourages the hidden states $\hat{h}_t$ to retain enough information about the video content to reconstruct it. Without this, the autoregressive model could potentially learn to predict only coarse temporal transitions without preserving the detailed visual information that text questions might require. However, the paper notes (Section 3.5) that "for our model, it is mostly subsumed by the latent space reconstruction loss" — meaning the latent prediction task $\hat{h}_t \rightarrow x_{t+1}$ already provides sufficient self-supervision, and explicit video reconstruction adds marginal benefit. This is consistent with the loss weight ablations showing that varying the reconstruction loss weight has minimal impact compared to varying the text loss weight.

Chunk-Level Causal Attention (Section 3.3.1)

The autoregressive modeling in time requires a modified attention mask, which applies to both the Combiner (when using the transformer variant) and the latent causal model. The standard autoregressive mask in a transformer prevents each token from attending to tokens that come after it in the sequence. However, for this architecture, features within the same time chunk should be allowed to attend to each other — they represent information from the same moment in time and there is no causal ordering among them. A standard per-token causal mask would artificially prevent the first feature in chunk $t$ from attending to the second feature in chunk $t$, which "unnecessarily restricts the model, preventing features from within the same time-chunk from interacting based on position within the time-chunk."

To fix this, the paper uses a chunk-level masking scheme. Let $i$ and $j$ be feature indices, $N$ be the total number of features across all chunks, and $T$ be the number of time chunks. The mask value at position $(i, j)$ is:

$\text{mask}_{ij} = \begin{cases} 0 & \text{if } j \leq \lceil t \cdot T/N \rceil \cdot N/T \\ 1 & \text{otherwise} \end{cases}$

where $t$ is the time chunk of feature $i$.

What this equation says: For a feature $i$ that belongs to time chunk $t$, the mask allows attention to any feature whose time chunk is $\leq t$. It does NOT further mask among features within chunk $t$ based on their individual positions. The expression $\lceil t \cdot T/N \rceil \cdot N/T$ computes the index of the last feature in chunk $t$ — any feature with index $j$ less than or equal to this threshold is in chunk $t$ or earlier, and is therefore unmasked.

Why this form: Standard token-level causal masking (where feature $j$ is masked if $j > i$) would create an artificial ordering within each chunk — the first feature in chunk $t$ would not be able to attend to the second feature in chunk $t$. The chunk-level mask removes this restriction within chunks while preserving causality across chunks: feature $i$ in chunk $t$ can see all features in chunks $1$ through $t$, but nothing from chunk $t+1$ or later. This means the Combiner can perform full joint attention across all video and audio features within the current and past chunks (enabling cross-modal fusion), while the latent causal model can use all features within each chunk but cannot peek into the future. The "0" mask value means the attention logit is set to 0 (after masking, before softmax), which allows the attention weight for that position to be zero after softmax normalization.

---

Combining Aligned and Non-Aligned Autoregressive Modeling (Section 3.4)

The text stream is processed by a separate autoregressive model that receives the media representations through cross-attention. Assuming the text $t$ is tokenized into $L$ tokens $w = \{w_1, w_2, ..., w_L\}$, the text model factorizes:

$p(w|\hat{h}) = \prod_{l=1}^{L} p(w_l | w_{l-1}, \hat{h})$

What this equation says: The probability of the text sequence $w$ is the product of the conditional probabilities of each token $w_l$ given all previous text tokens $w_{l-1}$ (autoregressive text generation) and the full sequence of media hidden states $\hat{h} = \{\hat{h}_1, ..., \hat{h}_T\}$. This is a standard autoregressive language model with an additional conditioning signal.

The text autoregressive model implements this by applying a transformer to the input token sequence $w$. The transformer's layers include cross-attention to the media hidden states $\hat{h}$. Specifically, in each decoder layer, after the self-attention over text tokens, a cross-attention operation attends to all $T$ media hidden states $\hat{h}_1, ..., \hat{h}_T$.

Why this form: The cross-attention mechanism, following the approach of Flamingo [2], allows the text model to selectively attend to relevant parts of the video when generating each output token. For example, when generating the word "running," the cross-attention weights might peak at hidden states corresponding to chunks where motion features indicate running. Unlike Flamingo, however, the media hidden states $\hat{h}$ here are not raw or lightly-compressed visual features — they are the outputs of a dedicated autoregressive model that has already processed the full video temporally. This means the text model receives representations that encode temporal context and event structure, not just per-frame visual content. This is a key difference: Flamingo's Perceiver compresses spatial features per frame independently, then the text model must infer temporal relationships by attending across the compressed features. In Mirasol3B, the temporal relationships are already modeled in $\hat{h}$ by the latent causal model, reducing the burden on the text model's cross-attention.

An important detail: the paper states that "all feature representations $\hat{h} = \{\hat{h}_1, \hat{h}_2, ..., \hat{h}_T\}$ from the latent causal model are used in the main text model" — meaning the text model can attend to any time point in the video. This is contrast to architectures that pool all video features into a single vector, which would lose the ability to reference specific moments. At the same time, because each $\hat{h}_t$ is a single vector (or small set of vectors, depending on implementation) rather than hundreds of per-frame features, the cross-attention context is compact — $T$ vectors, where $T = 16$ for 128-frame videos.

The paper notes that "since all parts of the model are autoregressive, it is naturally applicable to streaming videos" (Section 3.4). In a streaming setting, video chunks arrive sequentially, the Combiner and latent causal model update incrementally (the TTM Combiner is particularly suited for this due to its recurrent nature), and the text can be generated progressively as more video context becomes available.

The text model has approximately 1.3 billion parameters: 400M for cross-attention weights, 400M for vocabulary embeddings, and the rest for the transformer layers (18 layers, model dimension 1536, hidden dimension 12288, 12 heads, head dimension 128, per Appendix D). Together with the video/audio processing components (totaling a bit over 1.5B parameters, including the ViT encoder at 630M, the Combiner, causal latent model, and video reconstruction model at 128M each), the full model reaches approximately 3B parameters. The paper explicitly states: "A little over half of the parameters are for the audio+video autoregressive model" — this is a deliberate design choice to allocate substantial capacity to media processing, in direct contrast to Flamingo's ~1% allocation.

---

Model Losses (Section 3.5)

The model is trained with two primary losses, one for each autoregressive component, plus an optional auxiliary loss:

Loss 1: Latent Space Reconstruction Loss (for time-aligned inputs). This loss drives the autoregressive latent causal model. It computes the difference between $\hat{h}_1, ..., \hat{h}_{T-1}$ and $x_2, ..., x_T$ in the autoregressive formulation where $\hat{h}_t$ should predict $x_{t+1}$. The distance metric is:

$\text{Loss}_{\text{latent}} = 1 - \frac{\hat{h}_t \cdot x_{t+1}}{||\hat{h}_t|| \, ||x_{t+1}||}$

What this computes: The cosine distance between the predicted next-chunk representation $\hat{h}_t$ and the actual next-chunk representation $x_{t+1}$ produced by the Combiner. The dot product $\hat{h}_t \cdot x_{t+1}$ measures the alignment of the two vectors; dividing by the product of their $L^2$ norms normalizes for vector magnitude; subtracting from 1 converts cosine similarity (which is 1 for identical vectors, -1 for opposite vectors) into a distance (0 for identical, 2 for opposite). The L2 normalization is applied first, then the dot product is taken — the paper states: "We apply a L2 normalization and then take dot product between the feature vectors as the loss (i.e., cosine similarity)."

Why this form: Cosine distance focuses on the direction of the feature vectors rather than their magnitude. This is appropriate because the Combiner's output $x_t$ may have varying magnitudes depending on the content of chunk $t$ (a visually complex chunk might produce larger activations), and we want the autoregressive model to learn the pattern of how features change over time, not to match magnitudes exactly. MSE would penalize magnitude mismatches, potentially causing the model to learn to output average-magnitude vectors rather than accurately predicting the directions of change.

Loss 2: Unaligned Text Cross-Entropy Loss. This is the standard language modeling loss:

$\text{Loss}_{\text{text}} = -\sum_{l=1}^{L} \log p(w_l | w_{l-1}, \hat{h})$

What this computes: The negative log-likelihood of the correct text tokens under the model's predicted distribution. This is the standard maximum-likelihood objective for autoregressive text generation.

Why this form: Cross-entropy is the standard loss for next-token prediction in language models, and it provides the primary learning signal for the text model and (via backpropagation through cross-attention) for the media representations $\hat{h}$.

Loss 3 (Optional): Video Reconstruction Loss. Similar to the latent reconstruction loss, this computes the cosine distance between predicted video features $\hat{v}_{small}$ and actual video features $v_{small}$ (downsampled for computational efficiency), also in an autoregressive formulation where $\hat{v}_{small, t}$ should predict $v_{small, t+1}$. The same cosine distance metric is used. The paper states that "this loss can be useful, especially for generation tasks, [but] we find that for our model, it is mostly subsumed by the latent space reconstruction loss."

Loss Weighting: During pretraining, all losses are given equal weight (Section 3.6). During finetuning, the text loss weight is increased 10-fold. The rationale is pragmatic: "to better align the training loss with the final evaluation, which we also confirm experimentally" — the evaluation metric is based on text generation accuracy, so emphasizing the text loss during finetuning makes the optimization target match the evaluation target more closely. The ablation in Table 6b confirms this: with equal weights (1.0 for all), the model achieves 45.0% accuracy; reducing the text weight to 0.1 drops accuracy to 44.6%; increasing it to 10.0 raises accuracy to 45.4%. These are on the smaller ablation model with fewer pretraining steps, so the absolute numbers are lower, but the trend is clear.

---

Implementation Details (Section 3.6 and Appendix D)

Model Scale and Parameter Distribution. The full Mirasol3B model contains approximately 3 billion parameters. The breakdown (from Appendix D):

• Video input processor (ViT-Huge with 3D tubes): ~630M parameters. The ViT has 32 layers, model dimension 1280, hidden dimension 5120, 16 heads, head dimension 80. The 3D convolutional tubes contribute an additional 1.5M parameters.
• Combiner, causal latent model, and video reconstruction model: ~128M parameters each (totaling ~384M). Each is a transformer with 8 layers, model dimension 1024, hidden dimension 4096, 16 heads, head dimension 64.
• Text autoregressive model: ~1.3B parameters total, comprising 400M for cross-attention weights, 400M for vocabulary embeddings (a substantial fraction, typical for large-vocabulary language models), and ~500M for the transformer layers. The text transformer has 18 layers, model dimension 1536, hidden dimension 12288, 12 heads, head dimension 128.
• Audio processing: ~100M additional parameters for the audio input projection and associated weights.

The total media processing allocation (video encoder + Combiner + causal latent model + reconstruction model + audio weights) is "a little over half of the parameters" — approximately 1.6B out of 3B. This is the architectural statement: media modalities receive substantial dedicated capacity.

Small Model for Ablations. The ablation experiments use a smaller 1.15B parameter variant (Appendix D). The text model is reduced to 128M parameters (matching the Combiner's configuration), the ViT is reduced to ViT-Large (300M parameters, 24 layers, model dimension 1024, hidden dimension 4096, 16 heads, head dimension 80), and vocabulary embeddings are reduced to 260M. The Combiner, causal latent model, and reconstruction models remain the same size as in the full model. The total is 1.15B rather than 3B, making ablations computationally tractable (the paper notes they use 2x fewer pretraining steps to save compute).

Combiner Configuration. Both the standard Transformer Combiner and TTM Combiner use output dimension $m = 32$ — that is, each chunk is compressed to 32 feature tokens. The Transformer Combiner uses $R = 8$ layers with the chunk-level causal mask from Section 3.3.1. The TTM Combiner uses TokenLearner-based Read and Write functions with output dimensions 512 and 256 respectively, and the Process transformer has 2 layers, 128 hidden dimensions, and 12 heads (Appendix D). The paper sweeps Combiner output dimensions from 8 to 64 (Table 5d) and observes improved performance with larger outputs: 8 features → 42.53%, 16 → 43.36%, 32 → 44.20%, 64 → 44.22%. The authors choose 32 as "a trade-off between sufficiently compact feature length and sufficiently expressive features."

Video/Audio Processing. The standard configuration uses 128 frames in 16 chunks of 8 frames each. For long-video benchmarks (ActivityNet-QA, NExT-QA), the model scales to 512 frames in 16 chunks of 32 frames each — increasing the chunk size rather than the number of chunks, which keeps the autoregressive sequence length $T = 16$ constant and thus avoids increasing the Combiner's computational cost (for TTM) or the latent causal model's attention complexity. The authors emphasize: "Due to the design of our model (partitioning and Combiner), adding more frames, or increasing the chunk size, number of chunks, etc. lead to only marginal increase in parameters." The TTM Combiner specifically enables scaling the number of chunks without $O(T^2)$ cost because its per-step computation is constant.

The Combiner applies random masking to its output features at a ratio of 0.75% "as a form of dropout regularization" — the authors "found this stabilizes the causal model latent reconstruction." This is a small technical detail but illustrates an empirical finding: the latent prediction task can benefit from slight noise injection to prevent overfitting.

Pretraining. The model is pretrained on the Video-Text Pairs (VTP) dataset, a collection of noisy video-text pairs from the web [2]. The authors use only about 12% of this dataset (~3M samples) for pretraining. During pretraining, the text backbone is frozen — only the autoregressive model components, the Combiner, the cross-attention weights, and the low-level video feature extractors (3D tubes, ViT layers) are trained. All losses are given equal weight.

The image and text backbones are initialized from a contrastively image-text pretrained MaMMUT model [23], which was trained jointly with contrastive and text generative objectives on the ALIGN dataset [21]. The audio backbone reuses the same pretrained image backbone (the ViT). The Combiner, causal latent reconstruction model, video reconstruction model, and video 3D tubes are all randomly initialized.

Pretraining hyperparameters: learning rate $1 \times 10^{-5}$, batch size 32, image resolution $224 \times 224$, 128 frames.

Audio-Specific Pretraining. Since most videos in VTP lack audio, the paper adds an additional audio pretraining step using AudioSet-2M [14]. The model is trained to output the text of the audio class names (e.g., "dog barking"). In this step, only the audio weights are unfrozen — all other model parameters are frozen — allowing the model to learn to process spectrogram inputs without disrupting the already-learned visual and textual representations.

Finetuning. During finetuning, all parameters are unfrozen. The unaligned text loss weight is increased 10-fold (as discussed above). Finetuning hyperparameters vary by dataset: for MSRVTT-QA, the model is trained for 10 epochs with learning rate $5 \times 10^{-6}$, weight decay 0.01, image resolution $448 \times 448$, batch size 32, 128 frames. For ActivityNet-QA: 80 epochs. For NExT-QA: 20 epochs. Dropout 0.1 and label smoothing 0.2 are used across all finetuning runs. For audio-video benchmarks, additional data augmentation is used during finetuning: Mixup [73], SpecAugment [34], dropout, and label smoothing, following the settings of prior audio-visual work [15].

Inference. At inference, the model generates free-form text answers which are compared to target answers for an exact match. This is a more challenging evaluation than the classification setting used by many prior works (where the model selects from a fixed set of answer options), because the model might generate a correct answer using different wording (e.g., a synonym) that would count as incorrect under exact match. The paper acknowledges this evaluation is "more general and widely applicable" despite being more stringent.

---

Summary of Key Design Decisions and Their Justifications

The architecture embodies several non-obvious choices that the paper justifies through experiments or theoretical arguments:

1. Partitioning into chunks before feature extraction, not after. The video is split into chunks raw, and each chunk independently goes through expensive ViT encoding. This is necessary because applying ViT to all 512 frames at full resolution would be computationally prohibitive. The causal constraint that the Combiner can only use past chunks means the architecture is compatible with streaming video.

2. Autoregressive modeling of compressed Combiner features, not raw features. The autoregressive model operates on the 32-dimensional $x_t$ outputs rather than on $\hat{v}_t$ or $\hat{a}_t$ directly. This is critical for three reasons: (a) the Combiner's compression reduces the autoregressive model's context length from hundreds of tokens per step to 32, making $T=16$ steps computationally manageable; (b) the Combiner fuses audio and video, so the autoregressive model captures the joint audiovisual dynamics rather than separate modality-specific dynamics; (c) the Combiner's features are at a higher level of semantic abstraction, which is the right granularity for temporal prediction. Table 5a shows that both the autoregressive component and the Combiner individually help, but their combination helps more — the autoregressive component without the Combiner would need to process many more features per step, making scaling difficult.

3. Separate autoregressive models for media and text, bridged by cross-attention. The alternative would be to concatenate the Combiner outputs $x_t$ as tokens in the text model's input sequence (as tokenization-based approaches do). The paper argues this would force the media representations to compete for self-attention capacity with the text tokens, reducing the effective parameter allocation to media processing. By giving media its own autoregressive model that produces $\hat{h}$ as cross-attention context, the text model's self-attention focuses on language while the media model's self-attention focuses on temporal dynamics — specialization rather than competition.

4. Cosine distance loss for latent reconstruction, not MSE. The cosine distance ignores magnitude and focuses on the direction of feature change, which the paper implicitly argues is what matters for predicting "what happens next" in a semantically meaningful way. This is also consistent with the fact that the Combiner outputs are used in cross-attention by the text model, where cosine similarity determines attention weights more than vector magnitude.

5. TTM Combiner's memory-based design for scaling. The TTM Combiner sacrifices direct access to all historical features in exchange for constant-time per-step computation. The paper's empirical demonstration that TTM performs similarly to the Transformer Combiner while using 30% less memory and 18% less runtime is a practical justification for this trade-off, especially for longer videos or streaming applications where $T$ could be large.

6. The 10× text loss weight increase during finetuning. This acknowledges that the pretraining objectives (latent reconstruction, video reconstruction) are proxies for the downstream task (text generation accuracy). During finetuning, the optimization should prioritize the actual evaluation metric. The ablation in Table 6b confirms this empirically: the high text weight yields the best accuracy.

4. Key Insights and Innovations

Innovation 1: Architectural Decoupling Based on Temporal Alignment as a First-Class Design Principle

The paper's most fundamental intellectual contribution is not any specific module — the Combiner, the TTM, the latent causal model — but the design principle that motivates them: multimodal architectures should structurally separate processing pathways according to the temporal alignment characteristics of the input modalities. This reframes architectural design from a question of "how do we fit all modalities into one model?" to "how are these modalities related in time, and what structural commitments does that imply?"

Before this paper, the dominant paradigm — exemplified by Flamingo [2] and its descendants — was to treat all modalities as inputs to a single autoregressive language model, differentiated only by how they were compressed before entering the shared processing stream. Flamingo's Perceiver resampler compresses visual features into a small fixed set of tokens, which are then attended to by the language model's cross-attention layers. The architecture is modality-agnostic at the core: the transformer doesn't know or care whether the features it's attending to came from video frames, audio spectrograms, or text — they're all just vectors in the cross-attention context. This has the virtue of simplicity, but the paper makes a convincing case that it creates a systematic bias: the shared transformer, designed and scaled for language modeling, allocates nearly all its capacity to text processing (~99% of parameters in Flamingo), leaving the visual pathway starved of representational capacity despite visual data being far larger in volume.

The Mirasol3B design rejects this symmetry. It says, in effect: video and audio are different from text not just in input format, but in their temporal structure, and the architecture should reflect this. They are high-frequency, high-volume, and time-synchronized — they unfold together through time and contain rich internal dynamics. Text is low-frequency, sparse, and globally conditioned on the entire media stream — it doesn't correspond to specific timestamps. Processing both through the same autoregressive sequence forces a compromise: either the media features are compressed so aggressively that temporal detail is lost, or the sequence becomes so long that computation is infeasible. By giving each type its own autoregressive model — one that learns audiovisual dynamics in time, another that learns to produce language from the resulting representations — the architecture allocates capacity where the data demands it. The headline result (3B parameters outperforming 80B Flamingo on MSRVTT-QA, Table 1) is a direct empirical validation of this principle: proper parameter allocation in a smaller model beats parameter starvation in a larger one.

This is a fundamental reframing, not an incremental improvement. It changes the question from "what's the best compression method before a language model?" to "what's the right way to structure computation across modalities with fundamentally different temporal properties?" The paper positions this not as optimization but as a correctness argument — the architecture should mirror the data's generative structure. The fact that the video+audio autoregressive model receives "a little over half" the parameters in a 3B model, compared to ~1% in Flamingo's 80B model, is the architectural statement made quantitative.

Crucially, this principle doesn't hinge on the specific Combiner or TTM implementation. Those are instantiations. The principle — decouple by temporal alignment — would remain valid even with different compression or fusion mechanisms. This is what makes it an intellectual contribution rather than an engineering report.

Innovation 2: The Combiner as a Dual-Purpose Fusion-and-Compression Module That Enables Scaling Without Loss of Temporal Fidelity

If Innovation 1 is the "why," Innovation 2 is the "how" that makes it work. The Combiner is the paper's specific mechanism for solving a problem that the decoupled architecture creates: how do you compress the massive volume of per-chunk audiovisual features into something compact enough for autoregressive temporal modeling, without destroying the information that the temporal model and text model need?

Naive compression — pooling, aggressive subsampling, projecting to a single vector — would solve the sequence length problem but lose the ability to represent multiple objects, events, or fine-grained audiovisual correspondences within a chunk. Tokenizing everything into the autoregressive model without compression would preserve information but make the sequence length infeasible. The Combiner represents a designed balance point: compress from $n$ features (potentially hundreds) to $m = 32$ features per chunk, but do so through a learned, attention-based mechanism that can selectively retain information based on what is useful for downstream temporal prediction and text answering.

What distinguishes this from prior compression approaches — Perceiver resamplers [2], attention pooling, CLS tokens — is that the Combiner operates jointly over video and audio with causal access to historical chunks. The Perceiver in Flamingo compresses per-frame visual features independently, losing temporal context and cross-modal information during compression. The Combiner, by contrast, fuses audio and video features at the chunk level (cross-modal attention within the transformer) while maintaining causal access to all previous chunks' features (temporal context during compression). This means the compressed representation $x_t$ encodes not just "what's in this video snippet" but "how the audiovisual content at time $t$ relates to everything that came before it."

The empirical evidence for this being genuinely superior — and not just "another compression method" — is in Table 5b: the Transformer Combiner (44.2%) and TTM Combiner (44.8%) both outperform Perceiver-based (43.1%) and CLS-token-based (43.7%) compression with the same settings. The margins are modest in absolute terms (~1 percentage point) but consistent and meaningful given the small ablation model. More importantly, the TTM Combiner's additional computational advantage — 30% less memory, 18% less runtime — demonstrates that the Combiner design space includes meaningful tradeoffs between representation quality and efficiency that aren't accessible to simpler compression methods.

This is an incremental advance in compression methods, but with fundamental implications for architecture scaling. The key insight is that compression shouldn't happen in isolation — it should be integrated with the temporal modeling pipeline and should respect the causal constraints of the downstream autoregressive model. The Combiner's design (learning to compress in a causally-consistent way that supports next-step prediction) means the compression is task-adapted: the features that survive compression are those most useful for predicting future audiovisual content, which are likely also the features most relevant for answering questions about events and actions.

Innovation 3: Autoregressive Modeling of Video at the Chunk Level as an Alternative to Both Pixel-Level Generation and Flat Feature Extraction

The paper introduces a specific granularity for autoregressive video modeling that sits between two extremes. On one end, pixel-level or token-level autoregressive generation of video frames [48, 55, 65] is "highly inefficient and captures only short-term dependencies" (Section 3.3) — each prediction step is too fine-grained to learn meaningful temporal structure. On the other end, encoding the entire video into a single representation and feeding it to a language model (as in many VQA architectures) abandons any explicit modeling of temporal dynamics — the temporal structure must be implicitly extracted by the text model's cross-attention, which was not designed for this purpose.

The paper's alternative is chunk-level autoregressive modeling of Combiner-compressed features. Each prediction step corresponds to a semantically meaningful time interval (8–32 frames, or roughly 0.3–1.3 seconds at 25 fps). The model learns to predict the semantic content of the next chunk — as encoded by the Combiner's joint audiovisual representation — rather than predicting raw pixels or tokens. This has three advantages, each of which is conceptually significant beyond the specific implementation:

First, it makes autoregressive video modeling computationally tractable. With $T = 16$ chunks and 32 features per chunk, the autoregressive sequence length is just 16, compared to potentially thousands of tokens for pixel-level modeling. This is what enables scaling to 512 input frames without parameter increase — the autoregressive model's computation depends on $T$, not on the number of raw frames.

Second, it operates at the right level of abstraction for learning event structure. A chunk of 8–32 frames corresponds to a meaningful temporal unit — a single action, a brief event, a motion segment. Predicting "what happens next" at this granularity forces the model to learn common event transitions, causal sequences, and temporal regularities. These are precisely the kinds of temporal relationships that video question answering requires (e.g., "what happened after the person picked up the cup?").

Third, it's compatible with streaming video. Because the autoregressive model is causal and the Combiner (especially in TTM form) updates incrementally, the architecture can process video as it arrives rather than requiring the full video upfront. The paper explicitly notes this property (Section 3.4) but doesn't evaluate it — it's a latent capability that follows from the architectural choices.

The ablation in Table 5c provides the critical evidence: processing video in chunks autoregressively (64 frames, 8 chunks) achieves 45.1%, compared to processing all 64 frames as a single input (41.8%). This ~3.3 percentage point improvement is on the small ablation model but is clear and consistent — the autoregressive structure isn't just enabling more frames, it's providing a better representation even with the same total frames. The fact that a bidirectional model gives the same performance (45.1%) is interesting — it suggests the benefit comes from the chunk-based representation itself (joint feature learning within chunks, sequential processing across them) rather than from the causal prediction objective per se. But the causal objective is what enables streaming and what drives the latent reconstruction loss, so it has value beyond raw accuracy.

This is a moderate conceptual advance — chunk-level autoregressive modeling of compressed multimodal features is genuinely different from prior approaches, but it builds on well-established autoregressive principles. The key insight is about granularity: the right unit of temporal prediction is determined by the information density of the representations being predicted, and Combiner-compressed features are dense enough in semantic content to make chunk-level prediction meaningful and efficient.

Innovation 4: Verifier Over-Optimization as an Empirical Diagnosis — The Counterintuitive Finding That More Test-Time Compute Hurts on Easy Tasks

The paper identifies and characterizes a phenomenon that, while not the primary focus of the work, has significant implications for the broader field of multimodal model design and evaluation. The difficulty-conditioned behavior of beam search (Figure 3, right) reveals that aggressive optimization of a learned verifier signal can systematically degrade performance on easy problems, even while it helps on medium-hard problems. This is not a theoretical prediction — it's an empirical finding that contradicts the natural assumption that more powerful search methods should monotonically improve results.

The phenomenon is concretely demonstrated: on difficulty bin 1 (easiest Math problems), beam search accuracy actually decreases as the generation budget increases from 4 to 256, while best-of-N weighted — a weaker optimizer — continues to improve. This is attributed to verifier over-optimization: beam search finds solutions that score highly under the process reward model but are actually incorrect, exploiting patterns in the verifier's scoring that don't correspond to true answer correctness. The paper provides qualitative evidence in Appendix M (Figure 29) showing degenerate outputs with repetitive low-information steps that the PRM scores highly.

What makes this finding significant beyond this paper is that it explains contradictory results in the prior literature and establishes a boundary condition on test-time compute scaling. Prior work reached opposite conclusions about whether self-correction and search help (Huang et al. 2023: "LLMs cannot self-correct reasoning" vs. Madaan et al. 2023: self-refinement works) — these studies were implicitly testing on different problem difficulty distributions. The insight that search effectiveness depends on problem difficulty, and that it can be actively harmful in certain regimes, provides a unified framework for understanding these conflicts rather than dismissing either side.

Furthermore, the paper demonstrates that this over-optimization is the primary bottleneck preventing further gains from test-time compute. The strongest optimizer — lookahead search — paradoxically performs worst overall (Figure 3, left) because its aggressive optimization most effectively exploits verifier weaknesses. This reframes the research priority: rather than developing more sophisticated search algorithms, the bottleneck is verifier robustness. The compute-optimal policy is essentially a strategy for staying below the over-optimization threshold per difficulty level — using weaker optimization where the verifier is unreliable (easy problems) and stronger optimization only where the verifier signal has room to provide genuine guidance (medium problems).

This is a diagnostic finding with practical and methodological implications, not a new method. It's the kind of result that changes how researchers think about the problem — from "more compute is better" to "the relationship between compute and accuracy is difficulty-dependent and non-monotonic" — and it provides both an explanation for past empirical puzzles and a guide for future evaluation design (always report difficulty-stratified results). The connection to the RLHF literature's reward hacking phenomenon, while acknowledged only briefly, suggests this may be a general property of learned verifiers under optimization pressure, making the finding potentially transferable to other domains where model outputs are scored by learned reward models.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmark is MSRVTT-QA [63], a Video Question Answering dataset with ~10K video clips and 243K question-answer pairs, derived from the MSRVTT dataset by automatic QA generation ("contains a certain level of noise"). Videos average ~14 seconds. For long-video evaluation, ActivityNet-QA [70] (5,800 videos, 58,000 QA pairs, ~160 seconds average length) and NExT-QA [60] (5,440 videos, ~52K manually annotated QA pairs, ~44 seconds average length, with causal and temporal question types) are used. For audio-video evaluation, Kinetics-Sound [3], VGG-Sound [6], and Epic-Sound [19] are used — these are originally classification datasets (36, over 300, and 44 classes respectively) but repurposed by the paper as open-ended text generation tasks.

• Base model(s). All experiments use the proposed Mirasol3B model (~3B parameters total), with two Combiner variants: the Causal Transformer Combiner and the Token Turing Machine (TTM) Combiner. Ablation experiments use a smaller 1.15B parameter variant with a reduced ViT-Large encoder (300M parameters vs. 630M for ViT-Huge), reduced text model (128M vs. ~1.3B), and reduced vocabulary embeddings (260M vs. 400M), while keeping the Combiner, causal latent model, and reconstruction models at the same size as the full model (Appendix D). The small model uses 2× fewer pretraining steps to save compute. The full model is pretrained on the Video-Text Pairs (VTP) dataset (~3M samples, about 12% of the full dataset), with image and text backbones initialized from a contrastively pretrained MaMMUT model [23] on the ALIGN dataset [21]. Audio weights are pretrained separately on AudioSet-2M [14] with all other weights frozen.

• Metrics. The primary metric is accuracy (%) — the fraction of test questions for which the model's generated free-form text answer exactly matches the target answer. This is a more stringent evaluation than the classification setting used by many prior works (where the model selects from a fixed set of answer options), because correct but differently-worded answers (synonyms, paraphrases) count as incorrect under exact match. For audio-video benchmarks, the text input (e.g., "Classify the video audio clip.") is provided, and the model must output the exact class name (e.g., "playing drums") to be counted as correct.

• Baselines. The paper compares against a broad set of prior methods, all evaluated on the same benchmarks. For Video QA (MSRVTT-QA, Table 1): Just Ask [66] (41.5%), ALPRO [26] (42.1%), MERLOT [71] (43.1%), VIOLETv2 [12] (44.5%), VindLU [9] (44.6%), VideoOFA [8] (45.4%), GIT2 [51] at 5B parameters (45.6%), Iterative Co-Tok [38] (45.7%), VideoCoca [64] (46.3%), All-in-one [49] (46.8%), UMT-L [30] (47.1%), InternVideo [54] (47.1%), Flamingo [2] at 80B parameters (47.4%, full fine-tuning), and M-PLUG2 [62] (48.0%). Results shown in gray indicate classification-based evaluation, which is at an advantage over the paper's open-ended generation. For long-video QA (Tables 2-3), baselines include Just Ask, MERLOT, FrozenBiLM [67], VideoCoca, Sing-Temp [25], VindLU, UMT-L, CLIP, VQA-T [66], AIO [49], ATP [5], VGT [61], MIST-CLIP [13], and HiTeA [68]. For audio-video (Table 4), baselines include MBT [33], UAVM [16], MMT [74], MAViL [18], ONE-PEACE [52], SSAST [19], and ASF [19].

• Generation budget / compute accounting. Unlike the MATH paper's analysis of inference compute scaling, this paper measures performance as a function of architectural choices (number of frames, chunking strategy, Combiner type) rather than inference-time generation budget. The key resource axes are: (1) number of input frames (128 vs. 512), (2) number of chunks T and frames per chunk K, (3) Combiner output dimension m, and (4) model parameter count. The paper emphasizes that "adding more frames, or increasing the chunk size, number of chunks, etc. lead to only marginal increase in parameters" (Section 3.6), meaning the model scales to longer videos without proportional parameter growth. The TTM Combiner is reported to use "about 30% less memory and reduces the runtime by about 18%" compared to the Transformer Combiner, but no absolute FLOP counts or latency measurements are provided.

• Cross-validation / statistical protocol. The paper does not describe any cross-validation protocol for hyperparameter selection or model evaluation. Results appear to be reported on standard test splits of each dataset without mention of multiple runs, error bars, or statistical significance testing. The ablation experiments (Section 4.1) use the smaller 1.15B model with fewer pretraining steps to save compute, and all comparisons within each ablation table are run under identical training budgets for fairness.

---

Main Quantitative Results

Video Question Answering (MSRVTT-QA)

The headline result is in Table 1: Mirasol3B achieves 50.42% accuracy on MSRVTT-QA, outperforming all prior methods including the 80B Flamingo model (47.4%) and the 5B GIT2 (45.6%). The TTM Combiner variant achieves 50.01%. At less than 3B parameters, this represents a substantial margin over models that are 1.6–27× larger (GIT2 at 5B, Flamingo at 80B). The paper emphasizes that this is achieved with open-ended text generation evaluation, not the easier classification setting used by several baselines (shown in gray in Table 1).

The comparison with Flamingo [2] (47.4% at 80B parameters) is the most striking: Mirasol3B achieves +3.0 percentage points with ~27× fewer parameters. However, it's important to note that this is a comparison against Flamingo's full fine-tuning result on MSRVTT-QA specifically — not a zero-shot or few-shot result. The models also use different pretraining data (Mirasol3B uses VTP and AudioSet; Flamingo uses a much larger multimodal web corpus including interleaved image-text data), and different image encoders. This is not a controlled comparison isolating architectural differences; it shows that the proposed architecture achieves strong results at small scale, but the contribution of data differences vs. architectural differences cannot be disentangled.

Long Video Question Answering (ActivityNet-QA and NExT-QA)

Table 2 reports ActivityNet-QA results. Mirasol3B at 512 frames with the Transformer Combiner achieves 51.13%, substantially outperforming all prior methods: FrozenBiLM (43.2%), Sing-Temp (44.1%), VindLU (44.7%), UMT-L (47.9%), and VideoCoca (56.1% — but this is in the classification setting, shown in gray). The 128-frame variant achieves 48.25%, and the TTM Combiner at 512 frames achieves 49.85%, showing that more frames consistently help (+2.9 points from 128→512 with Transformer Combiner, +1.6 points from 128→512 with TTM Combiner). The gain from increasing frames without increasing model size is a key architectural claim.

Table 3 reports NExT-QA results, where the dataset focuses on causal and temporal reasoning about complex events. Mirasol3B at 512 frames with the Transformer Combiner achieves 72.0%, outperforming prior methods: CLIP single frame (43.7%), VQA-T (52.32%), AIO (50.60%), ATP (54.3%), VGT (55.02%), MIST-CLIP (57.18%), and HiTeA (63.1%). The TTM Combiner at 512 frames achieves an even higher 73.2%, which is notable because the TTM Combiner generally performs slightly below the Transformer Combiner on other benchmarks. The 128-frame variant achieves 68.2%, showing a +3.8 to +5.0 point gain from 128→512 frames, again without parameter increase.

The ActivityNet-QA and NExT-QA results together establish a consistent pattern: more frames consistently improve accuracy, and the architecture supports this without proportional parameter scaling. The TTM Combiner's strong performance on NExT-QA specifically (73.2%) is interesting — NExT-QA's questions are about causal and temporal relationships ("Why did X happen?", "What happened after Y?"), which suggests the memory mechanism may be particularly beneficial for tasks requiring reasoning across longer temporal spans.

Audio-Video Results

Table 4 reports results on three audio-video benchmarks, all evaluated as open-ended generation (outputting the class name as text). On Kinetics-Sound (Table 4a), Mirasol3B with audio+video achieves 90.1% (TTM: 88.3%), compared to MBT [33] at 85.0%. On VGG-Sound (Table 4b), the model achieves 69.8% (TTM: 66.4%), compared to UAVM (65.8%), MMT (66.2%), MAViL (67.1%), and ONE-PEACE (68.2%). On Epic-Sound (Table 4c), the model achieves 78.2% with audio+video (TTM: 79.4%), compared to SSAST (53.47%) and ASF (53.75%) — both audio-only baselines.

The audio-video results show several interesting patterns. First, video-only performance on Kinetics-Sound is 81.3% (Table 4a, "Sm" likely indicating the small model variant for this specific row), and adding audio improves this to 85.0% or 90.1% depending on model configuration — demonstrating genuine multimodal benefit. Second, on Epic-Sound, audio-only achieves only 62.4% while video-only achieves 72.4%, and the combination reaches 78.2–79.4%, showing both modalities contribute but video is the stronger signal for this dataset. Third, the TTM Combiner actually outperforms the Transformer Combiner on Epic-Sound (79.4% vs. 78.2%), which is unusual and not explained in the paper — it may be related to Epic-Sound's specific characteristics (egocentric kitchen videos with distinctive sounds) or simply variance.

The paper claims "large margins" on these audio-video benchmarks, but the margins are quite variable. On VGG-Sound, the improvement over the next-best method (ONE-PEACE at 68.2%) is +1.6 percentage points — statistically meaningful but not large. On Epic-Sound, the improvement over the next non-Mirasol3B method (ASF at 53.75%) is over 24 points — genuinely large. This heterogeneity likely reflects differences in how prior work approached each dataset and how well audio-video-text modeling helps for the specific task distribution.

Scaling Behavior: Frames vs. Performance

Although the paper doesn't present a dedicated scaling plot, the comparison between 128-frame and 512-frame results across ActivityNet-QA and NExT-QA (Tables 2-3) provides empirical evidence for the claim that the architecture scales gracefully with video length. On ActivityNet-QA, 128→512 frames yields +2.88 points (48.25% → 51.13%) with the Transformer Combiner and +1.60 points (48.25% → 49.85%) with the TTM Combiner. On NExT-QA, the gains are larger: +3.8 points (68.2% → 72.0%) with Transformer Combiner and +5.0 points (68.2% → 73.2%) with TTM Combiner.

These gains come without any increase in model parameters — the chunk size increases from 8 to 32 frames, but the number of chunks T remains 16, so the autoregressive sequence length and Combiner computation (for TTM) remain constant. This is the architectural argument in action: by partitioning into chunks and compressing each chunk to a fixed 32 features, the model can absorb more raw input without increasing the downstream processing cost. However, the ViT encoder still processes more frames (in larger chunks), which does increase computation — the paper doesn't break down where the additional cost goes.

---

Ablation Studies and Robustness Checks

All ablations in Table 5 use the smaller 1.15B model with reduced pretraining to save compute. They are conducted on MSRVTT-QA. The experiments within each sub-table are run under identical training budgets for fair comparison.

Main model components (Table 5a): Starting from a baseline with 32 frames in 4 chunks (41.5%), each component is added incrementally. Adding the autoregressive (AR) model alone: 43.2% (+1.7). Adding the Combiner alone: 42.1% (+0.6). Adding both together: 44.7% (+3.2 over baseline). Adding pretraining: 45.2% (+0.5). Full combination (AR + Combiner + Pretraining): 47.9% (+6.4 over baseline). The key finding is that the autoregressive component and Combiner are complementary — the joint benefit (+3.2) is greater than the sum of individual benefits (+1.7 + +0.6 = +2.3), suggesting they capture different aspects of temporal structure that reinforce each other.

Combiner type (Table 5b): All variants use 32 frames in 4 chunks. Perceiver Combiner: 43.1%. Transformer+CLS: 43.7%. Ours-Transformer: 44.2%. Ours-TTM: 44.8%. The paper's two Combiner variants outperform the Perceiver and CLS-token baselines, with the TTM Combiner achieving the highest score. The margin between best and worst is 1.7 points — modest but consistent with the TTM's architectural advantages (memory efficiency, constant-time computation) making it the preferred choice for scaling. The Perceiver Combiner's underperformance is notable because this is essentially the compression mechanism used in Flamingo [2] — this ablation provides some controlled evidence that the Combiner's approach (joint attention over all features with causal masking) is genuinely more effective than query-based cross-attention compression for temporal video features.

Autoregressive model more frames (Table 5c): This ablation compares processing strategies while varying the frame/chunk configuration. Baseline (64 frames, 1 chunk, no autoregressive): 41.8%. Ours-Autoreg (64 frames, 8 chunks): 45.1% — a +3.3 point improvement from chunking and autoregressive modeling with the same total frames. Ours + Bidirectional (64 frames, 8 chunks): 45.1% — identical, suggesting the benefit comes from chunk-based processing and joint feature learning within chunks rather than from the causal prediction objective per se. Ours-Autoreg (128 frames, 8 chunks): 45.8% — further improvement from more frames. The key takeaway: partitioning the video into chunks and processing them sequentially (whether autoregressive or bidirectional) is substantially better than processing the entire video as one flat input, even with the same total frames. This validates the core architectural premise that temporal modeling in chunks is beneficial independent of the causality constraint.

Combiner dimension (Table 5d): Sweeping the number of output features per chunk m: 8→42.53%, 16→43.36%, 32→44.20%, 64→44.22%. Performance saturates between 32 and 64 features — the jump from 32 to 64 yields only +0.02 points, while the jump from 16 to 32 yields +0.84 points. This empirically justifies the choice of m = 32 as the sweet spot between compactness and expressiveness. The fact that 32 features (out of potentially hundreds of input features per chunk) can achieve near-maximal performance suggests that much of the per-chunk information is redundant or that the Combiner is effectively selecting the most informative 32 features. The near-zero gain from doubling to 64 indicates that the autoregressive model and text model don't benefit from additional per-chunk features beyond this point — 32 is sufficient to encode the semantic content of a short video snippet.

Loss weights (Table 6b): On the small model with equal weights (1.0 for causal, video reconstruction, and text losses): 45.0%. Reducing text loss weight to 0.1: 44.6% (-0.4). Increasing text loss weight to 10.0: 45.4% (+0.4). The effect is modest (+0.8 points between extremes) but consistent: higher text loss weight during finetuning helps because the evaluation metric is text generation accuracy. This informed the decision to use a 10× text loss weight during finetuning of the full model.

Autoregressive vs. non-autoregressive with equalized total dimensions (Table 6a): This ablation controls for total feature dimensionality to ensure the autoregressive benefit isn't simply coming from having more features. Baseline (32 frames, 1 chunk, 256 dimensions total): 40.4%. Baseline (128 frames, 1 chunk, 256 dimensions total): 44.8% — more frames help. Autoregressive (128 frames, 16 chunks, 16 dimensions per chunk, 256 total): 45.5% — the autoregressive version with the same total dimensionality outperforms the flat baseline packing all frames into one chunk. This is the cleanest evidence that chunking and autoregressive processing provide a representation benefit beyond simply using more features or more frames — the temporal structure itself is informative.

---

Critical Assessment

Claim: Decoupling autoregressive modeling by temporal alignment enables small models to outperform much larger ones.

The MSRVTT-QA comparison (Table 1) — 3B Mirasol3B at 50.42% vs. 80B Flamingo at 47.4% — is the paper's primary evidence for this claim. This is a real and impressive result, but it supports a narrower claim than the paper implies. What the experiment actually demonstrates is that Mirasol3B with its specific architecture, pretraining data (VTP, AudioSet, MaMMUT initialization), and finetuning procedure outperforms the published Flamingo result on one specific dataset. This could be due to the architectural decoupling, or it could be due to any number of confounding factors: different pretraining data (Flamingo was trained on a much larger and more diverse multimodal corpus including interleaved image-text data, which may not be optimal for MSRVTT-QA specifically), different image encoders (ViT-Huge with 3D tubes vs. Flamingo's NFNet), different image resolutions, different finetuning protocols, or simply that Flamingo wasn't optimized for MSRVTT-QA as a target benchmark.

A controlled experiment isolating the architectural contribution would train Mirasol3B and a Flamingo-style baseline (Perceiver compression of video into the text model without the separate media autoregressive component) on the same data with the same compute budget and compare. This experiment is not reported. The Perceiver Combiner ablation (Table 5b, 43.1% vs. Ours-Transformer 44.2%) provides weak evidence in this direction — it compares compression methods within the Mirasol3B architecture but doesn't compare against a true Flamingo-style architecture where all temporal modeling happens in the text model's cross-attention. The 1.1 percentage point gap is small and on the small model, making it suggestive but not conclusive.

The claim would be more robust if the paper showed: (1) a Flamingo-style baseline trained on the same VTP data with matched compute, (2) results on multiple datasets (not just MSRVTT-QA) showing consistent advantage, and (3) scaling behavior showing that the gap widens with video length (since the architectural decoupling is supposed to help most with long videos). The long-video results on ActivityNet-QA and NExT-QA (Tables 2-3) show strong performance, but Flamingo is not evaluated on these benchmarks — the comparisons are against other models with different architectures, data, and training procedures.

Verdict: The claim that decoupling enables parameter efficiency is supported directionally but the magnitude is confounded with data and training differences. The core architectural insight is sound, but the 3B-vs-80B headline overstates the evidence for the architecture specifically.

Claim: The Combiner enables scaling to 512 frames without parameter increase while preserving temporal information.

This claim is directly supported by Tables 2 and 3, which show consistent accuracy improvements from 128 to 512 frames on both ActivityNet-QA and NExT-QA, using the same model architecture without increasing the parameter count. The Combiner's fixed output dimension (m = 32 across all configurations) is the mechanism — regardless of how many frames are in each chunk (8 or 32), the autoregressive model always receives 32 features per chunk. On NExT-QA, the gain from 128→512 frames is +3.8 to +5.0 points (Table 3), which is substantial for a benchmark where the next-best method achieves 63.1%.

However, the paper does not report the computational cost of processing 512 frames vs. 128 frames. While the parameter count doesn't increase, the ViT encoder processes 4× more pixels in total (512 frames vs. 128 at the same resolution), the 3D tubes operate over larger temporal windows, and the Combiner (especially the transformer variant) processes more features per chunk (32 frames per chunk vs. 8). The paper states that the TTM Combiner uses 30% less memory and 18% less runtime than the Transformer Combiner, but doesn't give absolute numbers. For a practitioner deciding whether to scale from 128 to 512 frames, it matters whether the computational cost increases by 2×, 4×, or 10× — and this information is absent. The "without increase in model parameters" framing is technically correct but potentially misleading, since parameters are not the only (or even primary) cost of model scaling.

A more complete evaluation would include: (1) FLOPs or runtime measurements for 128 vs. 512 frames, (2) a comparison showing that the accuracy gain from 512 frames could alternatively be achieved by spending the same additional compute on a larger model with 128 frames (an inference-compute vs. parameter-count tradeoff analogous to Section 7 of the MATH paper), and (3) results showing when frame scaling saturates (does 1024 frames help further?).

Verdict: The claim is supported for accuracy improvement but incomplete for the cost side. The "without parameter increase" framing is accurate but understates the practical compute cost.

Claim: Joint audiovisual learning via the Combiner improves over modality-specific processing.

The audio-video results (Table 4) provide clear evidence for this. On Kinetics-Sound, audio+video (90.1%) substantially outperforms video-only (81.3%, though this is on the small model). On Epic-Sound, audio+video (78.2–79.4%) outperforms both audio-only (62.4%) and video-only (72.4%). On VGG-Sound, the model achieves 69.8% compared to the next-best prior work at 68.2% (ONE-PEACE). The margins are large on some datasets (Kinetics-Sound, Epic-Sound) and narrow on others (VGG-Sound), but the pattern of audiovisual combination helping is consistent.

What's not shown is how much of this benefit comes from the Combiner's joint fusion specifically, versus simply having access to both modalities through any fusion mechanism. The ablation in Table 5a shows that the Combiner adds +0.6 points on MSRVTT-QA (video-only with text), which is modest. No ablation compares the Combiner against a simple concatenation of separately compressed audio and video features (e.g., applying separate Perceivers to each modality and concatenating the outputs). Such an experiment would isolate the Combiner's joint fusion benefit from the benefit of simply having more input information. Without it, we cannot be certain that the Combiner's cross-modal attention during compression is doing meaningful work beyond what a simpler fusion approach would achieve.

Verdict: The claim that audiovisual learning helps is strongly supported (large margins in Table 4). The claim that the Combiner's joint fusion specifically is responsible is plausible but not rigorously isolated.

Claim: Chunk-level autoregressive modeling captures temporal dependencies better than flat video processing.

The ablation in Table 5c provides the clearest evidence: with the same 64 total frames, chunking into 8 autoregressive segments (45.1%) substantially outperforms processing all frames as one input (41.8%). The equalized-dimension ablation (Table 6a) strengthens this: with the same total 256 Combiner dimensions, the autoregressive version with 16 chunks × 16 dims (45.5%) outperforms the flat version with 1 chunk × 256 dims (44.8%, though with more frames: 128 vs. 32). These are well-controlled within-architecture comparisons that isolate the autoregressive chunking effect.

The finding that bidirectional processing gives the same performance (45.1% in Table 5c) is interesting and slightly undercuts the causal modeling rationale — it suggests the benefit is from chunk-based representation learning (joint features per segment, sequential processing) rather than from the forward-prediction objective per se. For video understanding tasks where the full video is available at inference time (as opposed to streaming applications), this means the causality constraint is not providing accuracy benefits, only architectural compatibility with streaming.

Verdict: Strongly supported by well-controlled within-architecture ablations. The mechanism (chunk-based processing vs. causal prediction) is partially disentangled by the bidirectional comparison.

Genuine Weaknesses

Single model family, single initialization source. All experiments use PaLM-derived components (or more precisely, MaMMUT-initialized weights from contrastive pretraining on ALIGN). We cannot know whether the results depend on this specific initialization or whether the architecture would work with other base models.

No error bars, no multiple runs. The paper reports single-number accuracies throughout with no mention of variance, standard deviations, or confidence intervals across random seeds. For a 500-question test set (MSRVTT-QA test split is not explicitly stated, but the dataset has ~10K clips and 243K QA pairs), the standard error on a 50% accuracy measurement would be approximately 2–3 percentage points — meaning some of the narrow margins (e.g., Mirasol3B 50.42% vs. M-PLUG2 48.0%) could overlap within one standard error. Without variance estimates, we cannot assess the statistical reliability of the rankings.

No scaling study across model sizes. The paper reports one model size (3B, with a 1.15B ablation variant) and compares against models of other sizes from different papers. There is no experiment showing how Mirasol3B performance changes as the model scales from, say, 1B to 3B to 10B. Such a scaling curve would distinguish whether the 3B configuration is near-optimal or whether further scaling would continue to improve, and would provide evidence that the architectural advantages persist at larger scales.

The pretraining data is a fraction of what competitors use. The model is pretrained on only 3M samples (12% of VTP), while Flamingo and other large models are typically trained on billions of image-text pairs. This makes the strong performance more impressive (achieving more with less data), but also raises questions: is the architecture genuinely more data-efficient, or is the MaMMUT initialization providing most of the visual-semantic knowledge, with VTP pretraining just adapting it to video? The fact that "pretraining" adds only +2.7 points in the ablation (Table 5a: 45.2% with pretraining vs. 42.1% without, given the Combiner is already present from the earlier row) suggests the initialization may be doing heavy lifting.

Missing comparison: Mirasol3B without the audio+video autoregressive model. The most important missing experiment is a direct comparison between the full architecture and a version where the Combiner outputs \(x_t\) are concatenated and fed directly to the text model's cross-attention (or input sequence), without the intermediate autoregressive latent causal model. This would isolate the contribution of the dedicated media autoregressive component, which is the paper's central architectural claim. The ablation in Table 5a adds the autoregressive model on top of a baseline that doesn't have the Combiner, showing +1.7 points — but what's the benefit of the autoregressive model given the Combiner? The +3.2 combined benefit tells us they're complementary, but the decomposed contributions (AR alone +1.7, Combiner alone +0.6, combined +3.2) suggest the AR component is the larger contributor. A direct "Combiner → text model" vs. "Combiner → AR model → text model" comparison would make this unambiguous.

Audio-video evaluation as open-ended generation is not fully justified. The paper converts audio-video classification datasets to open-ended generation by providing the prompt "Classify the video audio clip." and requiring exact string match. This is acknowledged as "more challenging" but the paper does not report how often correct answers are rejected due to string mismatch (e.g., "playing drum" vs. "playing drums"). For a fair comparison against classification baselines, the model's actual classification accuracy (using a mapping from generated text to class labels) should be reported alongside the exact-match accuracy, or an analysis of failure modes should be provided.

The Combiner dimension scaling experiment uses the small model. Table 5d shows saturation at m=32 with 44.20% and m=64 with 44.22%. This is on the small 1.15B model. It's possible that the full 3B model, with its larger autoregressive latent model and text model, could benefit from more than 32 features per chunk — the saturation point might shift with model scale. The paper doesn't verify that m=32 remains optimal for the full model.

6. Limitations and Trade-offs

Assumption: Difficulty-Aware Allocation Works, But Difficulty Estimation Itself Is the Hardest Part

The compute-optimal test-time scaling framework rests on the ability to estimate question difficulty before allocating the inference budget. The paper uses two methods: oracle difficulty (requiring ground-truth answers to compute pass@1 from 2048 samples) and predicted difficulty (averaging the PRM's final-answer scores across the same 2048 samples to approximate pass@1 without ground truth). Both approaches require generating and scoring 2048 complete solutions per question, which the authors explicitly acknowledge in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The authors further frame this as "an exploration-exploitation tradeoff — compute spent assessing difficulty versus compute spent solving the problem — flagging it as a key avenue for future work."

The consequence: The headline 4× efficiency gain (e.g., compute-optimal at 16 generations matching best-of-N at 64, Figure 4) is computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be: difficulty estimation (generating and scoring thousands of samples per question) + strategy execution. The former could easily dominate the latter. For a single question, spending 2048 generations just to estimate difficulty — only to then allocate 16–256 generations for actually solving it — makes the overall efficiency worse than simply running best-of-256 on every question and skipping difficulty estimation entirely. The framework becomes practical only if difficulty can be estimated cheaply enough that the amortized cost across many questions is negligible, or if questions are reused many times (so the estimation cost is paid once and amortized over repeated queries). Neither condition is demonstrated in the paper. The computed-optimal curves in Figures 4 and 8 should therefore be understood as upper bounds on achievable efficiency — the realized gains in deployment would be lower, possibly negative, depending on how difficulty is estimated.

Evidence in the paper: The difficulty estimation procedure is described in Section 3.2 (2048 samples, oracle vs. predicted bins). The cost exclusion is explicitly disclosed. No experiment measures total cost (estimation + execution), no ablation studies cheaper estimation methods, and no sensitivity analysis shows how performance degrades with fewer than 2048 estimation samples. This limitation is purely unaddressed — the paper identifies it but provides no mitigation.

Mitigation status: The authors explicitly call for future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), and suggest that dynamic difficulty estimation (starting with a few samples and adjusting the budget online) could subsume the estimation cost into the solution process. Neither approach is developed or evaluated in this paper. Until this gap is closed, the framework is an analytical contribution with practical deployment barriers, not a turnkey method.

---

Hard Problems Remain Effectively Unsolved: Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

Across all methods — PRM search, iterative revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show essentially no improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% — the pretraining-scaled 14× larger model outperforms test-time compute for these problems across all values of the inference-to-pretraining ratio R. The paper is transparent about this, stating in the Section 7 takeaway:

"on the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

The consequence: Test-time compute amplifies existing capability but does not create it. If the base model's pass@1 rate on a problem class is near zero — meaning it essentially never generates a correct solution even with 2048 independent attempts — then no amount of search, revision, or compute-optimal allocation will help. There are no correct solutions in the proposal distribution to find or refine. This establishes a hard boundary condition: for genuinely novel or out-of-distribution reasoning tasks that exceed the base model's training distribution, pretraining remains the only viable path. Compute-optimal test-time scaling is effective precisely where the model already has some non-trivial chance of success — it improves the probability of finding correct answers, but cannot conjure them from a model that fundamentally lacks the required knowledge or reasoning capability. This matters for practitioners because it means that difficult benchmark performance cannot be improved by throwing inference compute at a model that wasn't trained for the task — the model must first be capable enough that correct answers exist in its output distribution at some non-negligible rate.

Evidence in the paper: Figure 3 (right, bin 5 curves), Figure 7 (right, bin 5 bar), Figure 9 (bin 5 lines all near zero and below the pretraining baseline), and the explicit acknowledgment in Section 5.3. The evidence is consistent and unambiguous across search, revisions, and FLOPs-matched comparisons. The authors do not attempt to mitigate this — they characterize it as a fundamental limitation of the approach.

Mitigation status: Not mitigated and likely not mitigable within the test-time compute paradigm. The paper's contribution is precisely in characterizing where test-time compute helps (bins 1–3, easy-to-medium problems) and where it doesn't (bins 4–5). This is valuable as a boundary condition, but it means the approach offers no path forward for the hardest problems. For practitioners, this implies a hybrid deployment strategy: use test-time compute for routine questions within the model's capability range, but route hard questions to larger models or human experts. The difficulty estimator described in Section 3.2 could serve this routing function, though the estimation cost problem (Limitation 1) applies here as well.

---

Single Benchmark, Single Model Family: Generality of the Difficulty-Dependent Findings Is Unverified

All experiments use the MATH benchmark (500 test questions, high-school competition-level math) with PaLM 2-S* as the base model. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is unverified by any cross-model or cross-domain replication. Several aspects of the central findings could be model-specific or domain-specific:

• The PRM's quality and over-optimization behavior (Figure 3, where beam search degrades easy-problem performance) depend on the specific distribution of PaLM 2-S* outputs. A model family with different calibration properties or different error patterns — for example, one that produces more diverse incorrect answers or more conservative probability estimates — might exhibit different difficulty-dependent scaling curves. The over-optimization threshold (the point at which search starts exploiting verifier weaknesses) is a function of verifier quality, which in turn depends on the base model's output distribution.

• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families and sizes. The training procedure (Section 6.1, edit-distance-based pairing of incorrect-to-correct trajectories) was developed for PaLM 2-S* specifically. A model with different instruction-following behavior or different in-context learning dynamics might learn a fundamentally different revision policy from the same data.

• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning, multi-step deduction, and algebraic manipulation. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems due to verifier over-optimization, revisions helping easy problems via local refinement, the optimal sequential-to-parallel ratio shifting with difficulty — generalize to other reasoning domains (code generation, logical deduction, scientific QA, planning) or to tasks requiring factual recall rather than inference. The finding that over-optimization is the primary bottleneck (Section 5.3) may be specific to domains where the PRM can be fooled by superficially plausible but incorrect reasoning chains — a property that might not hold in domains with more objective intermediate verifiability (e.g., code where unit tests provide ground-truth step-level feedback).

Evidence in the paper: The paper uses exactly one benchmark (MATH, Section 4) and one model family (PaLM 2-S*). There are no experiments on other reasoning benchmarks, other model families, or other domains. The authors do not claim generality beyond this scope — they acknowledge the single-model focus — but the paper's framing as establishing "the first systematic scaling analysis" of test-time compute strategies implies, without stating, that the findings are broadly applicable. The evidence for or against this implication is absent.

Mitigation status: Not addressed. The paper does not attempt cross-model or cross-domain replication. The findings should be treated as specific to PaLM 2-S* on MATH until replicated. The difficulty-dependent framework (Section 3.1) — that optimal strategies vary with problem difficulty — is a conceptual contribution that likely generalizes, but the specific thresholds, strategy choices, and quantitative gains (e.g., beam search being optimal for bins 3–4, sequential revisions for bins 1–2) may not transfer.

---

The 14× Larger Model Baseline Is Weakened by Non-Compute-Optimal Training and Greedy Decoding

The FLOPs-matched comparison in Section 7 asks whether test-time compute with a smaller model can substitute for pretraining a larger model. The baseline is a model with approximately 14× more parameters than PaLM 2-S*, using greedy decoding with no additional test-time compute. The paper makes a specific design choice about how this larger model is trained:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This means the larger model is scaled only in parameters, with training data held fixed — following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm (Hoffmann et al., 2022) where both data and parameters are scaled equally. A Chinchilla-optimal model trained with 14× more total FLOPs (distributed as 7× more parameters and 2× more data, or some other optimal ratio) would likely outperform a parameter-only-scaled model on the same total compute budget.

The consequence: The pretraining baseline is weaker than it could be, which may inflate the reported advantages of test-time compute. The headline finding — that test-time compute enables a smaller model to outperform a 14× larger one on easy-to-medium problems (Section 7, Figures 1 and 9) — is comparing against a suboptimally trained larger model. If the larger model were trained with compute-optimal scaling (both more parameters and more data, per Hoffmann et al., 2022), its performance on MATH would likely be higher, and the crossover point where test-time compute loses its advantage would shift toward easier problems or lower inference-to-pretraining ratios R. The paper's reported boundary conditions (test-time compute wins on bins 1–3 at low R, pretraining wins on bins 4–5 and high R) should therefore be interpreted as generous to test-time compute — the true boundary likely favors pretraining more than the paper reports.

Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search, no revisions. Giving the larger model even a modest test-time compute budget (say, best-of-8 or a few sequential revisions) would create a much stronger baseline. The paper's question is whether test-time compute can substitute for pretraining, but a fairer comparison would ask: given a total FLOPs budget, what is the optimal split between pretraining compute and inference compute? This would require giving the larger model some inference compute as well, which the paper's FLOPs-matched analysis does not do.

Evidence in the paper: The training paradigm choice is explicitly disclosed in Section 7. The 14× larger model's greedy-only decoding is stated in Section 7. No experiment compares against a Chinchilla-optimally trained larger model, and no experiment gives the larger model any test-time compute budget. The paper is transparent about these choices, but the headline comparisons do not account for the resulting bias.

Mitigation status: The authors acknowledge the limitation and defer compute-optimal pretraining comparisons to future work (Section 7). A stronger experiment — training a larger model with matched total FLOPs but compute-optimal parameter/data scaling, then comparing against the smaller model with test-time compute — would provide a more definitive answer about the substitutability of pretraining and inference compute. Until such an experiment exists, the FLOPs-matched conclusions should be treated as preliminary and likely favorable to test-time compute.

---

The Revision Model Suffers from Systematic Correct-to-Incorrect Reversion, and the Mitigation Is Incomplete

The revision model is trained on trajectories where the in-context examples are all incorrect, followed by a correct target (Section 6.1). This training setup creates a predictable failure mode at inference time: when the revision chain produces a correct answer at some intermediate step, the model has never been trained on what to do when the current answer is already correct. It has only seen incorrect → correct transitions, so a correct answer in context provides no signal about whether to preserve it or revise it. The paper quantifies this problem:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach." (Section 6.1)

The consequence: Even when the revision model eventually produces a correct answer, there is a substantial probability (~38%) that a subsequent revision step will change it to an incorrect one. This means that taking the final answer in a revision chain is unreliable — the best answer in the chain may appear at some intermediate step and be lost later. The paper's mitigation is to use majority voting or verifier-based selection across all steps in the revision chain (Section 6.1): rather than always taking the last revision, the system evaluates all intermediate answers and selects the best one. This converts the problem from "the revision chain must end at the correct answer" to "the correct answer must appear somewhere in the chain and be recognized by the verifier or majority." This works (Figure 6 left shows improving pass@1 across steps, and Figure 6 right shows sequential revisions outperforming parallel), but it is fundamentally a post-hoc correction rather than a fix to the underlying model behavior. The model still wastes computation generating incorrect revisions of correct answers, and the verifier or majority voting must be reliable enough to identify the correct answer among a chain of candidates — on hard problems where the correct answer appears rarely and the verifier is imperfect, this selection problem becomes harder.

Evidence in the paper: The 38% reversion rate is stated in Section 6.1. The mitigation (within-chain selection) is described in the same section and evaluated implicitly in Figure 6. The ReST^{EM} experiment in Appendix K (Figure 16) provides further evidence that revision training is fragile: attempting to optimize the revision model with on-policy RL-style training caused performance to degrade substantially with sequential revisions, likely because on-policy data collection amplified spurious correlations in revision trajectories. This suggests the revision model's behavior is sensitive to training data distribution in ways that are not fully understood or controlled.

Mitigation status: Partially mitigated by within-chain selection (majority voting or verifier-based), which converts the reversion problem from a correctness failure to an efficiency problem (wasted compute generating incorrect revisions). The underlying issue — that the model was never trained to recognize when no revision is needed — remains unaddressed. A principled solution would be to include "no revision needed" examples in training data, teaching the model to output the same correct answer when the current answer is already right. This is not explored in the paper. For practitioners deploying revision models, the 38% reversion rate means that within-chain selection is essential — deploying sequential revisions without it would yield substantially worse performance than the paper reports. The ReST^{EM} negative result also suggests that iterative self-improvement of revision models (a natural extension proposed in Section 8) may be unstable without careful regularization or data filtering.

---

Sequential Revisions Impose a Latency Penalty That Is Not Accounted for in the Compute Budget

The paper measures and compares test-time compute in terms of "generations" — the total number of complete solutions sampled, regardless of whether they are generated in parallel or sequentially. This is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial: revision step i+1 depends on the output of revision step i, so the chain of L revisions takes L times the latency of generating a single solution, even though the total generation count is L. In contrast, parallel best-of-N with N independent samples can be executed simultaneously with sufficient hardware, taking roughly the same wall-clock time as generating a single solution.

The compute-optimal policy frequently favors sequential or hybrid strategies: on easy problems (bins 1–2), purely sequential revisions are optimal (Figure 7, right), and on medium problems (bins 3–4), balanced sequential-to-parallel ratios are optimal. A strategy that allocates 128 generations as 64 sequential × 2 parallel (a 32:1 sequential-to-parallel ratio) takes approximately 64× longer wall-clock time than a strategy that runs 128 parallel samples simultaneously, even though both use 128 total generations.

The consequence: For latency-sensitive applications — interactive assistants, real-time decision-making, any deployment where users are waiting for responses — the sequential-heavy strategies favored by the compute-optimal policy may be impractical regardless of their accuracy advantages. A user waiting 64 seconds for an answer (64 sequential generations at ~1 second each) might prefer a slightly less accurate answer delivered in 1 second from parallel best-of-64. The paper's compute-optimal framework optimizes only for accuracy given a generation budget; it does not incorporate a latency constraint or a latency-accuracy tradeoff. Practitioners deploying in latency-constrained environments would need to either (a) restrict the policy to parallel-only strategies (sacrificing the gains from revisions on easy problems) or (b) accept longer response times for certain question types.

Evidence in the paper: The sequential-to-parallel ratio sweep (Figure 7) shows the optimal operating point at various budgets. The text in Section 6 describes sequential and parallel strategies. The paper never discusses latency, wall-clock time, or throughput. The FLOPs-matched comparison (Section 7) accounts for total FLOPs but not for the serialization cost of sequential computation. No experiment reports response time for different strategies, and no analysis discusses the latency implications of the compute-optimal policy's strategy choices.

Mitigation status: Not addressed. The paper's focus is on total compute (FLOPs/generations) rather than latency. This is a reasonable scope for an initial analysis, but it means the compute-optimal policies reported are optimal only in throughput-limited regimes (batch processing, offline evaluation) where parallel hardware is abundant and per-query latency doesn't matter. For latency-sensitive deployments, the optimal policy would need to incorporate a latency constraint, which would shift the optimal sequential-to-parallel ratio toward more parallelism (since parallel samples can be batched) and potentially change which difficulty bins benefit from revisions at all. This is a fundamental tradeoff that the paper does not resolve: the accuracy gains from sequential revisions come at a latency cost that is invisible in the current evaluation framework.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a design principle rather than just a model — and that principle is what makes it landscape-shifting. The core claim is architectural, not empirical: multimodal models should structurally separate processing according to the temporal alignment characteristics of their input modalities, rather than funneling everything through a single processor that inevitably must compromise. Video and audio are time-aligned, high-frequency, and high-volume; text is sparse, global, and non-aligned. A single autoregressive pipeline cannot serve all these masters well — it must either starve the media modalities of capacity (as in Flamingo, where ~1% of parameters handle visual processing) or inflate sequence lengths to the point of infeasibility. The Mirasol3B architecture — separate autoregressive models for aligned and non-aligned modalities, bridged by cross-attention — is the concrete instantiation, but the principle is what carries forward.

This represents a genuine reframing, not an incremental improvement, but it is a reframing of architectural design philosophy rather than a new learning paradigm. The field's default approach for multimodal models has been to extend the language modeling paradigm to other modalities — tokenize images and videos, interleave them with text tokens, and let one big autoregressive transformer handle everything. This paper makes the case, with empirical evidence, that this symmetry is a mistake. The modalities have different temporal structure, different information densities, and different volumes; treating them identically means the architecture fights the data rather than reflecting it. The 3B-vs-80B Flamingo comparison on MSRVTT-QA (Table 1: 50.42% vs. 47.4%) is not the primary contribution — it's the evidence that the principle pays off. The primary contribution is the principle itself.

This reframing has several downstream effects on the research landscape:

It makes visual pathway capacity a first-class design consideration. Before this work, the visual encoder in a multimodal model was largely treated as a feature extractor — get the right architecture (ViT, ResNet), the right resolution, the right pretraining, and feed the features into the language model. Mirasol3B argues that this underinvests in what should be nearly half the model. The finding that allocating "a little over half" of the 3B parameters to media processing enables a small model to match or exceed much larger ones shifts the conversation from "what's the minimum visual encoding we can get away with?" to "how much capacity do the media modalities actually need?" This makes VLMs like Flamingo look underexplored in an important dimension — their visual pathways might be starved not because that's optimal, but because that's where the field happened to start.

It changes how we think about long video understanding. Prior work on long-form video has focused on architectural mechanisms to handle long sequences: hierarchical attention [46], temporal windows [13], memory-augmented transformers [59], contrastive learning across timescales [4]. These approaches treat length as a problem to be managed — how do we fit more frames into the same architecture? Mirasol3B reframes the question: rather than retrofitting long-sequence handling onto a flat video representation, structure the representation itself so that length is naturally absorbed. Partitioning into chunks, compressing each to a fixed number of features via the Combiner, and modeling their temporal evolution autoregressively means that the video's length affects only the number of autoregressive steps, not the per-step computation (especially with the TTM Combiner's constant-time property). This makes 512-frame processing (Tables 2-3) a natural consequence of the architecture rather than a special capability requiring dedicated long-sequence machinery. The implication for the field is that representation design and temporal modeling should be co-designed — the question isn't just "how do we process long videos?" but "at what granularity and with what compression should we represent video chunks so that temporal modeling is both effective and tractable?"

It reconciles the tension between "more modalities" and "more parameters" in multimodal models. Adding audio, depth, IMU data, or other time-aligned sensor streams to a standard VLM typically bloats either the input sequence (if tokenized) or the feature set (if encoded). The Combiner mechanism offers a template: time-aligned modalities can be fused and compressed jointly at the chunk level before entering the temporal model, meaning the cost of adding a new modality is in the encoder (which can often be shared or lightweight) rather than in the downstream autoregressive processing. The shared ViT backbone for video and audio (Section 3.1) demonstrates this: the same architecture processes both, and the Combiner handles the fusion, so adding audio increases parameters by only ~100M (out of 3B). This suggests a path toward models that incorporate many synchronized sensor streams — multi-view video, depth, thermal — without the parameter explosion that would result from feeding all of them into a unified transformer.

It increases the attractiveness of purpose-built architectures over uniform transformers. The success of the Transformer has led to a homogenization of architecture design — if attention works, use it everywhere, for everything. Mirasol3B is a counterexample: it uses transformers extensively (ViT encoder, Combiner, latent causal model, text model), but the overall architecture is heterogeneous, with different components specialized to different temporal granularities and fusion patterns. The Combiner fuses and compresses within chunks; the latent causal model captures cross-chunk dependencies; the text model handles language generation with cross-attention to media features. This is a heterogeneous, multi-scale design that would be difficult to express as a single flat transformer. The strong results at small scale (3B) suggest that architectural specialization may be an underexplored axis for improving multimodal model efficiency — as the field moves toward ever-larger uniform models, this paper provides evidence that thoughtful structural differentiation can achieve more with less.

It makes streaming video processing an architectural property rather than an engineering workaround. Because both the Combiner (with causal attention or TTM memory) and the latent causal model process inputs autoregressively in time, the architecture is naturally compatible with streaming — video chunks arrive sequentially, the Combiner updates incrementally, and the autoregressive model conditions on all available history. This isn't evaluated in the paper, but it's a latent capability that follows from the design. For applications like live video understanding, autonomous driving perception, or real-time multimodal assistants, this means the architecture doesn't need to be adapted for streaming — it was built for it. This contrasts with models that require the full video for bidirectional encoding or that process all frames through a joint attention mechanism before generating text.

Follow-Up Research This Work Enables

Comparing Mirasol3B against a Flamingo-style baseline trained on identical data with matched compute. The most important missing experiment is a controlled comparison where the same Pretraining data (VTP, 3M samples), the same video encoder (ViT-Huge with 3D tubes), the same text model architecture and initialization (MaMMUT), and the same compute budget are used to train both the full Mirasol3B architecture and a Flamingo-style variant where the Combiner's outputs are fed directly to the text model's cross-attention without the autoregressive latent causal model. This would isolate the contribution of the dedicated media autoregressive component — the paper's central architectural claim. The current comparisons against Flamingo (Table 1) confound architecture with pretraining data, model scale, and image encoder differences. A clean ablation would measure: same inputs, same total parameters (keeping the latent causal model's parameters but using them differently in the text pathway), same training steps, different temporal modeling strategy. The prediction from the paper's framework is that the autoregressive media model helps most on tasks requiring temporal reasoning (NExT-QA causal and temporal questions, Table 3) and least on tasks requiring only static content recognition. The bidirectional comparison in Table 5c (same performance as autoregressive) suggests the benefit may come from chunk-based processing rather than forward prediction, making this a genuine open question.

Scaling the Combiner output dimension as a function of video complexity rather than using a fixed m = 32. The Combiner dimension scaling experiment (Table 5d) shows saturation at m = 32 on the small model with MSRVTT-QA (videos averaging 14 seconds). This saturation point is almost certainly dataset-dependent and task-dependent. A video containing a single person performing a simple action can likely be compressed to fewer features than a video with multiple interacting agents, complex backgrounds, and rapid scene changes. A natural extension is to learn an adaptive Combiner that outputs a variable number of features based on content — for example, using a learned halting mechanism or an information-theoretic criterion (e.g., keep enough features to reconstruct the original features within some tolerance). This would make the compression ratio content-adaptive: simple scenes get aggressive compression, freeing capacity for longer videos or more parallel processing; complex scenes get more features, preserving the detail needed for fine-grained understanding. The paper's fixed m = 32 design choice, while empirically justified by the saturation experiment, leaves this adaptive dimension unexplored. A strong follow-up would measure: (a) the relationship between video complexity (e.g., number of objects, motion magnitude, scene change frequency) and the minimum m needed to preserve downstream task performance, (b) whether a single fixed m is optimal across all chunks in a video or whether some chunks (those containing key events) need more features, and (c) whether an adaptive Combiner trained with a sparsity or compression penalty can match fixed-m performance with lower average feature count.

Scaling to more time-aligned modalities and evaluating on multi-sensor benchmarks. The Combiner is presented as a general mechanism for fusing and compressing time-aligned modalities — audio and video in the current implementation, but the formulation (Section 3.2: \(u_t = (\hat{v}_t, \hat{a}_t)\) can be any set of time-aligned features) is generic. A natural extension is to add more synchronized sensor streams: depth maps (from RGB-D cameras), IMU data (accelerometer, gyroscope), thermal imagery, or LiDAR point clouds. Each would require its own encoder (or a shared backbone adapted to the modality), but the Combiner and autoregressive model would remain unchanged — the per-chunk concatenation simply grows from \(\hat{v}_t, \hat{a}_t\) to \(\hat{v}_t, \hat{a}_t, \hat{d}_t, \hat{i}_t, \hat{l}_t\). The key research question is whether the Combiner's compression (from potentially very large combined features down to m = 32) remains sufficient as modalities are added, or whether the compression ratio needs to scale. Datasets like Something-Something v2 (temporal action recognition with fine-grained motion), EPIC-Kitchens (egocentric video with audio and narration), or autonomous driving datasets (nuScenes, Waymo Open Dataset with multi-sensor streams) provide natural testbeds. A strong follow-up would train Mirasol3B on 3–5 synchronized modalities, measure the marginal benefit of each additional modality, evaluate whether the Combiner's cross-modal attention learns to fuse complementary signals (e.g., audio for events outside the field of view, depth for occluded objects), and characterize the scaling of Combiner dimension m with modality count.

Training a difficulty predictor to close the estimation cost gap for compute-optimal deployment. The paper explicitly flags difficulty estimation cost as the barrier to practical deployment of the compute-optimal framework (Section 3.2: generating 2048 samples per question is "largely for simplicity"). A concrete follow-up is to train a lightweight classifier that predicts the difficulty bin directly from the question text and/or the first few output tokens of a single generation, without requiring 2048 full solutions. The training data would come from the 500-question MATH test set with oracle difficulty labels already computed (Section 3.2). A small transformer or even a bag-of-words classifier could be trained to predict the five-bin difficulty class from the question text alone. The evaluation metric is: how closely does the predicted difficulty bin match the oracle bin, and — more importantly — what is the accuracy achieved by the compute-optimal policy when using the predicted bin vs. the oracle bin? If the predicted bin accuracy is comparable to Figure 4's "predicted" curve (which used PRM scores from 2048 samples, still expensive), the method is deployable. If a simple text-based classifier can achieve comparable binning accuracy, the 2048-sample estimation becomes unnecessary, and the 4× efficiency gain becomes practically realizable. A negative result — that text alone is insufficient for accurate difficulty estimation — would suggest that difficulty is inherently a property of the model-question interaction (not just the question) and that cheap estimation may require architectural support (e.g., a difficulty prediction head trained jointly with the main model).

Combining the Combiner-based media autoregressive model with text-conditioned search or iterative refinement for video QA. The paper studies the Combiner and autoregressive media modeling as a pure feedforward architecture — the text is generated once, conditioned on the media features. But the compute-optimal framework from the MATH paper (Sections 5–6) shows that iterative strategies — beam search against a verifier, sequential revisions — can substantially improve accuracy for a given compute budget. A natural synthesis is to apply these test-time compute strategies to video QA: use the Combiner-based autoregressive model to generate initial answers, score them with a learned verifier (trained as a PRM on the video QA task), and either search over answer candidates (beam search) or iteratively revise answers (sequential refinement). The key challenge is that the "generation" unit for video QA is more expensive than for text-only QA — each candidate answer requires processing the full video through the Combiner and autoregressive model, not just sampling text tokens. This means the tradeoffs between parallel sampling, beam search, and revisions may differ from the text-only case. A concrete experiment would: (a) train a verifier on Mirasol3B's outputs for MSRVTT-QA, (b) compare best-of-N vs. beam search vs. revision chains at matched generation budgets, (c) measure whether the difficulty-dependent patterns from the MATH paper (beam search helps on medium problems, revisions help on easy problems) replicate in the video domain. This would test whether the compute-optimal framework generalizes beyond text-only reasoning to multimodal tasks where the generation cost is dominated by media processing rather than text generation.

Evaluating the TTM Combiner on streaming video where the memory mechanism is actually needed. The paper presents the TTM Combiner as more efficient than the Transformer Combiner (30% less memory, 18% less runtime) and notes that the architecture is "naturally applicable to streaming videos" (Section 3.4), but all evaluations are on offline video QA where the full video is available at inference time. A concrete follow-up is to evaluate on a streaming video benchmark — such as Ego4D forecasting (predicting future actions given a partial video stream), online temporal action detection (detecting actions as soon as they occur without peeking into the future), or streaming video QA (answering questions about the video so far, with incremental answers as more video arrives). In this setting, the TTM Combiner's constant-time per-step computation and fixed-size memory become genuine requirements rather than nice-to-have optimizations — the Transformer Combiner's \(O(t^2)\) attention over all historical features would become infeasible for long streams. The key measurements would be: (a) accuracy on streaming tasks as a function of video length (does TTM maintain accuracy for very long streams where Transformer Combiner would run out of memory?), (b) the tradeoff between memory size and accuracy (how small can the TTM memory be before temporal dependencies are lost?), (c) whether the TTM's read and write operations learn interpretable memory management strategies (e.g., retaining features from key moments, overwriting features from routine segments). A negative result — that the TTM's fixed-size memory loses critical information for long streams — would motivate research into adaptive memory sizes or hierarchical memory architectures.

Practical Applications and Downstream Use Cases

Long-video understanding for content indexing and retrieval at scale. The architecture's ability to process 512 frames (Tables 2-3) without parameter increase makes it directly applicable to video indexing pipelines where long videos (lectures, meetings, sports broadcasts, surveillance footage) need to be searchable by natural language queries. A media company ingesting thousands of hours of video daily could deploy Mirasol3B to generate timestamped answers to questions like "find all moments where someone mentions the quarterly results" or "show me all goals scored in the second half." The Combiner's fixed per-chunk output dimension (32 features per chunk, 16 chunks) means the cross-attention context for the text model is constant regardless of video length — a 5-minute video and a 2-hour video both produce 16 sets of media features (if chunked into 16 segments), so the text generation cost doesn't grow with video duration. The practical benefit is that the model can be applied uniformly to videos of varying lengths without needing to adjust architecture or retrain — a single model handles everything from short clips to feature-length content. The ActivityNet-QA results (51.13% at 512 frames, Table 2) provide a baseline accuracy estimate for this use case, though deployment would require additional tuning for the specific query distribution and tolerance for inexact matches (the open-ended generation evaluation is strict; production systems might use fuzzy matching or embedding-based retrieval).

Audio-visual event detection in smart environments and autonomous systems. The audio-video results in Table 4 — 90.1% on Kinetics-Sound, 69.8% on VGG-Sound, 78.2% on Epic-Sound — demonstrate that the joint audiovisual modeling captures complementary signals that neither modality alone provides. This has immediate application in environments where audio and video provide redundant but non-identical information about events: smart homes (detecting falls, glass breaking, appliance alarms), autonomous vehicles (siren detection, engine sounds correlated with visual hazards), industrial monitoring (machine sounds indicating malfunction, visual confirmation of smoke or leaks). The key practical advantage is robustness: when one modality is occluded or noisy (poor lighting, loud background noise), the other can compensate, and the Combiner's joint fusion allows the model to learn these cross-modal correspondences during training. The Epic-Sound results are particularly relevant — this dataset features egocentric kitchen videos where sounds (chopping, sizzling, boiling) are tightly coupled with visual actions, matching the characteristics of many real-world monitoring scenarios. Deployment would require adapting the audio-video classification formulation (currently: input "Classify the video audio clip.", output class name) to a continuous monitoring setting where events need to be detected with temporal localization — likely by sliding the chunked autoregressive model over the incoming stream and triggering detections when the generated text matches target event descriptions.

Cost-efficient video QA for accessibility and education. The finding that a 3B model can match or exceed an 80B model on video understanding tasks (Table 1, MSRVTT-QA: 50.42% vs. 47.4%) has direct economic implications for deployment. Running an 80B model at scale for video QA — generating descriptions for visually impaired users, answering student questions about educational videos, providing real-time commentary on live events — is expensive in both hardware and energy. A 3B model that achieves comparable or better accuracy can be deployed on substantially cheaper hardware (potentially a single GPU rather than a multi-GPU cluster) or even on-device for some applications. The Mirasol3B architecture's parameter efficiency means it can serve more queries per dollar of infrastructure, making video understanding accessible to organizations and use cases that cannot afford large-model deployments. The 4× efficiency gain from compute-optimal strategies in the MATH paper (Figure 4) suggests an analogous opportunity for video QA: if difficulty estimation can be made cheap (see follow-up research above), compute-optimal allocation could further reduce inference cost by routing easy questions through lightweight processing and reserving heavy computation for hard questions. A university deploying video QA for students reviewing lecture recordings could use a single Mirasol3B instance to handle queries from thousands of students, with the TTM Combiner's memory efficiency keeping per-query latency manageable even under load.

When to Prefer This Method

The paper positions Mirasol3B against the dominant paradigm of unified autoregressive multimodal models (exemplified by Flamingo [2] and tokenization-based approaches like CM3 [1] or video-language models that interleave visual tokens with text). The architectural choice is clearly articulated: decouple autoregressive modeling by temporal alignment rather than processing all modalities through a single autoregressive pipeline. Based on the paper's evidence and the architectural analysis, the decision criteria are:

• Prefer decoupled autoregressive modeling (Mirasol3B-style) when the video/audio content is the primary information source and the text is a query or description applied globally to the media. The model allocates "a little over half" its parameters to media processing (Section 3.6), compared to ~1% in Flamingo [2]. This pays off when visual and auditory detail matter — fine-grained action recognition, temporal event ordering, audiovisual correspondence. All the paper's benchmarks fall into this category: video QA, long-video QA, audio-video classification. The strong performance at small scale (3B vs. 80B Flamingo, Table 1) suggests the parameter allocation is more efficient for these tasks.

• Prefer decoupled autoregressive modeling when video length varies substantially and a single model must handle both short clips and long videos. The chunking and Combiner mechanism allows scaling from 128 to 512 frames without parameter increase (Tables 2-3), keeping the downstream autoregressive sequence length constant (T = 16 chunks). This avoids the sequence length explosion that would occur if each frame's features were tokenized and fed to a unified transformer. For applications where video duration is unpredictable — user-uploaded content, surveillance footage, streaming media — this uniform processing cost is a practical advantage.

• Prefer the TTM Combiner variant when memory or latency constraints are tight, or when streaming video processing is required. The TTM Combiner uses 30% less memory and 18% less runtime than the Transformer Combiner (Section 3.2), with comparable or slightly better accuracy on several benchmarks (Table 3: 73.2% TTM vs. 72.0% Transformer on NExT-QA; Table 4c: 79.4% TTM vs. 78.2% Transformer on Epic-Sound). The constant-time per-step computation (vs. \(O(t^2)\) attention in the Transformer Combiner) makes it the only viable option for streaming or very long videos where the history grows unbounded. The memory mechanism also enables deployment on hardware with limited VRAM.

• Prefer a unified autoregressive approach (Flamingo-style or tokenization-based) when text generation is the primary task and video provides only supplementary context. The paper doesn't evaluate on tasks where text dominates (e.g., multimodal dialogue where the video is a minor reference, or document understanding where images supplement text). In such settings, allocating half the parameters to media processing may be wasteful — the text model's capacity is what determines generation quality, and the cross-attention to compressed media features may be sufficient. The Flamingo-style approach (lightweight visual compression into a large text model) is architecturally more natural for text-heavy tasks.

• Prefer a unified autoregressive approach when the training budget is massive and model scale is the primary driver of performance. The paper's strong results at 3B parameters are achieved with only 3M pretraining samples (12% of VTP), which is a fraction of what large models like Flamingo (trained on billions of image-text pairs) use. This suggests Mirasol3B is data-efficient but leaves open the question of whether the architectural advantages persist as both data and parameters scale. A unified autoregressive model at 80B+ parameters with comparable or larger training data might close or reverse the accuracy gap through sheer scale, even with suboptimal parameter allocation. The paper provides no evidence at larger scales, so the decision becomes: at small-to-medium scale (≤10B parameters) with limited data, decouple; at large scale (≥80B parameters) with abundant data, the tradeoff is unknown and either approach could dominate depending on the specific architecture and training recipe.