Multimodal2025~106 minarXiv:2503.20215

Qwen2.5-Omni Technical Report

Jin Xu, Zhifang Guo, Jinzheng He +11Qwen / Alibaba CloudMar 26, 2025cs.CL

🎯 Pitch

A single model that sees, hears, reads, and speaks in real time does not just match specialized systems—its speech-input math reasoning (GSM8K 85.4%) virtually equals its text-input performance, and it beats dedicated TTS engines on intelligibility without any post-processing.

1. Executive Summary

This report introduces Qwen2.5-Omni, an end-to-end multimodal model that perceives text, images, audio, and video while simultaneously generating text and natural speech responses in a streaming manner. The model is built on three named mechanisms: TMRoPE (Time-aligned Multimodal RoPE) — a position embedding that synchronizes audio and video timestamps by interleaving their representations and decomposing rotary embeddings into temporal, height, and width components — Thinker-Talker architecture — in which a Transformer decoder (Thinker) generates text and high-level representations while a dual-track autoregressive decoder (Talker) converts those representations into streaming speech tokens without requiring word-level alignment — and block-wise streaming processing — where both audio and visual encoders use chunked attention along the temporal dimension to enable real-time pre-filling. On multimodal understanding, Qwen2.5-Omni achieves state-of-the-art performance on OmniBench (56.13%, surpassing prior omni models by a large margin) and delivers speech instruction-following comparable to its text-input performance — for instance, reaching 85.4% on GSM8K with speech input versus 88.7% with text input. For speech generation, the model attains 1.42%, 2.33%, and 6.54% word error rate on the SEED test-zh, test-en, and test-hard sets respectively, outperforming specialized TTS systems like MaskGCT and CosyVoice 2, establishing that a single unified model can match or exceed modality-specialized counterparts across understanding and generation tasks only when architectural separation (Thinker-Talker) and temporal alignment (TMRoPE) prevent cross-modal interference.

2. Context and Motivation

The Core Problem: Building a Unified Model That Perceives and Generates Across All Modalities in Real Time

The fundamental challenge Qwen2.5-Omni addresses is deceptively simple to state but extraordinarily complex to engineer: how do you build a single model that can simultaneously see, hear, read, and speak — processing all these input modalities together while generating both text and natural speech outputs, all in a streaming, real-time fashion?

This matters because human communication is inherently multimodal and real-time. When we converse, we don't just exchange text — we process facial expressions, gestures, tone of voice, environmental sounds, and spoken words all at once, and we respond fluidly with both language and speech. The paper frames this directly in its opening paragraph:

"In daily life, humans are capable of simultaneously perceiving the visual and auditory information around them. After processing this information through the brain, they express feedback through writing, vocalization, or using tools (and physical actions), thereby engaging in information exchange with various organisms in the world and exhibiting intelligence."

Yet the AI systems we've built to date are profoundly fragmented along modality lines. Large Language Models (LLMs) have achieved remarkable fluency in text, but they're deaf and blind by default. Language-Visual-Language Models (LVLMs) can see images and videos, but they can't hear. Language-Audio-Language Models (LALMs) can process speech and sound, but they can't see. And virtually none of these models can speak — they produce only text, requiring a separate text-to-speech pipeline bolted on afterward. The paper explicitly names this fragmentation as the gap it seeks to close:

"However, efficiently unifying all these different understanding modalities in an end-to-end fashion, utilizing as much data as possible, and providing responses in both text and speech streams akin to human communication still presents a significant challenge."

The practical stakes here are substantial. A truly unified omni-model would enable voice assistants that can watch a video with you and discuss its contents, educational tools that can see a student's diagram and hear their explanation while providing spoken feedback, accessibility systems that can describe the visual world to blind users in natural speech, and interactive agents that engage in fluid, human-like conversation with full contextual awareness. Each of these applications currently requires stitching together multiple specialist models — a vision model, a speech recognition model, an LLM, and a text-to-speech system — introducing latency, error propagation, and loss of cross-modal context at each junction.

Why Existing Approaches Fall Short

The paper identifies several specific limitations in prior work that motivate its architectural innovations.

Modality-specialized models are siloed and cannot benefit from cross-modal synergy. The state of the art in early 2025 consists of separate families of models: Qwen2.5-VL handles vision-language tasks, Qwen2-Audio handles audio-language tasks, and Qwen2.5 handles pure text. While these individual models are strong — Qwen2.5-VL achieves 58.6% on MMMU and Qwen2-Audio achieves strong ASR performance — they cannot jointly reason about a video's visual content and its accompanying audio track. The paper's experiments on OmniBench (Table 8) reveal the consequence of this fragmentation: prior omni models like UnifiedIO2 (6.8B) achieve only 33.98% on this benchmark, and even Gemini-1.5-Pro reaches just 42.91%. The gap between these numbers and Qwen2.5-Omni's 56.13% demonstrates that simply having separate vision and audio capabilities is not enough — the model needs to be trained jointly on interleaved multimodal data to develop genuine cross-modal understanding.

Video understanding requires temporal synchronization of audio and visual streams, which existing position encoding schemes cannot handle. Most multimodal models treat video as a bag of frames — a sequence of images with simple temporal position IDs. But real video comes with audio, and the audio waveform is sampled at a completely different rate than the visual frames (typically 40ms per audio frame versus variable frame rates for video, often around 30fps or ~33ms per frame). The paper points out that existing position encoding approaches like standard RoPE or even M-RoPE (Multimodal Rotary Position Embedding, introduced in Qwen-VL) do not account for this mismatch. Without explicit temporal alignment, the model cannot learn that a particular sound in the audio track coincides with a specific visual event — the two modalities drift apart in the model's internal representation. This is not a minor edge case; it's central to tasks like "what is the person in the red shirt saying?" where the model must locate the correct speaker visually and extract their speech from the audio track.

Generating both text and speech from a single model introduces cross-modal interference that degrades both outputs. This is perhaps the most subtle but practically devastating challenge. If you simply extend an LLM's vocabulary to include speech tokens alongside text tokens and train it to generate both, the model faces a fundamental conflict: the hidden representations that are good for predicting the next text token (which cares about semantic content, syntax, and reasoning) are quite different from the representations needed to predict the next speech token (which cares about prosody, speaker identity, emotion, and phonetic detail). The paper's Thinker-Talker architecture is explicitly motivated by this interference problem. The authors draw an analogy to human biology:

"This design is inspired by the way humans utilize different organs to produce various signals, which are simultaneously coordinated through the same neural networks."

In humans, the brain (analogous to Thinker) processes sensory input and formulates thoughts, while the vocal apparatus (analogous to Talker) translates those thoughts into speech. They're connected but distinct — and critically, they don't interfere with each other. Prior unified models that attempted to generate speech tokens directly from the LLM decoder (the approach taken by early speech-language models) implicitly forced the same hidden states to serve both semantic and acoustic masters, limiting the quality of both.

Streaming audio generation creates an anticipatory challenge: speech must begin before the complete text response is known. This is a problem unique to real-time speech output. In a non-streaming TTS system, you can wait for the full text transcript to be generated, then synthesize the entire speech waveform at once with full context. But in a streaming voice assistant, you need to start speaking immediately — within a few hundred milliseconds of the user finishing their question. The paper frames this through the lens of initial packet latency, which it identifies as "a critical indicator of the system's streaming performance." The model must determine the appropriate prosody, emotion, pace, and even the first phonemes before it knows what the complete response will say. This requires the speech generation module to receive rich anticipatory signals from the language model — high-level representations that encode the intended semantic trajectory — rather than merely converting finalized text to speech.

Existing speech generation models are either non-streaming or sacrifice quality for streaming capability. The paper evaluates against specialist TTS systems like MaskGCT and CosyVoice 2, which are state-of-the-art but are designed as separate post-processing steps — they take complete text transcripts as input and produce speech. These systems achieve excellent quality (MaskGCT: 2.27% WER on SEED test-zh; CosyVoice 2: 1.45% WER) but cannot operate in the tight latency budget required for real-time voice interaction. Streaming alternatives exist but have historically traded off naturalness and robustness for speed. Qwen2.5-Omni's streaming Talker with its sliding-window DiT codec decoder aims to close this gap — maintaining competitive WER while operating in a streaming regime.

The Three Specific Technical Gaps the Paper Tackles

The paper formalizes its contribution around three concrete challenges, which are worth examining separately because each motivates a distinct architectural innovation:

Gap 1: No systematic method for joint training of text, images, video, and audio with proper temporal alignment. Prior multimodal training pipelines treated different modalities as parallel data streams with independent position encodings. The paper argues that for video-with-audio specifically, a "time-interleaving" organization is necessary — chunking the audio and visual streams into synchronized segments — and that existing position embeddings cannot express this structure. The proposed TMRoPE directly addresses this by decomposing the rotary embedding into temporal, height, and width dimensions, where the temporal dimension is shared and synchronized across modalities.

Gap 2: Cross-modal output interference when a single model generates both text and speech. The paper explicitly states the design rationale:

"it is essential to manage potential interference among outputs from different modalities, ensuring that the training processes for outputs such as text and voice tokens do not disrupt each other."

This is not merely a hypothesis — the paper provides implicit evidence through its architecture choice. If generating speech tokens directly from the LLM decoder worked well, there would be no need for a separate Talker. The very existence of the Thinker-Talker split is a claim that shared decoding is insufficient.

Gap 3: Architectural support for real-time streaming understanding and low-latency speech output. This encompasses two sub-problems. First, the multimodal encoders must support pre-filling — the ability to process incoming audio and video chunks as they arrive, rather than waiting for the complete input. This requires replacing full-attention over the entire sequence with block-wise attention that processes chunks independently along the temporal dimension. Second, the speech codec decoder (which converts discrete speech tokens to waveforms) must operate with a restricted receptive field so that the first audio samples can be output without waiting for the entire token sequence. The paper's sliding-window DiT with its 4-block receptive field (2 lookback + 1 lookahead) is the proposed solution for this second sub-problem.

The Connection to Prior Work: Building on Qwen2.5-VL and Qwen2-Audio

Qwen2.5-Omni does not start from scratch. Its components are initialized from existing strong models:

The Thinker LLM is initialized from Qwen2.5 parameters (Section 3)
The vision encoder is the same as Qwen2.5-VL, a 675M-parameter ViT
The audio encoder is initialized from Whisper-large-v3

This inheritance is important because it means the paper's contributions are not about training better unimodal encoders — those already exist. The innovations are entirely in the integration architecture: how to connect these components so that they can be jointly trained without interference, how to synchronize their temporal representations, and how to add a speech generation pathway that doesn't degrade the understanding capabilities.

The paper positions itself as unifying two previously separate model families — Qwen2.5-VL (vision-language) and Qwen2-Audio (audio-language) — into a single model that matches or exceeds both on their respective benchmarks while also enabling new cross-modal capabilities that neither possessed alone. The OmniBench results (Table 8) are the clearest evidence for this: the model achieves 55.25% on the speech subset, 60.00% on sound events, and 52.83% on music, dramatically outperforming specialist omni models. This is not just "a model that can do vision and audio" — it's a model where vision and audio understanding mutually reinforce each other through joint training.

The Streaming Imperative: Why Real-Time Matters

A recurring theme throughout the paper is streaming — the model is designed from the ground up to process and generate information incrementally. This isn't just an engineering optimization; it fundamentally shapes the architecture in four ways detailed in Section 2.4:

Block-wise audio encoding (2-second chunks with chunked attention) replaces full-attention over the entire audio input, enabling the model to begin processing before the user finishes speaking.
Block-wise vision encoding with flash attention and 2×2 token merging enables efficient processing of high-resolution video frames as they arrive.
The dual-track Talker generates speech tokens autoregressively as Thinker produces text, rather than waiting for the complete text transcript — this is what the paper means by "anticipating content's tone and attitude before the entire text is fully generated."
The sliding-window DiT codec decoder restricts its receptive field to 4 blocks, enabling the first chunk of audio waveform to be generated as soon as the first speech tokens are available, rather than waiting for the full token sequence.

Each of these design choices reflects a concrete latency budget. While the paper doesn't provide exact millisecond targets, the engineering implications are clear: a voice assistant that takes 2-3 seconds to respond feels unnatural, while one that begins speaking within 200-300ms feels conversational. Achieving this with a 7B-parameter model that's simultaneously processing video, audio, and text while generating both text and speech tokens is a substantial systems challenge that motivates much of the architectural complexity.

How Qwen2.5-Omni Positions Itself Relative to Other Omni-Models

The paper evaluates against a landscape of competing unified models in Tables 2-8, and its positioning is worth analyzing:

MiniCPM-o: A contemporary omni-model that appears competitive on several benchmarks (e.g., 71.9% on MathVista, 2372 on MME, 64.0% on MMStar). Qwen2.5-Omni slightly trails on some vision benchmarks but significantly outperforms on OmniBench (56.13% vs 40.5%).
Baichuan-Omni-1.5: Another recent omni-model. Qwen2.5-Omni outperforms it on OmniBench (56.13% vs 42.9%) and on VoiceBench average (74.12 vs 71.14).
GPT-4o-mini: OpenAI's closed-source lightweight multimodal model. Qwen2.5-Omni outperforms it on most benchmarks including MMMU, MathVista, MMBench, and Video-MME.
Gemini-1.5-Pro: Google's larger multimodal model. Qwen2.5-Omni still outperforms it on OmniBench (56.13% vs 42.91%) and MMAU audio reasoning (65.60% vs 54.90%), though the model size comparison isn't direct.

The paper's implicit claim is that Qwen2.5-Omni represents a new state of the art for open-source unified multimodal models at the ~7B parameter scale, and that its architectural innovations — particularly TMRoPE for cross-modal temporal alignment and Thinker-Talker for interference-free speech generation — are what enable it to significantly outperform prior omni-models on tasks requiring genuine cross-modal integration (OmniBench) while maintaining competitive performance on single-modality benchmarks.

What the Paper Leaves Unaddressed

Several important problems are explicitly deferred to future work, which provides context for understanding the paper's scope:

Video OCR and audio-video collaborative understanding are identified as "critical issues that have often been overlooked by researchers in previous academic studies" (Section 6). The paper acknowledges these as weaknesses without claiming to have solved them.
Output modalities beyond text and speech: The conclusion states an aspiration to develop "expanded output capabilities across various modalities like images, videos, and music." The current model only generates text and speech, not visual outputs.
Faster inference: The conclusion names "a more robust and faster model" as a future goal, implicitly acknowledging that the current 7B-parameter model with its streaming architecture may still have latency limitations for some deployment scenarios.

These admissions help frame what Qwen2.5-Omni is and isn't: it's a model that unifies multimodal understanding with dual text-speech generation, achieving state-of-the-art cross-modal integration, but it doesn't yet handle visual generation, doesn't fully solve all video understanding challenges, and has room for optimization in inference speed.

3. Technical Approach

3.1 Reader Orientation

Qwen2.5-Omni is a single end-to-end neural network that takes in text, images, audio, and video simultaneously, and produces both text and natural speech as output — all in a streaming, real-time fashion. The system solves the problem of unified multimodal perception and generation by architecturally separating the "thinking" (understanding inputs, generating text) from the "talking" (converting thoughts into speech waveforms), while introducing a novel position encoding that synchronizes audio and video streams in time so the model can reason about who said what when in video content.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components organized into a two-stage pipeline:

Multimodal Encoders (Audio + Vision) — convert raw audio waveforms and video/image pixels into sequences of hidden representations. The audio encoder processes 40ms chunks of mel-spectrogram features; the vision encoder (a 675M-parameter ViT) processes image frames with dynamic resolution. Both use block-wise attention along the temporal dimension to enable streaming.
Thinker (Transformer Decoder LLM) — receives the encoded multimodal representations plus text tokens, processes them jointly through standard autoregressive Transformer layers, and produces two outputs: (a) high-level hidden representations at every timestep, and (b) text tokens sampled from a vocabulary distribution. Thinker is initialized from Qwen2.5-7B parameters.
Talker (Dual-Track Autoregressive Decoder) — receives Thinker's hidden representations as input (a continuous stream of high-dimensional vectors) along with the discrete text tokens Thinker has sampled, and autoregressively generates discrete speech tokens representing the audio to be spoken. Talker is a separate transformer decoder that shares all of Thinker's historical context.
TMRoPE Position Embedding — a positional encoding scheme applied at the input to Thinker that decomposes rotary position embeddings into three dimensions (temporal, height, width) and synchronizes audio and video by interleaving their representations in 2-second chunks ordered by actual timestamps.
Streaming Codec Decoder (Sliding-Window DiT + BigVGAN) — converts the discrete speech tokens output by Talker into audible waveforms. Uses a Flow-Matching DiT with restricted receptive field (4 blocks: 2 lookback + 1 lookahead) and a modified BigVGAN vocoder, both operating chunk-by-chunk to minimize initial packet latency.

Information flow: Raw multimodal input → encoders produce chunked representations → TMRoPE adds synchronized positional information → Thinker processes the full sequence and generates text tokens + hidden states → Talker reads hidden states and text tokens concurrently → Talker outputs speech tokens → streaming DiT converts speech tokens to mel-spectrogram → BigVGAN converts mel-spectrogram to waveform → audio output stream begins before full response is generated.

3.3 Roadmap for the Deep Dive

First, the TMRoPE position embedding mechanism, because it governs how all modalities are spatially and temporally organized before Thinker processes them, and it is the foundation for cross-modal synchronization.
Second, the multimodal perception pipeline (audio encoder, vision encoder, and the time-interleaving method), since these components determine what information reaches Thinker and in what format.
Third, the Thinker — how it processes the unified multimodal sequence, generates text, and produces the hidden representations that Talker depends on.
Fourth, the Talker architecture — how it converts Thinker's hidden states into speech tokens, why a dual-track design is necessary, and what the speech codec (qwen-tts-tokenizer) represents.
Fifth, the streaming infrastructure that spans all components — block-wise encoding, chunked pre-filling, and the sliding-window DiT for waveform generation — since streaming is not a single module but a set of constraints applied across the architecture.
Sixth, the three-stage pre-training procedure and the three-stage Talker post-training procedure, because the training curriculum determines how these components learn to work together without interference.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems architecture paper whose core idea is that a unified multimodal perceiving-and-generating model requires (1) temporally-aligned position encodings to synchronize video and audio, and (2) architectural separation between semantic processing (Thinker) and speech generation (Talker) to prevent cross-modal interference, with both components trained end-to-end.

TMRoPE: Time-Aligned Multimodal Rotary Position Embedding

What problem it solves. Standard position embeddings encode where a token is in a sequence, but they don't encode when in physical time different modalities' tokens correspond to each other. In a video with audio, the visual frame at 1.5 seconds and the audio segment at 1.5 seconds should have aligned temporal positions so the model can learn associations between them. Existing schemes — including M-RoPE (Multimodal RoPE), which decomposes position into temporal, height, and width dimensions — assign temporal IDs as simple incrementing counters. The problem is that audio is sampled at fixed intervals (one frame every 40ms) while video frame rates are dynamic and variable. If you simply increment temporal IDs independently for each modality, the temporal positions drift apart and the model cannot learn that a specific sound corresponds to a specific visual event at the same moment.

How TMRoPE extends M-RoPE. M-RoPE, introduced in Qwen-VL, decomposes the standard rotary position embedding into three components applied to different subsets of the attention head dimensions: temporal position, height position, and width position. For text, all three components use the same position IDs, making M-RoPE equivalent to standard 1D-RoPE. For images, temporal IDs are constant (all tokens from a single image share the same temporal position), while height and width IDs vary based on the token's 2-D position in the image grid. TMRoPE adds a critical modification: absolute temporal calibration. Instead of assigning arbitrary incrementing temporal IDs, TMRoPE scales temporal IDs so that each increment of 1 corresponds to exactly 40ms of physical time across all modalities. The paper states:

"we also use identical position IDs and introduce absolute temporal position encoding, with one temporal ID corresponding to 40ms"

For audio, this is straightforward — each audio frame naturally represents 40ms, so temporal IDs increment by 1 per frame. For video, the authors dynamically compute the temporal ID spacing based on the actual time gap between frames:

"Since the frame rate in video is not fixed, we dynamically adjust the temporal IDs between frames based on the actual time corresponding to each frame to ensure that one temporal ID corresponds to 40ms."

If a video is recorded at 30fps, two consecutive frames are 33.3ms apart, so their temporal IDs would differ by approximately 0.833 (33.3/40). If the frame rate drops to 15fps, the gap doubles. This means TMRoPE's temporal dimension operates on continuous-valued increments, not integer steps — the time-alignment is precise, not approximate.

How position IDs are assigned across modalities. When the model receives multiple modalities in a single input sequence, each modality's position numbering starts by incrementing from the maximum position ID of the preceding modality:

"In scenarios where the model's input encompasses multiple modalities, position numbering for each modality is initialized by incrementing the maximum position ID of the preceding modality by one."

This prevents position ID collisions across modalities while maintaining the temporal calibration within each modality. For instance, if the text prompt occupies positions 0–49, the first video frame might start at temporal position 50 (representing time t=0 of the video), and the first audio frame at temporal position 50 as well (also representing t=0), but their height/width/other dimensions distinguish them.

The interleaving structure. The final step is how these representations are physically arranged in the sequence fed to Thinker. The paper introduces a time-interleaving method for video with audio:

"we have a special design for video with audio called the time-interleaving method, which segments the representation in the video with audio into chunks every 2 seconds according to the actual time. We then arrange the visual representation at the front and the audio representation at the back within the 2 seconds, interleaving the representations of the video with audio."

Concretely, for a video with audio, the sequence looks like: [visual tokens from 0–2s] [audio tokens from 0–2s] [visual tokens from 2–4s] [audio tokens from 2–4s] [visual tokens from 4–6s] ... and so on. Within each 2-second chunk, visual tokens come first, audio tokens second. Across chunks, the pattern repeats. This interleaving means that when the transformer's attention mechanism processes the sequence, tokens at the beginning of a chunk attend to context from earlier chunks, but each chunk's visual and audio representations are adjacent in the sequence — making it easier for attention to learn cross-modal associations within the same time window.

Why 2-second chunks? The paper doesn't explicitly justify the 2-second window, but the design principle is clear: if chunks are too large, the model loses temporal locality (audio tokens at 0s might be far from visual tokens at 1.9s in sequence position, even though both are in the same chunk). If chunks are too small, the interleaving overhead dominates and the model loses longer-range context within each modality. Two seconds is a compromise — long enough to capture most utterance-level phenomena (a typical spoken sentence is 2–5 seconds), short enough that within-chunk positions remain temporally proximate.

What this enables. TMRoPE plus time-interleaving jointly solve the video-with-audio synchronization problem. During self-attention, a visual token at time t and an audio token at time t will have similar temporal position encodings, and they'll be physically adjacent in the sequence if they fall in the same 2-second chunk. This makes it structurally easy for the model to learn associations — the attention mechanism doesn't need to discover the correspondence from scratch; it's built into the positional geometry.

Multimodal Perception: Encoders and Input Processing

Text tokenization. Text is tokenized using Qwen's standard tokenizer, which applies byte-level byte-pair encoding (BBPE) with a vocabulary of 151,643 tokens. This is identical to the tokenizer used in Qwen2.5 and Qwen2.5-VL, ensuring compatibility with the pretrained LLM weights.

Audio encoding. Raw audio is first resampled to 16kHz (a standard speech processing sample rate). The waveform is then converted to a 128-channel mel-spectrogram using a window size of 25ms and a hop size (step between consecutive windows) of 10ms. This is a standard frontend for speech processing: the mel-spectrogram is a time-frequency representation that compresses the raw waveform (16,000 samples per second) into a more manageable 100 frames per second (one per 10ms hop), each with 128 frequency bins.

The mel-spectrogram frames are then processed by the audio encoder, which is initialized from Whisper-large-v3 and subsequently fine-tuned. The encoder is configured so that:

"each frame of audio representation roughly corresponds to a 40ms segment of the original audio signal"

This means the encoder performs temporal downsampling by a factor of approximately 4× (from 10ms per mel frame to 40ms per encoder output frame), likely through strided convolutions or pooling in the Whisper architecture. This 40ms granularity is the basis for TMRoPE's temporal calibration — one temporal position ID per 40ms audio segment.

Vision encoding (images and video without audio). The vision encoder is the same ViT-based model used in Qwen2.5-VL, with approximately 675 million parameters. It is trained on a mixture of image and video data. Key details:

Patch size: 14 pixels. Each 14×14 patch of the input image becomes one visual token.
Token merging: A simple MLP layer merges adjacent 2×2 tokens into a single token, reducing the sequence length by 4×. This is applied after the ViT processing to produce the final representation.
Dynamic resolution: Images of different resolutions can be packed into a sequence, likely using the dynamic resolution approach from Qwen2.5-VL where larger images are split into multiple sub-images.
Image as video: Each static image is treated as two identical frames for consistency with the video processing pipeline (which always expects frame sequences).
Dynamic frame rate for video: Video is sampled at a dynamic frame rate — the paper does not specify the exact sampling strategy, but the key point is that the frame rate adapts to the video content rather than being fixed. This creates the variable temporal gap between frames that TMRoPE is designed to handle.

Block-wise streaming attention for encoders. To support streaming (processing input incrementally before the full signal is received), both the audio and vision encoders are modified from full attention to block-wise attention along the temporal dimension:

"the audio encoder is changed from full attention over the entire audio to performing attention in blocks of 2 seconds each"

For the audio encoder, this means each 2-second block of audio frames attends only within that block, not to earlier or later blocks. This enables the encoder to process the first 2 seconds of audio while the user is still speaking the next 2 seconds. For the vision encoder, the paper notes that it "utilizes flash attention for efficient training and inference" — flash attention is a memory-efficient exact attention algorithm, not a block-wise approximation, but the vision encoder's processing of video frames is inherently frame-by-frame (each frame is processed independently by ViT), so streaming is achieved by processing frames as they arrive rather than batching them.

A note on "video without audio." The paper distinguishes between "Video (w/o Audio)" and video with audio. For video without audio, only the visual encoder processes the input. For video with audio, both encoders run, and the representations are interleaved via TMRoPE. This distinction matters because the training data includes both types, and the model must handle both cases gracefully.

Thinker: The Central Processing and Text Generation Module

What Thinker is. Thinker is a standard Transformer decoder (autoregressive language model) initialized from Qwen2.5-7B. It has the same architecture as Qwen2.5 — a series of transformer blocks with multi-head self-attention and feedforward layers, using RoPE position embeddings and a causal attention mask that prevents each token from attending to future tokens. It is called "Thinker" to distinguish its role from Talker, but architecturally it is a conventional LLM.

What Thinker receives. The input to Thinker is a sequence of hidden representations that combines all modalities. After the encoders produce their outputs and TMRoPE adds positional information, the representations are concatenated into a single sequence with the following order (interleaved for video+audio, sequential for simple multimodal inputs):

Optional system prompt and user text tokens (embedding lookup → hidden representation)
Image/video visual tokens (from vision encoder)
Audio tokens (from audio encoder)
Assistant text tokens (during autoregressive generation)

This unified sequence is processed by the transformer layers just like a text-only LLM processes a text sequence — the attention mechanism operates uniformly over all tokens regardless of modality.

What Thinker produces. Thinker has two output pathways:

Text tokens: At each generation step, Thinker produces a probability distribution over the 151,643-token vocabulary (the same as standard Qwen2.5). Text is generated autoregressively:

"Text is generated directly by Thinker. The logic of text generation is fundamentally the same as that employed by widely used LLMs, which generate text through autoregressive sampling based on the probability distribution over the vocabulary."

The generation can use standard techniques like repetition penalty and top-p sampling to improve diversity. This text stream is both (a) output to the user as the text response, and (b) fed to Talker as conditioning for speech generation.
Hidden representations: At every timestep (for every generated token), Thinker's final hidden states — the high-dimensional vectors at the output of the last transformer layer — are sent to Talker. These vectors encode the semantic content, intended meaning, and contextual information that Talker needs to produce appropriate speech. Critically, these hidden representations are available immediately as Thinker generates text, not only after the full response is complete. This is what enables streaming speech: Talker can begin generating speech tokens based on the hidden state of the first generated text token, before the second text token is even sampled.

Why text-to-text performance matters. Because Thinker is initialized from Qwen2.5-7B, a key evaluation is whether adding multimodal encoders and joint training degrades its pure text capabilities (catastrophic forgetting). Table 1 shows that Qwen2.5-Omni achieves 47.0% on MMLU-Pro, 71.5% on MATH, and 88.7% on GSM8K — numbers that sit between Qwen2-7B and Qwen2.5-7B, indicating that multimodal training causes some regression on text-only tasks (Qwen2.5-7B achieves 56.3% on MMLU-Pro vs. 47.0% for the omni model) but not catastrophic collapse.

Talker: The Speech Generation Module

Why a separate architecture is necessary. The paper's central design claim is that generating text and speech from the same decoder causes interference. The hidden states that are optimal for predicting "the next word should be 'weather'" are fundamentally different from the hidden states needed to predict "the next speech token should encode a rising intonation at 120Hz fundamental frequency." If a single decoder tries to do both — by simply expanding the vocabulary to include speech tokens alongside text tokens — the hidden representations become a compromise that serves neither master well.

The paper's solution is the Thinker-Talker split, explicitly analogized to human biology:

"This design is inspired by the way humans utilize different organs to produce various signals, which are simultaneously coordinated through the same neural networks."

Thinker handles semantic processing and text generation (the "brain"). Talker handles acoustic realization (the "mouth"). They are connected — Talker receives Thinker's hidden states — but they have separate parameters, separate output spaces, and separate training objectives.

Talker's architecture. Talker is a "dual-track autoregressive Transformer Decoder architecture, motivated by Mini-Omni." The term "dual-track" refers to the fact that Talker has two input streams:

Continuous hidden representations from Thinker — high-dimensional vectors (the same dimensionality as Thinker's hidden states, which for Qwen2.5-7B would be 3584 or 4096 depending on the exact model configuration). These arrive in a streaming manner, one per generated text token.
Discrete text tokens from Thinker — the actual sampled token IDs from the vocabulary. These are converted to embeddings (like standard LLM token embeddings) and fed alongside the hidden representations.

Both streams are essential for different reasons. The hidden representations carry rich semantic and prosodic information:

"The high-dimensional representations provided by Thinker implicitly convey this information [tone and attitude], enabling a more natural streaming generation process."

But hidden representations in an LLM are organized by semantic similarity — words with similar meanings have similar embeddings — not by phonetic similarity. Two phonetically distinct words like "read" (pronounced /rɛd/) and "peruse" (pronounced /pəˈruz/) might have nearly identical hidden representations because they mean roughly the same thing, but they require completely different speech tokens:

"Thinker's representations primarily express semantic similarity in the representational space rather than phonetic similarity. Consequently, even phonetically distinct words may have very similar high-level representations, necessitating the input of sampled discrete tokens to eliminate such uncertainty."

The discrete text tokens disambiguate the exact words that need to be spoken, while the hidden representations convey the intended prosody, emotion, pace, and speaking style.

Talker's output: speech tokens. Talker autoregressively generates discrete tokens representing speech. These tokens are produced by the qwen-tts-tokenizer, a custom speech codec:

"We designed an efficient speech codec named qwen-tts-tokenizer. qwen-tts-tokenizer efficiently represents key information of speech and can be decoded to speech streamingly through a causal audio decoder."

The paper provides no architectural details about qwen-tts-tokenizer beyond this description — it is a neural audio codec that compresses speech waveforms into a discrete token sequence. This is analogous to how text tokenizers compress text into token IDs, but operating on audio signals. Presumably, it is a VQ-VAE or similar vector-quantized autoencoder trained to reconstruct speech waveforms from discrete codes.

Critically, the speech generation does not require explicit word-level or phoneme-level alignment:

"The generation of speech does not require word-level and timestamp-level alignment with the text. This significantly simplifies the requirements for training data and the inference process."

This is a substantial practical advantage. Traditional TTS systems often require forced alignment — mapping each phoneme to a specific timestamp in the audio — which requires a separate alignment model and carefully annotated training data. Talker learns this mapping implicitly from the hidden representations, which already encode the temporal correspondence between text tokens and their acoustic realization through the shared attention mechanism.

How Talker operates during inference. When Thinker generates the first text token of the response, it simultaneously produces a hidden representation for that token. Talker receives this hidden representation and the token embedding, and begins autoregressively generating speech tokens. It continues generating speech tokens for as long as needed to "speak" that text token (a single text token like "weather" might require dozens of speech tokens depending on the codec's compression rate). Meanwhile, Thinker is generating the second text token. Talker receives the second token's hidden state and text embedding, and continues generating speech tokens for it. This pipelining is what enables streaming: speech output begins before the text response is complete.

Training Talker (three-stage post-training). Talker is trained after Thinker has been pre-trained and post-trained (instruction-tuned). The three stages are:

Stage 1 — In-Context Learning (ICL) for speech continuation: Talker is trained on a next-token prediction objective over speech tokens, using a large dataset of "dialogues that incorporate multimodal contexts and spoken responses." In addition to the speech token prediction loss, Talker also receives text supervision similar to Thinker's training — meaning it has a joint objective of predicting text and speech tokens. The key learning target is:

"Talker learns to establish a monotonic mapping from semantic representation to speech, while also acquiring the ability to express speech with diverse attributes that are contextually appropriate, such as prosody, emotion, and accent."

The training also includes "timbre disentanglement techniques to prevent the model from associating specific voices with infrequent textual patterns" — meaning that if a particular speaker's voice appears only in training examples about a specific topic, the model might learn to use that voice only for that topic. Timbre disentanglement is a regularization strategy that prevents this spurious correlation.
Stage 2 — Reinforcement Learning with DPO: The pretraining data contains label noise and pronunciation errors, leading to "model hallucinations" in speech (e.g., mispronounced words, inappropriate pauses, garbled audio). To improve stability, the paper applies Direct Preference Optimization (DPO):

The DPO objective used is:

$\mathcal{L}_{\text{DPO}}(P_\theta; P_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{P_\theta(y_w \mid x)}{P_{\text{ref}}(y_w \mid x)} - \beta \log \frac{P_\theta(y_l \mid x)}{P_{\text{ref}}(y_l \mid x)} \right) \right]$

where $P_\theta$ is the policy being optimized (the Talker model), $P_{\text{ref}}$ is a frozen reference policy (the Talker before DPO), $x$ is the input sequence (multimodal context + text to speak), $y_w$ is the "winning" (higher-quality) speech sequence, $y_l$ is the "losing" (lower-quality) speech sequence, $\beta$ is a temperature parameter controlling how far the policy can deviate from the reference, and $\sigma$ is the sigmoid function.

What it computes: a contrastive loss that increases the probability of generating the "winning" speech sequence relative to the "losing" sequence, while penalizing the policy if it diverges too far from the reference. The term inside the sigmoid is the difference between two log-ratios: how much the policy prefers $y_w$ over $y_l$ compared to how much the reference prefers $y_w$ over $y_l$ . When this difference is large and positive, the sigmoid approaches 1 and the loss is small. When it's negative (the policy prefers the losing response), the sigmoid approaches 0, log(0) → −∞, and the loss is large.

Why this form: DPO directly optimizes the policy from preference pairs without needing a separately trained reward model, which would be unstable and expensive. The log-ratio form compares the policy's relative preferences to the reference's, which prevents the policy from diverging too far and producing degenerate speech. The preference pairs are constructed based on "reward scores associated with word error rate (WER) and the punctuation pause error rate" — objectively measurable speech quality metrics that do not require human annotation.
Stage 3 — Multi-Speaker Instruction Fine-Tuning: The DPO-optimized Talker is fine-tuned on data from specific target speakers to "adopt specific voices and improve its naturalness." This is what enables the single-speaker evaluation results in Table 10, where Speaker A through Speaker D each have customized voice characteristics.

Streaming Infrastructure: Block-Wise Processing and Sliding-Window Generation

Why streaming matters for the architecture. The paper organizes its streaming discussion around initial packet latency — the delay between when the user finishes speaking and when the system begins outputting audio. This latency has four components:

"1) the delay caused by the processing of multimodal information inputs; 2) the latency from the moment the first text input is received until the first voice token is output; 3) the delay in converting the first segment of speech into audio; and 4) the inherent latency of the architecture itself, which is related to model size, computational FLOPs, and other factors."

The streaming design choices target components (1), (2), and (3). Component (4) — architecture size and FLOPs — is addressed by using a 7B-parameter model (relatively small by 2025 standards) and is flagged as future work in the conclusion.

Streaming Component 1: Block-wise audio encoding for pre-filling. "Chunked-prefills" is a mechanism used in modern LLM inference frameworks (like vLLM) where the prompt is processed in chunks rather than all at once, enabling the first output token to be generated before the full prompt has been encoded. To support this for multimodal inputs:

Audio encoder: Full attention over the entire audio sequence is replaced with attention in blocks of 2 seconds each. During inference, as soon as 2 seconds of audio have been received and encoded, the encoder's output for that block can be passed to Thinker, without waiting for the rest of the audio. This directly reduces component (1) of latency.
Vision encoder: Already processes frames independently via ViT, so video frames can be encoded as they arrive. Flash attention is used for efficient computation.

The paper doesn't specify whether blocks are completely independent (no cross-block attention) or whether there's some overlap or recurrent state passing between blocks, but the description "attention in blocks of 2 seconds each" suggests strict block independence — each block is a self-contained attention window.

Streaming Component 2: Talker's pipelined generation. Because Talker receives Thinker's hidden representations token-by-token as Thinker generates text, it can start generating speech tokens for the first text token as soon as Thinker produces it. The latency from "first text token generated" to "first speech token generated by Talker" is the forward pass through Talker for that single step — which is dominated by Talker's model size and is presumably small (Talker is an additional transformer decoder but its size is not specified in the paper). This addresses component (2) of latency.

Streaming Component 3: Sliding-window DiT for codec-to-waveform. Once Talker produces speech tokens, they must be converted to an audible waveform. This is the most latency-sensitive step because audio is generated at a high sample rate (typically 24kHz or higher for high-quality speech, meaning 24,000 samples per second), and generating the full waveform at once would require waiting for the complete speech token sequence.

The paper uses a two-stage pipeline:

Flow-Matching DiT (Diffusion Transformer): A diffusion model that converts discrete speech tokens (codes) into a mel-spectrogram. Flow matching is a generative modeling framework similar to diffusion models but using ordinary differential equations for the generative process. The key innovation for streaming is the sliding-window block attention mechanism:

"We group adjacent codes into blocks and use these for our attention mask. We limit the DiT's receptive field to 4 blocks, including a lookback of 2 blocks and a lookahead of 1 block."

For a given block of speech tokens being decoded, the DiT can attend to:
- The current block itself
- The 2 preceding blocks (lookback = 2)
- The 1 following block (lookahead = 1)
This restricted receptive field means that as soon as Talker generates speech tokens for the first block and the first block of the next chunk (for lookahead), the DiT can start generating the mel-spectrogram for the first block — without waiting for blocks further in the future. The mel-spectrogram is generated "in chunks using Flow Matching, ensuring that each code chunk has access to the necessary contextual blocks."
Modified BigVGAN vocoder: Converts the mel-spectrogram to the final waveform. BigVGAN is a universal neural vocoder (based on generative adversarial networks) that has a fixed receptive field. The paper uses it chunk-by-chunk:

"We also use this chunk-by-chunk method for BigVGAN's fixed receptive field to facilitate streaming waveform generation."

The entire pipeline — Talker generates speech tokens → DiT converts to mel-spectrogram chunks → BigVGAN converts mel chunks to waveform — operates on a rolling basis, so audio output begins as soon as the first few speech token blocks are available. This directly addresses component (3) of latency.

Why the 4-block window size? The paper doesn't justify this specific number, but the design principle is balancing quality and latency. A larger window (more lookback, more lookahead) gives the DiT more context to produce natural-sounding audio — speech prosody and coarticulation (how adjacent sounds influence each other) span multiple phonemes, which may span multiple speech token blocks. A smaller window reduces latency because fewer future blocks need to be generated before the current block can be decoded. Four blocks is a compromise that provides enough context for natural prosody while keeping the lookahead to only 1 block (minimizing the "wait for future tokens" delay). The choice of 2 lookback + 1 lookahead (rather than, say, 1+1 or 3+1) is likely empirically tuned.

Pre-Training: Three-Stage Curriculum

Stage 1: Encoder-only training with frozen LLM. The LLM (Thinker) parameters are frozen. Only the vision encoder and audio encoder are trained, using a large corpus of:

Audio-text pairs (for the audio encoder)
Image-text pairs (for the vision encoder)

Within this stage, training proceeds in a specific order:

"both initially focusing on training their respective adapters before training the encoders"

This implies that the encoders have adapter layers — lightweight trainable modules inserted between the frozen encoder and the frozen LLM — that are trained first to learn the mapping from encoder output space to LLM input space. Only after the adapters converge are the full encoder parameters unfrozen and trained. This progression prevents the encoder weights from being corrupted by random adapter initialization.

The paper states that stage 1 is "crucial in equipping the model with a robust understanding of core visual-textual and audio-textual correlations and alignments." This is because the encoders are initialized from pretrained models (Whisper-large-v3 and Qwen2.5-VL ViT) but their outputs were originally designed for different downstream tasks, not for feeding into Qwen2.5's embedding space. Stage 1 learns the cross-modal mapping.

Stage 2: Full model training with multimodal data. All parameters are unfrozen, and the model is trained on a much larger and more diverse dataset:

800 billion tokens of image- and video-related data
300 billion tokens of audio-related data
100 billion tokens of "video with audio" related data
Additional pure text data (quantity not specified) to maintain language proficiency

The sequence length is limited to 8,192 tokens during stages 1 and 2:

"To improve training efficiency, we limited the maximum token length to 8192 tokens in the previous stages."

The inclusion of "a larger volume of mixed multimodal data and a wider variety of tasks" is designed to "enhance the interaction and deepen the understanding between auditory, visual, and textual information." Critically, the video-with-audio data (100 billion tokens) is where TMRoPE's interleaving and temporal synchronization matter most — this is the data that teaches the model to associate visual events with corresponding sounds.

Stage 3: Long-sequence extension. The maximum token length is extended from 8,192 to 32,768 tokens. This stage incorporates:

Long audio data (presumably recordings longer than the ~5.5 minutes that fit in 8,192 tokens at the paper's encoding rate)
Long video data (videos with many frames)
Extended versions of text, image, and video data from earlier stages

The paper reports that "this data shows significant improvement in supporting long sequence data," but provides no quantitative ablation of stage 3's impact. The main practical effect is enabling the model to handle videos longer than a few minutes and audio recordings longer than a few minutes.

A key design choice: replacing hierarchical tags with natural language. The paper mentions following Qwen2-Audio's approach:

"We replace the hierarchical tags with the natural language prompts following Qwen2-Audio, which can improve better generalization ability and better instruction following ability."

Hierarchical tags are structured labels like <|audio|><|asr|><|en|> that were used in earlier Qwen-Audio models to specify the task, language, and modality. Replacing these with natural language prompts (e.g., "Please transcribe the following English audio:") makes the training distribution more similar to how users will actually interact with the model at inference time, which improves generalization to unseen instruction formats.

Post-Training: Instruction Fine-Tuning and Talker Specialization

Data format: ChatML. The post-training data uses the ChatML format, which is a simple markup for multi-turn conversations. Each turn is wrapped in <|im_start|> and <|im_end|> tags with a role identifier (user or assistant). Multimodal content is embedded using special tokens like <|vision_start|> and <|vision_end|> that bracket the visual content. The paper provides a concrete example showing a two-turn interaction where the user asks about a video and the assistant describes it, demonstrating how video, audio, and text are interleaved in the ChatML structure.

Thinker post-training. Thinker is fine-tuned on instruction-following data covering:

Pure text-based dialogue data (standard instruction tuning)
Visual-modality conversation data (image QA, video description)
Audio-modality conversation data (speech interaction, audio event description)
Mixed-modality conversation data (questions requiring joint visual and audio reasoning)

This phase uses standard supervised fine-tuning (SFT): the model is trained to predict the assistant's responses given the conversation history, with a cross-entropy loss over the text tokens. The ChatML format clearly separates user and assistant turns, so loss is only computed on assistant tokens.

Talker post-training (already detailed above). The three stages — ICL speech continuation, DPO stabilization, multi-speaker fine-tuning — are applied after Thinker has been instruction-tuned, ensuring that Talker learns to generate speech for the types of responses Thinker actually produces in conversational settings.

A critical training insight: validation loss as an unreliable signal for early stopping. While this is mentioned in the paper's description of revision models (which is not part of Qwen2.5-Omni — the reader should note that this insight is from a different model family, but the principle applies), the general issue is that after fine-tuning, the validation set distribution may not match the training distribution because the model's own outputs become part of the input context during multi-turn interactions. The authors note selecting checkpoints "slightly after the point where validation loss begins increasing" — this is a practical heuristic that acknowledges standard early stopping criteria break down when the model's behavior shifts the data distribution.

4. Key Insights and Innovations

The paper's most conceptually distinctive contribution is not that it uses a new position encoding, but rather its diagnosis that temporal misalignment is the primary bottleneck preventing genuine audio-visual joint understanding — and its solution of making position encodings physically time-calibrated rather than sequence-index-calibrated.

Prior work on multimodal position encoding (including Qwen-VL's M-RoPE, which this paper extends) treated the temporal dimension as a sequence counter: each frame or audio segment increments a counter by 1, regardless of how much physical time elapsed between them. This works for modalities in isolation but creates a fundamental mismatch when audio (sampled at a fixed 40ms per frame) and video (sampled at variable frame rates) must be jointly reasoned about. Under a naive counter-based scheme, the 3rd audio frame and the 3rd video frame might correspond to completely different moments in physical time if the video frame rate differs from 25fps, making it structurally impossible for attention to learn cross-modal associations that depend on temporal coincidence.

TMRoPE's advance is conceptual, not merely architectural. It reinterprets the temporal position ID as a physical time coordinate — each increment of 1 equals exactly 40ms — and dynamically computes frame spacing for variable frame rate video to maintain this calibration. This transforms the position embedding from a sequence ordering mechanism into a cross-modal synchronization signal. When a visual token at t=1.5s and an audio token at t=1.5s have identical temporal position components, the model's attention geometry naturally groups them — the positional similarity guides attention toward temporally coincident cross-modal pairs without the model needing to discover this correspondence from content alone.

The significance of this reframing extends beyond the specific implementation. It identifies a general design principle for any multimodal system that processes time-series inputs with different sampling rates: position encodings should encode physical time, not sequence index. This principle applies equally to robotics (synchronizing camera frames with joint encoder readings at different frequencies), medical AI (aligning ECG waveforms with video of surgical procedures), or any domain where multiple sensors capture correlated signals at mismatched rates.

The evidence for this innovation's impact is primarily in the OmniBench results (Table 8), where Qwen2.5-Omni achieves 56.13% overall — a dramatic improvement over the next best omni model (Baichuan-Omni-1.5 at 42.9%) and more than double the performance of similarly-sized prior work (UnifiedIO2-xxlarge at 33.98%). The OmniBench benchmark specifically tests "the ability to simultaneously understand and analyze information from multiple modalities" — tasks where audio and visual information must be jointly reasoned about. The fact that Qwen2.5-Omni outperforms Gemini-1.5-Pro (42.91%) by over 13 percentage points on this benchmark, despite Gemini being from a larger model family, strongly suggests that explicit temporal alignment is providing a capability that even scaled-up models with ad-hoc temporal handling cannot easily replicate. The time-interleaving method's 2-second chunking further amplifies this effect by placing temporally coincident visual and audio tokens adjacent in the sequence — attention heads with local receptive fields will naturally attend across modalities within each chunk.

Innovation 2: Architectural Separation of Semantics and Acoustics as Interference Prevention

The Thinker-Talker split is more than an engineering convenience — it is a diagnostic claim about the nature of cross-modal interference in unified generation models, backed by an implicit negative result.

The dominant assumption in multimodal generation has been that expanding a language model's output vocabulary to include non-text tokens (speech codes, image patches, etc.) is sufficient — the same transformer decoder can learn to predict text and speech tokens from shared hidden states because attention will naturally develop modality-appropriate representations at different positions. This assumption underlies models like AudioLM, SpeechGPT, and early unified speech-text LMs that generate speech tokens directly from the LLM decoder. The Thinker-Talker architecture argues, by its very design, that this assumption is wrong in a specific and important way: hidden states that are optimal for semantic prediction (which word comes next) are systematically different from hidden states needed for acoustic prediction (which speech token encodes the appropriate prosody, timing, and speaker characteristics for that word).

The paper identifies the core conflict as a representation geometry problem. Thinker's hidden representations are organized by semantic similarity — words with similar meanings cluster together. But speech generation requires representations organized by phonetic and prosodic similarity — words that sound similar need similar representations even if their meanings are unrelated. The paper gives the example of phonetically distinct synonyms: two words meaning roughly the same thing will have nearly identical hidden states in an LLM, but require completely different speech tokens. If Talker had to generate speech from these hidden states alone, it would face irreducible ambiguity. Conversely, if Thinker's representations were forced to also encode phonetic information, they would become worse at semantic reasoning — the two objectives are in tension.

The paper never explicitly proves this interference claim through ablation (it doesn't report results for a "unified decoder" variant that generates both text and speech from Thinker), but the architecture itself constitutes an implicit negative result: the authors presumably tried simpler approaches and found them wanting. The fact that they invested in designing, training, and evaluating a separate Talker module with its own three-stage post-training pipeline suggests that the interference problem is both real and severe enough to warrant substantial architectural complexity.

The conceptual contribution is identifying the specific mechanism of interference (semantic vs. phonetic representational geometry) rather than treating "multimodal generation" as a monolithic capability that can be solved by scale alone. This reframes the challenge: it's not that models need more parameters to handle multiple output modalities — it's that a single representational bottleneck cannot simultaneously serve two objectives that require differently structured embedding spaces. The human biology analogy (brain vs. vocal apparatus) is more than rhetoric — it captures the idea that these functions require distinct processing substrates that share information through a high-bandwidth but structured interface (hidden states from Thinker to Talker) rather than through shared parameters.

The evidence for this innovation's success is distributed across two sets of results. First, Qwen2.5-Omni's speech generation quality (Table 9) is competitive with or exceeds specialized TTS systems — achieving 1.42% WER on SEED test-zh versus 1.45% for CosyVoice 2 and 2.27% for MaskGCT — despite these being dedicated TTS models that don't need to simultaneously perform multimodal understanding. If cross-modal interference were degrading speech quality, we would expect the unified model to substantially underperform specialized TTS systems; the fact that it doesn't suggests the Thinker-Talker split is successfully isolating the speech generation pathway. Second, the text understanding benchmarks (Table 1) show that Qwen2.5-Omni's text capabilities, while somewhat below Qwen2.5-7B (47.0% vs. 56.3% on MMLU-Pro), are not catastrophically degraded — which they might be if the LLM backbone were contorting its representations to also support speech generation.

Innovation 3: End-to-End Speech Instruction Following That Closes the Text-Speech Gap

A persistent failure mode of speech-enabled language models has been that they perform dramatically worse when instructions are spoken rather than typed. The paper's results on this front (Tables 4 and the in-house voice-chat benchmark) represent, if not a complete solution, then a substantially narrowed gap that changes the viability calculation for voice-only interfaces.

Prior audio-language models like Qwen2-Audio showed enormous drops when switching from text to speech input. On the paper's in-house benchmark (Table 4), Qwen2-Audio achieved only 33.2% on MMLU with speech input, compared to Qwen2-7B's 69.3% with text — a gap of 36.1 percentage points. On GSM8K, the gap was even larger: 18.4% (speech) versus 82.3% (text). This is not merely an accuracy regression; it represents a qualitative failure — the model with speech input is performing near random chance on tasks that the same underlying LLM handles competently with text. The cause is well-known: speech encoders introduce noise (ASR errors), lose formatting information (punctuation, capitalization, mathematical notation), and strip out structural cues (line breaks, indentation) that LLMs rely on for reasoning tasks. The model sees degraded text and performs accordingly.

Qwen2.5-Omni achieves 65.6% on MMLU and 85.4% on GSM8K with speech input — gaps of only 3.7 and −3.1 percentage points respectively from Qwen2-7B's text performance. The negative gap on GSM8K (the speech model slightly outperforms the text baseline) is particularly striking and suggests the speech encoder may be providing additional prosodic cues that help disambiguate mathematical reasoning in some cases, or that the joint training has improved the LLM's robustness to input variation.

This is more than a benchmark improvement — it is a threshold-crossing result for practical voice interfaces. A gap of 36 percentage points on MMLU means a voice assistant is functionally useless for knowledge tasks; a gap of 3.7 points means it's comparable to text input for most practical purposes. The paper doesn't isolate which architectural choice drives this improvement, but the likely candidates are: (1) training the audio encoder jointly with the LLM on diverse multimodal data (stage 2 pre-training with 300B audio tokens) rather than keeping the audio encoder frozen or separately trained, allowing the encoder to learn representations optimized for the LLM's reasoning needs rather than just ASR accuracy; (2) the ChatML format's explicit separation of modalities, which allows the model to distinguish audio-derived content from text-derived content in its internal processing; and (3) the large-scale instruction tuning that includes speech-input task formats, teaching the model to handle the degraded-format input that speech encoders produce.

The conceptual significance is that it reframes the speech-input problem from an ASR quality problem (get better transcription, and the LLM will perform better) to a representation learning problem (train the encoder to produce representations that are useful for reasoning, not just accurate transcription). This aligns with findings from the vision-language literature, where early approaches that pipelined OCR → LLM were superseded by end-to-end vision encoders that learn to extract task-relevant visual features directly. The speech equivalent is that a jointly-trained audio encoder can learn to preserve mathematical notation structure, logical connectors, and other reasoning-critical features that would be lost in a pure ASR pipeline.

Innovation 4: DPO-Based Stabilization of Speech Generation with Objective Reward Signals

The paper's application of Direct Preference Optimization to speech generation introduces a methodological bridge between the RLHF-for-text and speech-synthesis communities, with a diagnostic finding about what kind of reward signal works for stabilizing neural speech generation.

Prior work in neural TTS has struggled with a specific class of failures that the paper calls "model hallucinations" — mispronunciations, inappropriate pauses, attention misalignment, and garbled audio segments. The dominant approaches for addressing these have been architectural (improving attention mechanisms, adding duration predictors) or data-based (curating cleaner training data, using forced alignment for explicit duration supervision). These are symptom-specific fixes: a duration predictor fixes timing errors, an improved attention mechanism fixes alignment failures, but each addresses one failure mode.

The paper's use of DPO is conceptually different: it frames speech generation errors as a preference learning problem rather than an architecture problem. The key innovation is in the reward signal construction. Instead of using human preference judgments (as is standard in text RLHF), the paper uses fully automated, objective metrics — word error rate (WER) and punctuation pause error rate — to rank speech samples as "winning" or "losing." This is possible because speech generation quality has well-defined objective correlates that text generation quality lacks: you can measure WER by running ASR on the generated speech and comparing to the reference text, and you can detect pause errors by comparing the timing of punctuation in the text to pauses in the generated audio.

The conceptual significance is twofold. First, it demonstrates that objective, automatically-computable reward signals can effectively drive preference optimization for speech — removing the need for expensive human annotation in the RL loop. This is not obvious a priori; WER captures word accuracy but not naturalness, prosodic appropriateness, or emotional expression. The fact that DPO with WER-based rewards improves both content consistency (Table 9: WER drops from ~7.97% to 6.54% on SEED test-hard after RL) and preserves speaker similarity (0.747 → 0.752 on the same set) suggests that optimizing for word accuracy doesn't come at the expense of other speech qualities — or that the DPO objective's KL penalty to the reference policy successfully prevents such degradation.

Second, it identifies a specific pathology that DPO addresses: the pretraining data contains "label noise and pronunciation errors" that cause "model hallucinations." Standard next-token prediction training on imperfect data learns to replicate the errors in the training distribution. DPO's preference pairs explicitly teach the model to avoid error patterns — the losing samples are the erroneous generations, and the model learns to shift probability mass away from them. This is a fundamentally different training signal than simply training on more (possibly still noisy) data.

The evidence is in the SEED test-hard results (Table 9), where the RL-optimized model (Qwen2.5-Omni-7B_RL) achieves 6.54% WER versus 7.97% for the ICL-only version — a 17.9% relative improvement on the most challenging test set. The test-hard set is designed to be difficult precisely because it contains the kinds of edge cases (rare words, complex prosody, challenging phonetic sequences) where model hallucinations are most likely. The improvement on this set specifically supports the claim that DPO is reducing hallucinations rather than just improving average-case performance.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans multiple benchmarks across modalities. For text, the paper uses MMLU-Pro (Wang et al., 2024f), MMLU-redux (Gema et al., 2024), LiveBench0831 (White et al., 2024), GPQA (Rein et al., 2023), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), MultiPL-E (Cassano et al., 2023), and LiveCodeBench 2305-2409 (Jain et al., 2024). For audio understanding, benchmarks include LibriSpeech, Common Voice 15, Fleurs, Wenetspeech, Voxpopuli, CoVoST2, Meld, VocalSound, GiantSteps, MusicCaps, MMAU (Sakshi et al., 2024), and VoiceBench (Chen et al., 2024b). For image understanding: MMMU (Yue et al., 2023), MMMU-Pro (Yue et al., 2024), MathVista (Lu et al., 2024b), MathVision (Wang et al., 2024b), MMBench-V1.1 (Liu et al., 2023c), MMVet (Yu et al., 2024), MMStar (Chen et al., 2024a), MME (Fu et al., 2023), MuirBench (Wang et al., 2024a), CRPE (Wang et al., 2024d), RealWorldQA (X.AI., 2024), MME-RealWorld (Zhang et al., 2024), MM-MT-Bench (Agrawal et al., 2024), AI2D (Kembhavi et al., 2016), TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), and OCRBench_v2 (Fu et al., 2024b). For video understanding: Video-MME (Fu et al., 2024a), MVBench (Li et al., 2024a), and EgoSchema (Mangalam et al., 2023). For multimodal understanding: OmniBench (Li et al., 2024b). For speech generation: SEED (Anastassiou et al., 2024) test-zh, test-en, and test-hard sets. Standard dataset splits are used throughout, and an in-house voice-chat benchmark converts text instructions from MMLU, CEval, IFEval, GSM8K, Math23K, and Math401 into speech for evaluation (approximately 90% of text instructions are utilized). The grounding evaluation uses RefCOCO, RefCOCO+, RefCOCOg (Kazemzadeh et al., 2014; Mao et al., 2016), ODinW (Li et al., 2022), and a self-curated point grounding benchmark.
Base model(s). Qwen2.5-Omni is a 7B-parameter unified multimodal model. The Thinker component is initialized from Qwen2.5-7B (Yang et al., 2024b), the vision encoder is the same 675M-parameter ViT from Qwen2.5-VL (Bai et al., 2025), and the audio encoder is initialized from Whisper-large-v3 (Radford et al., 2023). The 7B scale is described as representative for comparing against similarly-sized single-modality and omni models. No larger or smaller variants are evaluated, and the paper does not report results for Qwen2.5-Omni at other parameter scales.
Metrics. For text generation (all X→Text tasks), the primary metric is accuracy (percentage of questions answered correctly), with exact match or equivalent grading functions as defined by each benchmark's standard evaluation protocol. For speech generation (X→Speech), the paper uses Word Error Rate (WER) for content consistency — computed by running ASR on the generated speech and comparing to the reference text — and speaker similarity (SIM) for zero-shot speech generation, measured via a speaker embedding cosine similarity metric. For single-speaker speech generation, subjective naturalness is reported as NMOS (Naturalness Mean Opinion Score) on a self-created dataset, with human recordings serving as the reference. For visual grounding, accuracy is reported as the standard metric for referring expression comprehension benchmarks, while mAP (mean Average Precision) is used for open-vocabulary object detection on ODinW.
Baselines. The paper compares against a comprehensive set of models across modalities. For text-to-text: Gemma2-9B, Llama3.1-8B, Qwen2-7B, and Qwen2.5-7B. For audio understanding: SALMONN (Tang et al., 2024), SpeechVerse (Das et al., 2024), Whisper-large-v3 (Radford et al., 2023), Llama-3-8B and Llama-3-70B (Dubey et al., 2024b), Seed-ASR-Multilingual (Bai et al., 2024), MiniCPM-o (Yao et al., 2024), MinMo (Chen et al., 2025), Qwen-Audio (Chu et al., 2023a), Qwen2-Audio (Chu et al., 2024a), Megrez-3B-Omni (Infinigence), SpeechLLaMA (Wu et al., 2023), BLSP (Wang et al., 2023a), WavLM-large (Chen et al., 2022), CLAP (Elizalde et al., 2022), Pengi (Deshmukh et al., 2023), LLark-7B (Gardner et al., 2023), LP-MusicCaps (Doh et al., 2023), Gemini-Pro-V1.5 (Team et al., 2024), Ultravox-v0.4.1-LLaMA-3.1-8B, MERaLiON (He et al., 2024), Lyra-Base (Zhong et al., 2024), and Baichuan-Omni-1.5 (Li et al., 2025). For image understanding: GPT-4o-mini, Qwen2.5-VL-7B (Wang et al., 2024c), and the best-performing omni model from prior work (typically MiniCPM-o or Baichuan-Omni-1.5). For visual grounding: Gemini 1.5 Pro, Grounding DINO (Liu et al., 2024), and Qwen2.5-VL-7B. For video understanding: GPT-4o-mini, Qwen2.5-VL-7B, MiniCPM-o, and Lyra-Base. For multimodal understanding (OmniBench): Gemini-1.5-Pro, MIO-Instruct (7B) (Wang et al., 2024g), AnyGPT (7B) (Zhan et al., 2024), video-SALMONN (13B) (Sun et al., 2024), UnifiedIO2-xlarge (3.2B) and UnifiedIO2-xxlarge (6.8B) (Lu et al., 2024a), MiniCPM-o, and Baichuan-Omni-1.5. For speech generation: Seed-TTS (both ICL and RL variants) (Anastassiou et al., 2024), MaskGCT (Wang et al., 2024e), E2 TTS (Eskimez et al., 2024), F5-TTS (Chen et al., 2024c), CosyVoice 2 and CosyVoice 2-S (Du et al., 2024). For the in-house voice-chat benchmark, Qwen2-7B with text input and Qwen2-Audio with speech input serve as baselines.
Generation budget / compute accounting. The paper does not frame its evaluation in terms of a unified generation budget or FLOPs-matched comparison (unlike the test-time compute scaling paper analyzed in the reference example). Instead, each benchmark is evaluated using standard inference protocols — presumably single-pass greedy or sampling-based decoding for text generation, and the standard autoregressive generation for speech. The paper does not report inference FLOPs, latency measurements, or other compute metrics for any benchmark. This is a notable gap: while the streaming architecture is designed for low latency, no wall-clock time or FLOPs measurements are provided to quantify the streaming performance improvements claimed in Section 2.4.
Cross-validation / statistical protocol. The paper reports no cross-validation, statistical significance testing, confidence intervals, or error bars for any result. All numbers are point estimates. The in-house voice-chat benchmark is described as converting "approximately 90% of text-instructions" from standard benchmarks, but the specific conversion methodology and filtering criteria are not detailed. The self-curated point grounding benchmark and the NMOS evaluation dataset are mentioned but not described in terms of size, construction methodology, or annotator protocol. For the Talker RL stage, the preference pair dataset D is constructed using automated reward scores (WER and punctuation pause error rate), but the paper does not specify the dataset size, the threshold for determining winning vs. losing samples, or the β value used in the DPO objective (Equation 1). The single-speaker fine-tuning evaluates four named speakers (A, B, C, D) but provides no information about these speakers' characteristics, the amount of fine-tuning data per speaker, or the training protocol.

Main Quantitative Results

Text→Text Performance

The paper compares Qwen2.5-Omni's text-only capabilities against similarly-sized pure language models (Table 1). The headline result is that adding multimodal encoders and training on multimodal data causes measurable but non-catastrophic regression on text benchmarks. Qwen2.5-Omni-7B achieves 47.0% on MMLU-Pro, 71.0% on MMLU-redux, and 29.6 on LiveBench0831 — placing it between Qwen2-7B (44.1%, 67.3%, 29.2) and Qwen2.5-7B (56.3%, 75.4%, 35.9). The gap to Qwen2.5-7B is substantial (9.3 percentage points on MMLU-Pro, 4.4 on MMLU-redux), indicating that multimodal training does degrade the base LLM's text capabilities. On mathematics and science tasks, Qwen2.5-Omni achieves 30.8% on GPQA, 71.5% on MATH, and 88.7% on GSM8K — compared to Qwen2.5-7B's 36.4%, 75.5%, and 91.6%. The MATH score is notably strong (only 4 points behind Qwen2.5-7B), while the GPQA regression is more significant (5.6 points). On coding tasks, Qwen2.5-Omni achieves 78.7% on HumanEval, 73.2% on MBPP, 65.8 on MultiPL-E, and 24.6 on LiveCodeBench — consistently between Qwen2-7B and Qwen2.5-7B, with the gap to Qwen2.5-7B being 6.1 points on HumanEval, 6.0 on MBPP, and 4.6 on MultiPL-E. The paper interprets these results as demonstrating "exceptional capabilities" for Text→Text, but a more precise characterization is that the model retains strong text performance despite the addition of multiple modality encoders, though with a consistent regression of roughly 5–10% relative to the unimodal Qwen2.5-7B baseline.

Audio→Text Performance

The audio understanding evaluation (Tables 2 and 3) covers ASR, speech translation, audio reasoning, and voice chatting across 15+ benchmarks. The paper claims Qwen2.5-Omni "delivers better or comparable performance with other state-of-the-art methods on audio understanding."

On ASR, Qwen2.5-Omni achieves competitive but not always best results. On LibriSpeech test-clean/test-other, it scores 1.8/3.4 WER — comparable to Whisper-large-v3 (1.8/3.6) and Seed-ASR-Multilingual (1.6/2.8), but behind Qwen2-Audio (1.6/3.6). On Common Voice 15 English, it achieves 7.6 WER, outperforming Whisper-large-v3 (9.3), MinMo (7.9), and Qwen2-Audio (8.6). On Common Voice 15 Chinese (zh), it achieves 5.2 WER, the best among all compared models (MinMo: 6.3, Qwen2-Audio: 6.9). On Fleurs Chinese, it achieves 3.0 WER, tied with MinMo and well ahead of Whisper-large-v3 (7.7), Seed-ASR (not reported), and Qwen2-Audio (7.5). On Wenetspeech test-net, it achieves 5.9 WER, outperforming Seed-ASR-Chinese (4.7) on test-net but comparable on test-meeting (7.7 vs. 7.4 for MinMo). These ASR results are strong but reveal an inconsistency: the model is state-of-the-art on some benchmarks (Common Voice zh, Fleurs zh) but merely competitive on others (LibriSpeech, where it trails Seed-ASR and Qwen2-Audio). The paper does not discuss this variability.

On speech-to-text translation (CoVoST2), Qwen2.5-Omni achieves 30.2 BLEU on English→German, 37.7 on German→English, 41.4 on English→Chinese, and 29.4 on Chinese→English. It outperforms Qwen2-Audio on three of four directions (the exception being English→Chinese, where Qwen2-Audio achieves 45.2 vs. 41.4), and substantially outperforms older models like SpeechLLaMA (27.1 on De→En, 12.3 on Zh→En) and BLSP (14.1 on En→De). This demonstrates that the unified training has not degraded the speech translation capabilities inherited from Qwen2-Audio.

On audio reasoning (MMAU benchmark, Table 3), Qwen2.5-Omni achieves 65.60% average, dramatically outperforming Qwen2-Audio (49.20%) and Gemini-Pro-V1.5 (54.90%). The breakdown by subcategory is: Sound 67.87%, Music 69.16%, Speech 59.76%. The Music subscore (69.16%) is particularly striking — nearly 19 points above Gemini-Pro-V1.5 (49.40%) and 18 points above Qwen2-Audio (50.98%). This is the strongest evidence in the paper for the claim that joint audio-visual training improves audio understanding beyond what audio-only models achieve, since MMAU requires reasoning about audio content in ways that may benefit from multimodal context. However, the paper provides no ablation isolating whether this improvement comes from TMRoPE specifically, the larger audio training corpus (300B tokens), or the joint training with visual data.

On VoiceBench (Table 3, bottom), Qwen2.5-Omni achieves an average score of 74.12, outperforming MiniCPM-o (71.69), Baichuan-Omni-1.5 (71.14), and Ultravox (71.45). The sub-scores reveal particular strengths on OpenBookQA (81.10 vs. 78.02 for MiniCPM-o) and AdvBench (99.42), with weaker performance on IFEval (52.87 vs. 66.88 for Ultravox). The paper does not analyze these sub-score patterns.

The most significant Audio→Text result is in Table 4, evaluating speech instruction following on standard text benchmarks converted to speech. Qwen2.5-Omni achieves 65.6% on MMLU, 61.1% on CEval, 41.7% on IFEval, 85.4% on GSM8K, 87.1% on Math23K, and 62.2% on Math401. Compared against Qwen2-7B with text input (69.3%, 78.4%, 53.3%, 82.3%, 92.3%, 75.5%), Qwen2.5-Omni narrows the text-speech gap to 3.7, 17.3, 11.6, -3.1, 5.2, and 13.3 percentage points respectively. Compared against Qwen2-Audio with speech input (33.2%, 38.6%, 15.6%, 18.4%, 23.0%, 20.4%), Qwen2.5-Omni represents a dramatic improvement — halving or better the gap to text performance on every benchmark. The GSM8K result where speech slightly outperforms text (85.4 vs. 82.3) is anomalous and the paper does not explain it. The CEval gap remains large (17.3 points), suggesting Chinese-language knowledge tasks are more challenging for speech input than English or math tasks.

Image→Text Performance

The image understanding evaluation (Table 5) compares Qwen2.5-Omni against Qwen2.5-VL-7B (the unimodal vision-language counterpart), GPT-4o-mini, and other omni models. Qwen2.5-Omni achieves comparable performance to Qwen2.5-VL-7B on most benchmarks, with some notable differences. On MMMU (college-level problems), Qwen2.5-Omni achieves 59.2% vs. Qwen2.5-VL-7B's 58.6% — a slight improvement. On MathVista, it achieves 67.9% vs. 68.2% — essentially tied. On MathVision, 25.0% vs. 25.1% — tied. On MMBench-V1.1-EN, 81.8% vs. 82.6% — a small regression. On MMStar, 64.0% vs. 63.9% — tied. On MME, 2340 vs. 2347 — essentially tied. The overall pattern is that Qwen2.5-Omni's image understanding capabilities are nearly identical to Qwen2.5-VL-7B, with deviations typically within 1–2 percentage points. This is a positive result: adding audio encoders, Talker, and multimodal joint training did not catastrophically degrade vision capabilities.

Compared to other omni models, Qwen2.5-Omni outperforms the "Other Best" column on several OCR-related benchmarks: DocVQA (95.2% vs. 93.5%), OCRBench_V2 (57.8% vs. 56.3%), and TextVQA (84.4% vs. 84.9% — slightly behind). On general VQA, it outperforms GPT-4o-mini on MMBench (81.8% vs. 76.0%), MathVista (67.9% vs. 52.5%), and MMStar (64.0% vs. 54.8%), though the comparison across model providers and scales makes interpretation difficult.

The grounding evaluation (Table 6) shows Qwen2.5-Omni achieving competitive performance with Qwen2.5-VL-7B on referring expression comprehension: RefCOCO val 90.5% vs. 90.0%, RefCOCO testA 93.5% vs. 92.5%, RefCOCO+ val 85.4% vs. 84.2%, RefCOCO+ testA 91.0% vs. 89.1%, RefCOCOg val 87.4% vs. 87.2%, RefCOCOg test 87.9% vs. 87.2%. On ODinW (open-vocabulary object detection), Qwen2.5-Omni achieves 42.2 mAP, outperforming Qwen2.5-VL-7B (37.3) and Gemini 1.5 Pro (36.7), but substantially behind the specialist Grounding DINO (55.0). On point grounding, Qwen2.5-Omni achieves 66.5% vs. Qwen2.5-VL-7B's 67.3% — a minor regression. The ODinW improvement is notable and unexplained: why would a unified model with audio capabilities outperform the vision-only counterpart on open-vocabulary detection? The paper offers no hypothesis.

Video→Text Performance

The video understanding evaluation (Table 7) compares Qwen2.5-Omni against Qwen2.5-VL-7B, GPT-4o-mini, and other omni models. Qwen2.5-Omni achieves 64.3% on Video-MME without subtitles, 72.4% on Video-MME with subtitles, 70.3% on MVBench, and 68.6% on EgoSchema. Against Qwen2.5-VL-7B (65.1%, 71.6%, 69.6%, 65.0%), the model is comparable or slightly better — most notably on EgoSchema (68.6% vs. 65.0%, a 3.6-point improvement). Against GPT-4o-mini, Qwen2.5-Omni outperforms on all benchmarks except Video-MME without subtitles (64.3% vs. 64.8% — essentially tied). Against the best prior omni models, Qwen2.5-Omni establishes clear state-of-the-art: 70.3% on MVBench vs. 67.2% (Lyra), 68.6% on EgoSchema vs. 63.2% (Lyra). The Video-MME result with subtitles (72.4%) is particularly strong, suggesting the model effectively leverages the additional textual information from subtitles.

Multimodality→Text Performance

The OmniBench evaluation (Table 8) is arguably the paper's most important result for demonstrating cross-modal integration. Qwen2.5-Omni achieves 56.13% overall average, with sub-scores of 55.25% on Speech, 60.00% on Sound Event, and 52.83% on Music. This dramatically outperforms all prior omni models: Gemini-1.5-Pro (42.91%), Baichuan-Omni-1.5 (42.9%), MiniCPM-o (40.5%), UnifiedIO2-xlarge (38.00%), video-SALMONN (35.64%), MIO-Instruct (33.80%), and AnyGPT (18.04%). The margin over the next-best model (Baichuan-Omni-1.5) is 13.23 percentage points — an unusually large gap in multimodal benchmarking that suggests a qualitative difference in capability rather than incremental improvement. The Sound Event sub-score (60.00%) is the strongest, while Music (52.83%) is the weakest — consistent with music understanding being generally more challenging and training data for music being scarcer than for speech and environmental sounds.

X→Speech Performance

Zero-shot speech generation. Table 9 compares Qwen2.5-Omni against specialized TTS systems. After RL optimization, Qwen2.5-Omni achieves 1.42% WER on SEED test-zh (vs. 1.45% for CosyVoice 2, 2.27% for MaskGCT, 1.00% for Seed-TTS_RL), 2.33% on test-en (vs. 2.57% for CosyVoice 2, 2.62% for MaskGCT, 1.94% for Seed-TTS_RL), and 6.54% on test-hard (vs. 6.83% for CosyVoice 2, 10.27% for MaskGCT, 6.42% for Seed-TTS_RL). The paper claims Qwen2.5-Omni "outperforms MaskGCT and CosyVoice 2," which is technically true for test-zh and test-hard (1.42% < 1.45% for CosyVoice 2 on test-zh; 6.54% < 6.83% on test-hard), but on test-en, CosyVoice 2 achieves 2.57% vs. Qwen2.5-Omni's 2.33% — the ordering is reversed, and Qwen2.5-Omni does outperform CosyVoice 2 here. The more accurate summary is that Qwen2.5-Omni is competitive with CosyVoice 2 (slightly better on test-zh and test-hard, better on test-en) but still behind Seed-TTS_RL (1.00%, 1.94%, 6.42%). For speaker similarity, Qwen2.5-Omni_RL achieves 0.754, 0.641, 0.752 on test-zh, test-en, test-hard — comparable to CosyVoice 2 (0.748, 0.652, 0.724) and slightly behind Seed-TTS_RL (0.801, 0.766, 0.782). The RL optimization (comparing Qwen2.5-Omni_ICL to Qwen2.5-Omni_RL) improves WER across all three sets (1.70→1.42, 2.72→2.33, 7.97→6.54) with minimal impact on speaker similarity (0.752→0.754, 0.632→0.641, 0.747→0.752), demonstrating that the DPO stage successfully reduces errors without degrading voice characteristics.

Single-speaker speech generation. Table 10 reports results for four fine-tuned speakers (A, B, C, D). On SEED test-zh, the four speakers achieve WER of 1.29%, 1.37%, 1.30%, and 1.28% respectively — comparable to human recordings (1.25%) and the base RL model (1.30%). On test-en, they achieve 1.86%, 1.89%, 2.13%, and 1.83% — outperforming human recordings (2.14%) in three of four cases, which is an unusual result that the paper does not discuss (machine-generated speech having lower WER than human speech suggests the ASR system used for WER computation may be biased). On test-hard, WER ranges from 6.43% to 7.25%, comparable to the base RL model (6.54%). Subjective naturalness (NMOS) scores range from 4.46 to 4.51 for Chinese and 4.51 to 4.62 for English, with human recordings scoring 4.51 for Chinese (English human NMOS not reported). These NMOS scores are remarkably high and close to human-level, though the self-created dataset and unspecified evaluation protocol make these numbers difficult to interpret without additional context.

Ablation Studies and Robustness Checks

Effect of RL optimization on speech generation stability: Comparing Qwen2.5-Omni_ICL (before DPO) with Qwen2.5-Omni_RL (after DPO) in Table 9 shows consistent WER improvements across all SEED test sets, with the largest gain on test-hard (7.97 → 6.54, a 17.9% relative improvement). Speaker similarity remains essentially unchanged (e.g., test-zh SIM: 0.752 → 0.754), indicating that DPO targeted error reduction without distorting voice characteristics. The paper states that RL training produced "significant improvements in generation stability, with marked reductions in attention misalignment, pronunciation errors, and inappropriate pauses," but these specific failure mode reductions are claimed qualitatively without quantitative breakdown by error type.

Effect of speaker fine-tuning on naturalness and content consistency: Table 10 compares the base RL model against four speaker-fine-tuned variants. Content consistency (WER) is comparable across all variants and human recordings, with minor variations within ~0.1% WER on test-zh. Subjective naturalness (NMOS) for the fine-tuned speakers (4.46–4.62) approaches human-level quality (4.51 for Chinese). This demonstrates that speaker-specific fine-tuning can adapt voice characteristics without degrading intelligibility, though the lack of a non-fine-tuned NMOS baseline (the paper does not report NMOS for the base RL model or the ICL model) makes it difficult to quantify how much naturalness the fine-tuning adds.

Impact of TMRoPE and time-interleaving on multimodal understanding: The paper provides no ablation study that isolates TMRoPE's contribution to performance. There is no comparison against a model variant using standard M-RoPE without temporal calibration, or a variant without the 2-second interleaving. The OmniBench results (Table 8) are attributed to the overall Qwen2.5-Omni system, not to TMRoPE specifically. The claim that TMRoPE "enhances positional information modeling, maximizing the integration of various modalities" (Section 2.2) is therefore an architectural hypothesis rather than an empirically verified contribution.

Thinker-Talker split vs. unified decoding: The paper provides no experiment comparing the Thinker-Talker architecture against a baseline where Thinker generates both text and speech tokens directly (a unified decoder approach). The speech generation quality (Tables 9 and 10) and text understanding quality (Tables 1, 4–8) are evaluated separately, but there is no demonstration that the same performance could not be achieved with a simpler architecture. The claim that the Thinker-Talker split prevents cross-modal interference is logical but untested by ablation.

Block-wise streaming vs. full attention for encoders: The paper states that the audio encoder was changed "from full attention over the entire audio to performing attention in blocks of 2 seconds each" (Section 2.4), and the vision encoder uses flash attention. However, there is no comparison of streaming vs. non-streaming encoder configurations on any benchmark. The impact of block-wise attention on understanding accuracy is not quantified. The streaming design is evaluated only implicitly through the model's performance on standard benchmarks, which do not measure latency or streaming-specific quality degradation.

Sliding-window DiT receptive field size: The DiT's receptive field is limited to 4 blocks (2 lookback + 1 lookahead). No ablation over different window sizes is provided. The paper does not report the impact of this restricted receptive field on speech naturalness or WER compared to a DiT with full context. The WER and NMOS results in Tables 9 and 10 are for the complete model with the 4-block window, so the cost of the restricted receptive field in terms of speech quality is unknown.

Pre-training stage ablation: The paper describes a three-stage pre-training curriculum (encoders-only with frozen LLM → full model training → long-sequence extension) but provides no results for models trained with only stages 1 and 2, or for models skipping stage 3. The contribution of long-sequence training (32k tokens) is not quantified, and the paper's claim that "this data shows significant improvement in supporting long sequence data" (Section 3) is unsupported by any table or figure.

Data mixture ablation: The paper reports training data quantities (800B image/video tokens, 300B audio tokens, 100B video-with-audio tokens, plus unspecified text data) but provides no experiments varying these ratios. The contribution of video-with-audio data specifically — which is central to the TMRoPE interleaving method — is not isolated.

ChatML format impact: The post-training uses ChatML formatting. There is no comparison against alternative prompt formats or evidence that ChatML specifically contributes to the model's instruction-following quality.

Critical Assessment

Claim: Qwen2.5-Omni "achieves state-of-the-art performance on multimodal benchmarks like OmniBench." This claim is directly supported by Table 8, where Qwen2.5-Omni's 56.13% on OmniBench substantially exceeds all prior omni models. However, the claim should be qualified: state-of-the-art is relative to open-source omni models at similar scale. The paper does not compare against GPT-4o (full, non-mini), Gemini 1.5 Pro's best configuration, or any model substantially larger than 7B parameters. OmniBench results for models like GPT-4o or Claude 3.5 are not reported, so "state-of-the-art" means "best among models evaluated in this paper," which is a narrower claim.

Claim: Qwen2.5-Omni's "performance in end-to-end speech instruction following is comparable to its capabilities with text inputs." Table 4 partially supports this claim for English benchmarks (MMLU: 65.6% speech vs. 69.3% text baseline, a 3.7-point gap; GSM8K: 85.4% speech vs. 82.3% text — speech outperforms text). However, the gap is larger on Chinese benchmarks (CEval: 61.1% vs. 78.4%, a 17.3-point gap; Math401: 62.2% vs. 75.5%, a 13.3-point gap). The claim of "comparable" performance holds reasonably for English and math tasks but breaks down for Chinese knowledge tasks. Additionally, the text baseline is Qwen2-7B, not Qwen2.5-7B, so the comparison is against Qwen2.5-Omni's own predecessor rather than the strongest available text model of similar scale. A comparison against Qwen2.5-7B with text input would show larger gaps.

Claim: Qwen2.5-Omni's "streaming Talker outperforms most existing streaming and non-streaming alternatives in robustness and naturalness." The evidence in Tables 9 and 10 shows the model is competitive with specialized TTS systems, but the "most" qualifier is imprecise. On SEED test-zh WER, Qwen2.5-Omni (1.42%) outperforms MaskGCT (2.27%), E2 TTS (1.97%), F5-TTS (1.56%), and CosyVoice 2 (1.45%), but trails Seed-TTS_RL (1.00%). On test-en, it outperforms MaskGCT (2.62%), CosyVoice 2 (2.57%), F5-TTS (1.83%), and E2 TTS (2.19%), but trails Seed-TTS_RL (1.94%). On test-hard, it outperforms MaskGCT (10.27%), CosyVoice 2 (6.83%), and F5-TTS (8.67%), but trails Seed-TTS_RL (6.42%). So the claim holds against MaskGCT and CosyVoice 2 specifically, but not against the Seed-TTS family. The streaming vs. non-streaming comparison is not directly tested — the paper does not report latency measurements for Qwen2.5-Omni or the baseline systems, so there is no evidence that Qwen2.5-Omni achieves its competitive WER while also being faster or lower-latency than the non-streaming alternatives.

Missing experiments that would strengthen the paper:

Latency benchmarks. The streaming architecture is a major claimed contribution, but no wall-clock time, time-to-first-token, or initial packet latency measurements are reported for any configuration. Without these, the paper cannot substantiate claims about streaming performance improvements.
Ablation of TMRoPE. A comparison against standard M-RoPE without temporal calibration on OmniBench would directly test whether explicit temporal alignment is responsible for the cross-modal understanding gains, or whether they come from other factors (data scale, joint training, architecture size).
Ablation of Thinker-Talker split. A baseline where Thinker directly generates speech tokens (unified decoder) would test the core architectural hypothesis about cross-modal interference. Without this ablation, the Thinker-Talker architecture is a design choice supported by analogy and reasoning, not by empirical evidence.
Ablation of block-wise streaming attention. A comparison of understanding accuracy with full-attention vs. block-wise encoders would quantify the accuracy cost (if any) of the streaming design.
Scaling experiments. The paper evaluates only the 7B model. Results at smaller scales (e.g., 1B, 3B) would show whether the architectural innovations provide disproportionate benefits at certain scales, and results at larger scales would test whether the approach scales smoothly.
Error analysis for speech generation. The WER improvements from RL are attributed to reduced "model hallucinations," but no breakdown by error type (mispronunciation, repetition, omission, insertion) is provided. This makes it difficult to understand what specific problems RL solves and which remain.
Comparison against cascaded pipelines. A major motivation for end-to-end omni-models is avoiding error propagation in cascaded systems (ASR → LLM → TTS). The paper provides no comparison against a pipeline that chains Whisper-large-v3 + Qwen2.5-7B + CosyVoice 2 on end-to-end tasks like voice chatting. Such a comparison would directly quantify the benefit of unified training over modular composition.
Confidence intervals. All results are point estimates without error bars. For benchmarks with small test sets (e.g., OmniBench, MMAU, SEED test-hard), sampling variance could meaningfully affect rankings. Statistical testing would clarify which differences are reliable.

Weaknesses in evaluation design:

The test sets for speech generation (SEED) are standard but limited in scope. SEED evaluates content consistency (WER) and speaker similarity, but does not evaluate prosodic naturalness, emotional appropriateness, or conversational flow — qualities that are critical for voice assistants but not captured by WER.
The NMOS evaluation protocol is opaque. The paper reports subjective naturalness scores approaching human-level quality (4.46–4.62) but does not describe the number of raters, their qualifications, the rating scale design, the number of samples evaluated, or whether raters were blinded to condition. These scores are therefore not reproducible or interpretable.
The in-house voice-chat benchmark (Table 4) uses only "approximately 90% of text-instructions" from standard benchmarks. The filtering criteria are unspecified, and the conversion process (text-to-speech synthesis? human recording?) is not described. This makes it impossible to assess whether the benchmark fairly represents speech interaction scenarios or introduces systematic biases.
Table 10 reports that three of four fine-tuned speakers achieve lower WER than human recordings on SEED test-en. This anomalous result (machines more intelligible than humans on an English speech benchmark) is not discussed and raises questions about the WER measurement methodology — possibly the ASR system used for scoring is better at recognizing synthesized speech than natural speech with its variability in accent, speaking rate, and disfluencies.
No evaluation of streaming-specific failure modes. The paper claims the model can process and generate information in real-time, but provides no evaluation of how performance degrades under streaming constraints — for instance, whether understanding accuracy drops when only partial audio/video input is available, or whether speech naturalness suffers when Talker must generate speech tokens before the full text response is known (the anticipatory challenge). These are precisely the failure modes that the streaming architecture is designed to mitigate, and without evaluating them, the paper cannot demonstrate that the architecture succeeds at its intended purpose.
Single model family (Qwen). All results are for Qwen-initialized models. The architectural innovations (TMRoPE, Thinker-Talker) are evaluated in the context of Qwen's specific LLM, vision encoder, and audio encoder. Transferability to other model families (Llama, Gemma, etc.) is not demonstrated.

Summary of what the experiments demonstrate and what they do not:

The experiments convincingly demonstrate that Qwen2.5-Omni is a strong unified multimodal model that (a) matches or nearly matches unimodal specialists on single-modality benchmarks (Qwen2.5-VL-7B on vision, Qwen2-Audio on audio, Qwen2.5-7B on text at somewhat reduced levels), (b) substantially outperforms prior omni-models on cross-modal integration benchmarks (OmniBench), (c) dramatically narrows the speech-vs-text instruction-following gap, and (d) produces speech with quality competitive with specialized TTS systems.

What the experiments do not demonstrate — but what the paper's architectural narrative claims — is that TMRoPE specifically causes the cross-modal improvements, that the Thinker-Talker split specifically prevents interference (vs. simply adding more parameters), that the streaming design choices achieve their latency targets without accuracy degradation, or that the model's speech generation is genuinely real-time in a wall-clock sense. These are all plausible hypotheses consistent with the results, but the paper's evaluation methodology does not isolate them as causal factors. The paper's contributions are best understood as a successful systems integration — demonstrating that these architectural choices can be combined into a working, high-performing model — rather than as a controlled experiment demonstrating why each choice is necessary.

6. Limitations and Trade-offs

Streaming Latency Claims Are Unsubstantiated by Measurements

The assumption or constraint. The paper positions streaming as a central architectural contribution, dedicating Section 2.4 to "Designs for Streaming" and identifying four specific latency components (multimodal input processing delay, time-to-first-voice-token, codec-to-waveform conversion delay, and inherent architectural latency). The block-wise audio encoding (2-second chunks), the dual-track Talker for pipelined text-speech generation, and the sliding-window DiT with restricted 4-block receptive field are all explicitly designed to reduce initial packet latency. However, the paper provides no latency measurements whatsoever — no wall-clock time, no time-to-first-token, no initial packet delay, and no end-to-end response latency for any configuration or any hardware setup.

The consequence. A practitioner cannot determine whether Qwen2.5-Omni actually achieves real-time performance suitable for conversational voice interaction. The claimed streaming capability is entirely unvalidated from a systems perspective. Several non-obvious latency bottlenecks could undermine the architecture: the 2-second chunk size for audio encoding means the model waits for 2 seconds of audio before processing the first chunk — if this is strictly enforced, the minimum input processing delay is 2 seconds, which is far above the ~200-300ms threshold for natural conversational turn-taking. The Talker's dependency on Thinker's hidden states means speech generation is gated by Thinker's autoregressive text generation speed — if Thinker generates text slowly (a 7B-parameter transformer decoder on consumer hardware), the pipelining provides no latency benefit because Talker is starved for input. The DiT's lookahead of 1 block means waveform generation for block N cannot begin until block N+1's speech tokens are available, introducing a minimum buffering delay proportional to the block size (which the paper does not specify). Without measurements, none of these tradeoffs can be evaluated, and the streaming architecture remains a design claim rather than a demonstrated capability.

What evidence exists in the paper. None. The paper reports no latency numbers, no FLOPs measurements, no inference-time benchmarks, and no comparison of streaming vs. non-streaming configurations on any temporal metric. The evaluation sections (Tables 1–10) report only accuracy and quality metrics, with no indication of whether these results were obtained under streaming or offline (full-context) conditions. The block-wise encoder modifications, the sliding-window DiT, and the Talker pipelining are described architecturally but never evaluated for their latency impact. The paper provides no evidence that the streaming design choices (which impose constraints like limited receptive fields and chunked attention) do not degrade quality relative to a non-streaming configuration — this is a missing ablation entirely.

Mitigation status. Not addressed. The conclusion (Section 6) acknowledges "a more robust and faster model" as future work, implicitly recognizing that inference speed is a current limitation, but provides no quantification of the gap. No latency profiling, no hardware specifications for deployment, and no guidance on expected performance across different GPU configurations are offered.

The assumption or constraint. The paper presents TMRoPE (Time-aligned Multimodal RoPE) as a key innovation, stating that it "enhances positional information modeling, maximizing the integration of various modalities, enabling Qwen2.5-Omni to simultaneously understand and analyze information from multiple modalities" (Section 2.2). The architecture description implies that temporal calibration (scaling position IDs so that one increment equals exactly 40ms across audio and variable-frame-rate video) and the 2-second time-interleaving method are responsible for the model's strong cross-modal understanding, particularly on OmniBench. However, the paper provides no ablation study that isolates TMRoPE's effect. There is no comparison against a model variant using standard M-RoPE without temporal calibration, nor a variant without the time-interleaving method, nor a variant where audio and video are simply concatenated sequentially without chunked interleaving.

The consequence. A practitioner cannot determine whether TMRoPE specifically, or some other aspect of the training pipeline, drives the cross-modal performance. The OmniBench improvement (56.13% vs. 42.9% for the next-best model, Table 8) could be due to: (a) TMRoPE's temporal alignment, (b) the 100 billion tokens of video-with-audio training data (which prior omni models may have lacked regardless of position encoding), (c) the three-stage pre-training curriculum with 800B+ image/video tokens and 300B+ audio tokens, (d) the larger LLM backbone (7B vs. 3.2B–6.8B for many prior omni models), or (e) some interaction among these factors. Without ablations, TMRoPE remains an architectural hypothesis supported by intuition ("synchronizing timestamps should help cross-modal learning") rather than by causal evidence. This also means that practitioners adapting the approach to other model families cannot know whether implementing TMRoPE is necessary for cross-modal performance or whether standard position encodings with sufficient multimodal training data would suffice.

What evidence exists in the paper. The OmniBench results (Table 8) demonstrate the complete Qwen2.5-Omni system outperforming prior models, but no ablation isolates TMRoPE. The paper does not report OmniBench or any other cross-modal benchmark for a Qwen2.5-Omni variant with standard M-RoPE. The Audio→Text and Video→Text results (Tables 2–4, 7) evaluate individual modality understanding, not cross-modal synchronization — a model could perform well on these benchmarks without any temporal alignment between modalities. Only OmniBench directly tests joint audio-visual understanding, and no component-level ablation is provided.

Mitigation status. Not addressed. The paper describes TMRoPE in detail (Section 2.2, Figure 3) but does not acknowledge the absence of ablation or suggest that future work should isolate its contribution. The paper treats TMRoPE as an integrated architectural feature and evaluates only the complete system.

The Thinker-Talker Separation Is Not Validated Against a Unified Decoder Baseline

The assumption or constraint. The Thinker-Talker architecture is motivated by the claim that generating text and speech from a single decoder causes cross-modal interference: "it is essential to manage potential interference among outputs from different modalities, ensuring that the training processes for outputs such as text and voice tokens do not disrupt each other" (Section 1). The paper argues that Thinker's hidden representations are organized by semantic similarity while speech generation requires phonetic similarity, and that "even phonetically distinct words may have very similar high-level representations, necessitating the input of sampled discrete tokens to eliminate such uncertainty" (Section 2.3). This reasoning is used to justify the separate Talker module with its own parameters, training objectives, and three-stage post-training pipeline (ICL speech continuation, DPO stabilization, multi-speaker fine-tuning). However, the paper provides no comparison against a simpler architecture where Thinker directly generates speech tokens alongside text tokens (a unified decoder with an expanded vocabulary). There is no experiment testing whether the proposed interference actually manifests as degraded text or speech quality in a unified configuration.

The consequence. A practitioner cannot assess whether the substantial additional complexity of the Thinker-Talker architecture (a separate transformer decoder, a custom speech codec, a three-stage training pipeline, and a streaming DiT codec decoder) is justified. If a unified decoder with sufficient multimodal training data could achieve comparable speech quality and text understanding, the Thinker-Talker design represents unnecessary engineering overhead — more parameters, more training stages, more failure modes, and more difficult deployment. The interference argument is plausible but untested, and the paper's speech generation quality (competitive with CosyVoice 2 and MaskGCT, Table 9) and text understanding quality (some regression from Qwen2.5-7B, Table 1) cannot be causally linked to the architectural separation without a controlled comparison. The text regression (47.0% vs. 56.3% on MMLU-Pro) could be due to the Thinker-Talker split successfully preventing interference, or it could be due to the standard catastrophic forgetting that occurs when any LLM is fine-tuned on large amounts of non-text data — regardless of whether speech generation shares the decoder.

What evidence exists in the paper. The paper provides qualitative reasoning about the semantic-vs-phonetic representation conflict (Section 2.3) and reports separate evaluations for text understanding (Tables 1, 4–8) and speech generation (Tables 9–10). There is no unified decoder baseline, no ablation where Thinker generates speech tokens directly, and no experiment measuring cross-modal interference (e.g., degradation in text quality when speech generation is added to the same decoder, or degradation in speech naturalness when text generation shares the decoder). The evidence for the Thinker-Talker architecture is entirely at the level of design rationale and final system performance, not controlled comparison.

Mitigation status. Not addressed. The paper presents Thinker-Talker as a core architectural contribution without acknowledging the absence of a unified decoder baseline or suggesting that future work should compare the two approaches. The motivation is entirely conceptual, drawing an analogy to human brain-vocal apparatus separation.

Speech Instruction Following Degrades Significantly on Non-English Knowledge Tasks

The assumption or constraint. The paper claims that "Qwen2.5-Omni's performance in end-to-end speech instruction following is comparable to its capabilities with text inputs" (Abstract, Section 1, Section 5.1.2). This claim is primarily supported by English-language benchmarks in Table 4, where the speech-text gap is small (MMLU: 65.6% speech vs. 69.3% text baseline, a 3.7-point gap; GSM8K: 85.4% vs. 82.3%, speech outperforms text). However, the same table reveals that the gap is dramatically larger on Chinese-language knowledge benchmarks: CEval shows a 17.3-point gap (61.1% speech vs. 78.4% text), Math401 shows a 13.3-point gap (62.2% vs. 75.5%), and IFEval (which includes Chinese-language instruction following) shows an 11.6-point gap (41.7% vs. 53.3%). The paper does not acknowledge or discuss this language-dependent disparity.

The consequence. For practitioners deploying Qwen2.5-Omni in Chinese-language voice assistant applications, the speech understanding degradation is severe enough to be practically limiting. A 17.3-point drop on CEval means the model with speech input is performing closer to random-chance levels on some categories of Chinese knowledge, making it unreliable for educational, informational, or professional use cases where accuracy matters. The paper's headline claim of "comparable" speech-text performance is misleading for non-English deployments because it averages across benchmarks with very different language distributions, masking the Chinese-language regression. The underlying cause is not diagnosed — it could be that the audio encoder (initialized from Whisper-large-v3, which is English-optimized) performs worse on Chinese speech recognition, that the training data has an English skew, or that Chinese's tonal nature makes speech-to-reasoning transfer harder — but without diagnosis, practitioners cannot estimate whether fine-tuning on additional Chinese speech data would close the gap or whether it reflects a fundamental limitation of the encoder architecture.

What evidence exists in the paper. Table 4 directly shows the disparity: Chinese benchmarks (CEval, Math23K, Math401) have speech-text gaps of 13–17 points, while English benchmarks (MMLU, GSM8K) have gaps of 3–4 points (or reversed). The paper's in-house voice-chat benchmark description notes that "approximately 90% of text-instructions are utilized" for speech conversion but does not report the language breakdown of the filtered instructions. The paper's main evaluation (Table 2) does not separately report ASR performance for Chinese on the in-house voice-chat tasks, making it impossible to determine whether the CEval gap is due to poor Chinese speech recognition or poor reasoning from Chinese speech input.

Mitigation status. Not addressed or acknowledged. The paper does not discuss the Chinese-English performance disparity, does not break down results by language for the in-house benchmark, and does not suggest targeted improvements for non-English speech instruction following. The claim of comparable speech-text performance is stated without qualification in the abstract and conclusion.

No Evaluation of Streaming-Specific Degradation — Quality Under Latency Constraints Is Unknown

The assumption or constraint. The streaming architecture imposes several constraints that could degrade model quality compared to offline (full-context) processing: (1) the audio encoder uses block-wise attention in 2-second chunks rather than full attention over the entire audio input, which could miss long-range dependencies (e.g., a speaker's tone or topic established in the first 10 seconds that provides context for understanding a later utterance); (2) the vision encoder merges adjacent 2×2 tokens, reducing spatial resolution by 4×, which could impact fine-grained visual understanding; (3) Talker generates speech tokens before the complete text response is known, relying on Thinker's hidden states to anticipate prosody and emotion — this anticipatory generation could produce unnatural prosody if the later portion of the text response changes the appropriate intonation (e.g., a sentence that begins as a statement but ends as a question); (4) the sliding-window DiT restricts its receptive field to 4 blocks (2 lookback + 1 lookahead), which could reduce speech naturalness by limiting the acoustic context available for prosody modeling and coarticulation. The paper evaluates the model on standard benchmarks but does not report whether these evaluations were performed under streaming constraints (with block-wise encoders, restricted DiT context, and anticipatory Talker generation) or under offline, full-context conditions.

The consequence. A practitioner cannot determine whether the reported benchmark scores represent the model's performance in actual streaming deployment or an idealized offline configuration. If the evaluations used full-context processing (which is typical for benchmark evaluation — the full audio/video is available before inference begins), then the reported numbers overestimate real-world streaming performance. The degradation from streaming constraints could be substantial: long audio understanding tasks (e.g., transcribing a 30-minute lecture) might suffer from the 2-second chunk limitation if cross-chunk context is needed; video reasoning tasks might be impaired if the 2×2 token merging loses fine details; speech naturalness might degrade if the DiT's restricted receptive field causes discontinuities at chunk boundaries. None of these failure modes can be anticipated or quantified from the paper's evaluation.

What evidence exists in the paper. No streaming-specific evaluation is reported. All benchmarks in Tables 1–10 are standard offline benchmarks (MMLU, GSM8K, OmniBench, SEED, etc.) that do not test streaming-specific quality attributes like chunk-boundary artifacts, latency-constrained prosody naturalness, or partial-input understanding accuracy. The paper provides no comparison between streaming and offline configurations of the same model. There is no evaluation of how performance varies with chunk size, DiT receptive field size, or the amount of "lookahead" text available to Talker during anticipatory speech generation.

Mitigation status. Not addressed. The paper's Section 2.4 describes the streaming design choices and their latency motivation, but Section 5 (Evaluation) makes no connection between these design choices and the evaluation protocol. The paper does not specify whether evaluations used streaming or offline inference, and does not acknowledge the absence of streaming-specific quality measurements as a limitation.

Training Data Curation Details Are Insufficient for Reproduction

The assumption or constraint. The paper reports aggregate data quantities — 800 billion tokens of image/video data, 300 billion tokens of audio data, 100 billion tokens of video-with-audio data, plus unspecified text data in stage 2 pre-training (Section 3) — but provides almost no detail about the composition, sourcing, filtering, or quality control of this data. Key information that is missing includes: the sources of the video-with-audio data (which is critical for TMRoPE and interleaving), the language distribution of the audio data (which determines the speech understanding quality across languages, as discussed above), the methods used to filter out low-quality or misaligned video-audio pairs, the speaker diversity in the speech data used for Talker training, the specific tasks and datasets included in the "wider variety of tasks" mentioned in stage 2, and the composition of the "pure text data" used to maintain language proficiency. The Talker training data description is particularly sparse: the DPO preference pairs are constructed using WER and punctuation pause error rate as automated reward signals, but the dataset size, the threshold for winning vs. losing samples, the β parameter in the DPO objective (Equation 1), and the number of preference pairs are not specified. The multi-speaker fine-tuning (stage 3) evaluates four named speakers (A, B, C, D) but provides no information about these speakers, the amount of fine-tuning data per speaker, or the recording conditions.

The consequence. The model cannot be reproduced from the paper's description alone. A practitioner attempting to replicate Qwen2.5-Omni or adapt its architecture to a different base model would need to make uninformed guesses about data composition, curation, and training hyperparameters that could substantially affect performance. The data mixture ratios (800B image/video : 300B audio : 100B video-audio) are reported without justification, so a practitioner cannot know whether these ratios are carefully tuned or arbitrary, or how sensitive performance is to variations. The DPO training for Talker stabilization is a key component of the speech quality results, but without the β value and dataset construction details, the RL stage is essentially a black box. The speaker fine-tuning results (Table 10) showing near-human naturalness (NMOS 4.46–4.62) cannot be contextualized without knowing the speakers, the amount of adaptation data, or the evaluation protocol.

What evidence exists in the paper. Section 3 provides high-level data quantities and a brief description of the three-stage curriculum. Section 4.3 describes the Talker training stages but omits hyperparameters and dataset details. The evaluation (Section 5) uses standard benchmarks with publicly available test sets, but the training data that produced the model weights is not specified at a level that enables reproduction. The paper does not claim to release the training data, but the description is insufficient even for understanding the data engineering choices that underpin the model's performance.

Mitigation status. Partially addressed by context. The paper is a technical report from the Qwen team, which has previously released model weights (Qwen2.5, Qwen2.5-VL, Qwen2-Audio) along with more detailed technical documentation. Practitioners can potentially infer some data practices from prior Qwen papers. However, the specific data mixtures, the video-with-audio data sources, and the Talker training dataset construction are new to this model and are not documented in prior work. The paper does not state whether training data details will be released in a separate datasheet or model card.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around multimodal AI from modality-specialized models that are stitched together in post-processing toward end-to-end unified systems where cross-modal integration is a first-class architectural consideration. Before this work, the dominant paradigm for building multimodal assistants was composition: take a vision-language model (Qwen2.5-VL), an audio-language model (Qwen2-Audio), and a text-to-speech system (CosyVoice 2), and chain them together. Each component was trained independently, optimized for its own modality-specific objective, and connected through a narrow interface — typically text transcripts passed between ASR and LLM, then text output passed to TTS. This modular approach worked, but it introduced irreducible latency (each stage buffers its complete output before the next stage begins), error propagation (an ASR mistake becomes an LLM reasoning error that TTS faithfully vocalizes), and — most critically — the loss of cross-modal context at every interface boundary.

Qwen2.5-Omni reframes the problem. The paper's architecture argues, by its design, that multimodal understanding is not the sum of unimodal understandings — that a video's visual content and its audio track must be processed jointly, with explicit temporal synchronization, for the model to learn genuine cross-modal associations. The 56.13% result on OmniBench versus the next-best model's 42.9% (Table 8) is not an incremental improvement; it is a 13.2-percentage-point gap that suggests prior omni-models, even large ones like Gemini-1.5-Pro, were fundamentally limited by their lack of temporal cross-modal alignment. This is the paper's most disruptive empirical finding: scale alone cannot substitute for architectural support for cross-modal synchronization. If simply training a larger model on more multimodal data were sufficient, Gemini-1.5-Pro would not trail by 13 points. Something structural — the paper argues it is TMRoPE's time-calibrated position encoding and the 2-second interleaving — is necessary.

The second landscape shift concerns speech generation as an integrated capability rather than a downstream post-processing step. The paper's results on the speech-text instruction-following gap (Table 4) expose a failure mode that the composable-pipeline paradigm had normalized: Qwen2-Audio with speech input achieved only 18.4% on GSM8K versus 82.3% for text input — a gap so large that voice interaction was essentially a different, much dumber model than text interaction. The field had largely accepted this as inevitable, attributing it to ASR errors cascading into the LLM. Qwen2.5-Omni's 85.4% on GSM8K with speech input — slightly outperforming the text baseline — refutes the inevitability of this gap. The implication is that speech understanding degrades reasoning primarily because of information loss at the ASR→LLM interface, not because speech itself is inherently harder to reason about. When the audio encoder is trained end-to-end with the LLM on reasoning tasks, it can learn to preserve the structural features (mathematical notation, logical connectors, formatting cues) that a standalone ASR system strips away.

The third shift is methodological: the paper introduces fully automated, objective reward signals for preference optimization of speech generation via DPO with WER-based rankings. Text RLHF has relied on human preference judgments because text quality is inherently subjective and multidimensional. Speech generation, the paper shows, has a different property — key quality dimensions (word accuracy, pause appropriateness) are objectively measurable through ASR and timing analysis. This means the DPO training loop for Talker (Section 4.3) can be scaled without human annotation, using only automated metrics to construct preference pairs. The SEED test-hard improvement from ICL to RL (7.97% → 6.54% WER, Table 9) demonstrates that this signal is sufficient to drive meaningful quality improvements. This opens the door to iterative, self-improving speech generation where the model generates speech, an ASR system scores it, and the model is updated to prefer higher-scoring outputs — a closed loop that requires no human intervention.

The paper also resolves a tension in the multimodal literature between specialization and unification. Prior to this work, a reasonable hypothesis was that modality-specific models would always outperform unified models because they could optimize their architectures and training objectives for a single modality without compromise. Qwen2.5-Omni demonstrates the opposite on several fronts: it matches Qwen2.5-VL-7B on image understanding within 1–2 percentage points (Table 5), it outperforms Qwen2-Audio on most audio benchmarks (Tables 2–3), and it achieves speech generation quality competitive with CosyVoice 2, a dedicated TTS system (Table 9). The unification does not come at the cost of individual modality performance — and on genuinely cross-modal tasks (OmniBench), it provides gains that no specialist can achieve. This reframes the unification-vs-specialization debate from "can we avoid performance loss?" to "can we achieve synergistic gains?" — a much higher bar that Qwen2.5-Omni clears on OmniBench specifically.

Finally, the paper makes verifier robustness for speech generation a newly visible research problem. The DPO stage's improvement on SEED test-hard (17.9% relative WER reduction) while preserving speaker similarity demonstrates that the base speech model suffers from a specific class of errors — "model hallucinations" in the paper's terminology — that can be addressed through preference optimization with objective metrics. This is directly analogous to how the RLHF community recognized reward hacking as a central challenge for text generation, and it positions speech generation stability as a research problem in its own right with a concrete, measurable objective function (WER) and a demonstrated optimization method (DPO).

Follow-Up Research This Work Enables

TMRoPE ablation and generalization to other position encoding schemes. The paper presents TMRoPE as a unified design but provides no controlled experiment isolating its contribution. A direct follow-up would train two variants of Qwen2.5-Omni — one with TMRoPE (time-calibrated temporal IDs, 2-second interleaving) and one with standard M-RoPE (incrementing temporal IDs without time calibration, simple concatenation of modalities) — using identical data and training recipes, and evaluate on OmniBench, AV-Odyssey Bench, and a newly constructed diagnostic benchmark that specifically tests temporal synchronization (e.g., "did the sound of the door slam occur before or after the person entered the frame?"). If TMRoPE provides no benefit over M-RoPE at equal data scale, the paper's central design claim is weakened and the field learns that data volume, not positional encoding, drives cross-modal integration. If TMRoPE provides a substantial benefit, the experiment quantifies the value of temporal calibration and establishes it as a design principle for future multimodal architectures. A stronger test would implement TMRoPE-style temporal calibration in a non-Qwen architecture (e.g., LLaMA-based omni-model) to test whether the benefit transfers across model families.

Streaming quality degradation measurement. The paper's streaming architecture is evaluated only on offline benchmarks; no experiment measures how the streaming constraints (block-wise audio attention, restricted DiT receptive field, anticipatory Talker generation) affect quality. A rigorous follow-up would evaluate Qwen2.5-Omni in three configurations on the same benchmarks: (1) full offline mode (full-attention audio encoder, complete-text Talker generation, full-context DiT), (2) streaming mode with the paper's design choices (2-second audio blocks, token-by-token Talker pipelining, 4-block DiT window), and (3) intermediate configurations that vary chunk size and DiT receptive field. The evaluation would measure both accuracy (standard benchmarks) and latency (time-to-first-audio, end-to-end response time on standardized hardware). The hypothesis is that streaming constraints degrade quality, but the degradation should be small enough that the latency benefit justifies it. If streaming WER is, say, 2× higher than offline WER, or OmniBench accuracy drops by 10+ points, the architecture's core claim — that it enables real-time interaction without sacrificing quality — is undermined. Such a study would also establish the latency-quality Pareto frontier, showing practitioners what latency targets are achievable at what quality levels, and would identify which specific streaming constraint (audio chunking, Talker anticipation, or DiT windowing) is the primary bottleneck.

Cascaded pipeline versus end-to-end comparison on voice interaction tasks. The paper motivates end-to-end unification by arguing that composable pipelines lose cross-modal context, but it never directly compares Qwen2.5-Omni against a pipeline of equivalently-capable components. A strong comparative study would construct a cascaded system using Whisper-large-v3 for ASR, Qwen2.5-7B (or Qwen2.5-VL-7B for visual tasks) as the reasoning engine, and CosyVoice 2 for TTS, and evaluate both the pipeline and Qwen2.5-Omni on end-to-end voice interaction benchmarks (VoiceBench, the paper's in-house voice-chat benchmark, and a newly constructed task requiring joint audio-visual reasoning with spoken responses). Key metrics would include end-to-end accuracy, response latency (wall-clock time from end of user speech to beginning of system speech), and user-rated naturalness. The pipeline has the advantage of using specialized, potentially stronger components (Whisper-large-v3 beats Qwen2.5-Omni on some ASR benchmarks; CosyVoice 2 is a dedicated TTS system); the unified model has the advantage of shared context and no interface bottlenecks. This comparison would quantify the end-to-end benefit of unification and identify which components of the pipeline are the weak links — providing a roadmap for where unification matters most versus where modular composition suffices. If the pipeline matches or exceeds Qwen2.5-Omni on all metrics, the case for end-to-end omni-models weakens substantially.

Diagnosing the Chinese-English speech understanding disparity. Table 4 reveals a large and unexplained gap: speech-input CEval drops 17.3 points relative to text input, while speech-input MMLU drops only 3.7 points. A diagnostic study would systematically evaluate speech understanding across languages and task types to isolate the cause. The experiment would test three hypotheses: (1) the audio encoder (Whisper-large-v3 initialization) has worse Chinese ASR accuracy, causing information loss at the encoding stage — testable by measuring WER of the encoder's implicit transcription on Chinese and English benchmarks; (2) the training data is English-skewed, so the model has less exposure to Chinese speech-reasoning pairings — testable by fine-tuning on additional Chinese speech data and measuring the gap reduction; (3) Chinese's tonal nature makes speech-to-reasoning transfer inherently harder because prosodic features carry lexical information that must be preserved through the encoder bottleneck — testable by comparing the speech-text gap on tonal vs. non-tonal languages in the same model. Understanding the cause is essential for practitioners deploying in non-English markets and would inform whether the fix is data augmentation, encoder architecture changes, or language-specific fine-tuning.

DPO reward signal design space for speech generation. The paper uses WER and punctuation pause error rate as automated reward signals for DPO, achieving a 17.9% relative WER reduction on SEED test-hard (Table 9). This opens a design space that the paper does not explore: what other objective metrics could serve as reward signals, and how do they interact? A systematic study would train Talker variants using DPO with different reward combinations: WER alone, speaker similarity (SIM) alone, MOSNet-predicted naturalness scores, composite rewards combining multiple metrics, and human preference judgments as an upper bound. The evaluation would measure not only the targeted metrics (e.g., does SIM-based DPO improve speaker similarity?) but also cross-metric effects (does WER-based DPO inadvertently reduce speaker similarity or naturalness?). The paper's results suggest cross-metric effects are minimal — WER improved without SIM degradation — but a broader study would establish whether this holds across the reward space or whether there are tradeoffs (e.g., optimizing for naturalness might increase WER by encouraging more expressive but less precise pronunciation). This would produce a reward design methodology for speech DPO that the community can adopt, analogous to how the text RLHF community has developed best practices around reward model training and KL penalty tuning.

Extension to generated visual and musical outputs. The paper's conclusion explicitly names "expanded output capabilities across various modalities like images, videos, and music" as a future goal. The Thinker-Talker architecture provides a template: Thinker generates high-level semantic representations, and modality-specific "Talker" modules convert those representations into output tokens. Extending this to visual generation would require a "Painter" module — a diffusion-based decoder that takes Thinker's hidden states as conditioning and generates image patches autoregressively or via parallel decoding, analogous to how Talker generates speech tokens. The key research question is whether the same Thinker hidden states that encode information sufficient for text and speech generation also contain the spatial and compositional information needed for image generation, or whether the hidden representation bottleneck needs to be widened for visual outputs. A minimal viable experiment would add a lightweight image decoder (similar to the vision encoder in reverse) to a frozen Qwen2.5-Omni and fine-tune only the decoder on image-captioning data, measuring whether the Thinker representations can drive reasonable image generation without degrading text or speech quality. Success would demonstrate the generality of the Thinker-as-universal-interface architecture; failure would indicate that different output modalities require different representational spaces, motivating a multi-head Thinker design.

Practical Applications and Downstream Use Cases

Real-time voice assistants with visual context awareness. Qwen2.5-Omni enables a class of application that current voice assistants (Siri, Alexa, Google Assistant) cannot handle: conversations where the assistant can see what the user sees. A user could point their phone camera at a malfunctioning appliance and say "what's wrong with this and how do I fix it?" — the model processes the video feed (visual inspection of the appliance), the audio feed (the spoken question), and generates both a text explanation and spoken instructions. The critical enabler is the streaming architecture: the model can begin processing video frames and audio chunks before the user finishes speaking, and can begin verbalizing the response before the full text is generated. No existing deployed system offers this capability — current voice assistants are audio-only, and current vision-language models are text-output-only. The paper's results suggest this is practically viable: the model achieves 72.4% on Video-MME with subtitles (Table 7), 74.12 on VoiceBench (Table 3), and speech generation with 1.42% WER (Table 9), approaching the quality thresholds needed for consumer deployment. The remaining gap is latency — without published latency numbers, it's unclear whether the 7B model can process video and generate speech fast enough on-device or whether cloud inference is required, with its attendant privacy and connectivity concerns.

Accessibility tools combining scene description with spoken interaction. For blind or visually impaired users, the ideal assistive device would continuously describe the visual environment while engaging in natural spoken dialogue — answering questions about specific objects, reading text aloud, and describing people's expressions and actions, all in real time. Qwen2.5-Omni's combination of strong visual grounding (90.5% on RefCOCO val, Table 6), video understanding (68.6% on EgoSchema, Table 7), and natural speech generation makes this feasible in a single model. The key advantage over current solutions (which typically chain an image captioning model with a TTS system) is that the user can ask follow-up questions and receive spoken answers without the model "forgetting" the visual context — the entire interaction history, including visual tokens, is maintained in Thinker's context window. The streaming speech generation capability is essential here because users need immediate audio feedback to navigate their environment; waiting 2–3 seconds for a TTS system to synthesize a response from complete text would be disorienting. The paper's demonstration that speech instruction following nearly matches text performance (85.4% vs. 88.7% on GSM8K, Tables 1 and 4) is particularly relevant — it means users can ask complex questions verbally without suffering the dramatic accuracy degradation that characterized prior audio-language models.

Multimodal content moderation and analysis at scale. Platforms that host user-generated video content (YouTube, TikTok, educational platforms) need to analyze both the visual and audio content of videos for policy compliance, content categorization, accessibility (captioning, audio description), and recommendation. Current pipelines use separate models for visual analysis (object detection, scene classification), audio analysis (speech recognition, music detection, sound event classification), and text analysis (caption generation, content summarization) — a complex, expensive, and failure-prone architecture where errors in any component cascade. Qwen2.5-Omni offers a unified alternative: a single model processes the video stream with synchronized audio and visual understanding via TMRoPE, generating text descriptions, content labels, and captions in one forward pass. The OmniBench results (56.13%, Table 8) indicate the model can jointly reason about speech, sound events, and music in video context — precisely the capability needed for content moderation decisions like "does this video contain hate speech synchronized with violent imagery?" The economic case depends on inference cost relative to the composable pipeline, which the paper does not quantify, but the architectural simplicity of deploying and maintaining one model versus three to five coordinated models is a substantial operational advantage.

Self-improving speech interaction through automated RL loops. The paper's DPO training for Talker (Section 4.3) uses fully automated reward signals (WER, pause error rate) that require no human annotation. This enables a deployment scenario where the model continuously improves its speech generation quality based on real user interactions. The loop would work as follows: (1) a user asks a question and receives a spoken response, (2) an ASR system (which could be Qwen2.5-Omni itself in "listening mode") transcribes the generated speech, (3) the transcribed text is compared to the original text response that Thinker generated, producing a WER score, (4) interactions with high WER (indicating pronunciation errors, garbled speech, or timing issues) are flagged, (5) periodically, the model is fine-tuned using DPO with the high-WER generations as losing samples and reference-quality generations (or the original Thinker text) as winning samples. This creates a virtuous cycle where the model's speech quality improves over time from real deployment data, with no human annotation cost. The paper's demonstration that WER-based DPO works (7.97% → 6.54% on SEED test-hard) proves the concept; the practical question is whether the improvement saturates after one round or continues with iterative refinement, and whether the ASR system used for self-scoring is sufficiently accurate to provide reliable training signal. A production deployment would need to monitor for unintended degradation — the DPO might optimize for ASR-recognizability at the expense of naturalness, producing speech that is easy to transcribe but sounds robotic — which would require periodically injecting human naturalness evaluations into the loop.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Qwen2-Audio Technical Report2024· multimodal← LLM-judged similar
- InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output2024· multimodal← in-domain predecessor
- Qwen2 Technical Report2024· llm-systems← topic thread
- ERNIE 5.0 Technical Report2026· multimodal← similar topic
- GLM-OCR Technical Report2026· multimodal← similar topic
- EXAONE 4.5 Technical Report2026· multimodal← similar topic
- Qwen3.5-Omni Technical Report2026· multimodal← similar topic
- Qwen2.5-VL Technical Report2025· multimodal← similar topic
GPT-4o System Card2024· safety← LLM-judged similar
- Undesirable Memorization in Large Language Models: A Survey2024· safety← in-domain predecessor
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT2024· retrieval← topic thread
Gemini Robotics: Bringing AI into the Physical World2025· multimodal← in-domain predecessor
- Gemini Embedding: Generalizable Embeddings from Gemini2025· llm-systems← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer2025· multimodal→ LLM-judged similar
- Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding2025· multimodal→ in-domain predecessor
- ALIGN: Large-Scale Image-Text Representation Learning with Noisy Text Supervision2021· multimodal→ similar topic
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM2025· multimodal→ LLM-judged similar
- RL makes MLLMs see better than SFT2025· multimodal→ in-domain predecessor
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models2023· multimodal→ similar topic
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources2025· multimodal→ in-domain predecessor
- LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training2025· multimodal→ LLM-judged similar
- SmolVLM: Redefining small and efficient multimodal models2025· multimodal→ in-domain predecessor
- Phi-4-reasoning Technical Report2025· reasoning→ topic thread
- NVLM: Open Frontier-Class Multimodal LLMs2024· multimodal→ similar topic
- Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs2025· multimodal→ similar topic
PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs2025· pretraining→ topic thread
- Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training2025· pretraining→ in-domain predecessor
Qwen2-Audio Technical Report2024· multimodal→ similar topic
- LLaVA-OneVision: Easy Visual Task Transfer2024· multimodal→ in-domain predecessor
- Gemma 2: Improving Open Language Models at a Practical Size2024· pretraining→ topic thread
- Qwen2.5-VL Technical Report2025· multimodal→ similar topic
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs2025· multimodal→ similar topic
Qwen3.5-Omni Technical Report2026· multimodal→ similar topic
- Image Generators are Generalist Vision Learners2026· multimodal→ in-domain predecessor