Qwen3.5-Omni Technical Report

ArXiv: 2604.15804

🎯 Pitch

Qwen3.5-Omni achieves state-of-the-art results across 215 audio and audio-visual benchmarks while introducing a striking emergent capability—Audio-Visual Vibe Coding, where the model directly writes code from spoken and visual instructions. By scaling to hundreds of billions of parameters and training on over 100 million hours of audio-visual content, it matches Gemini-3.1 Pro on comprehensive understanding. A new streaming alignment technique, ARIA, eliminates the robotic instability that has plagued conversational speech synthesis, enabling natural, emotionally nuanced voice generation across 10 languages.

1. Executive Summary

This technical report introduces Qwen3.5-Omni, a fully omnimodal large language model built on the Thinker–Talker architecture that unifies understanding, reasoning, and generation across text, images, audio, and audio-visual inputs. The model introduces a Hybrid Attention Mixture-of-Experts (MoE) framework for both Thinker and Talker components, a 256k-token long-context modeling capability (supporting over 10 hours of audio and 400 seconds of 720P video at 1 FPS), and ARIA (Adaptive Rate Interleave Alignment) — a technique that dynamically aligns text and speech units during streaming to mitigate instability caused by mismatched tokenization rates between text and speech tokenizers (e.g., reducing skipped words and incorrect pronunciations in conversational speech). Qwen3.5-Omni-Plus achieves state-of-the-art results across 215 audio and audio-visual understanding, reasoning, and interaction subtasks, surpassing Gemini-3.1 Pro on general audio understanding and matching it on comprehensive audio-visual understanding, while demonstrating strong zero-shot cross-lingual speech generation that reduces error rates by approximately 72% on zh-to-ko relative to CosyVoice3 — establishing that scaling native omnimodal training with novel streaming alignment can produce unified systems that preserve text and vision capabilities equivalent to same-scale single-modality models only when the model architecture explicitly addresses encoder-decoder rate mismatches through alignment mechanisms like ARIA.

2. Context and Motivation

The Core Problem: Building AI Systems That Interact Like Humans — But Current Models Can't

The fundamental problem Qwen3.5-Omni addresses is deceptively simple to state but extraordinarily difficult to solve: how do we build an AI system that can perceive, reason about, and respond to the world in the same rich, multimodal way that humans do? Humans don't process text, images, audio, and video through separate channels — we integrate them seamlessly, using visual cues to disambiguate speech, auditory information to contextualize what we see, and linguistic understanding to reason across both. When we converse, we don't just produce text; we speak with appropriate prosody, emotion, and timing. When we see someone point at something while speaking, we don't process the gesture and the words independently — we fuse them.

The paper frames this explicitly in its opening sentence: "Human interaction with the world is inherently omnimodal and agentic, involving the integration of visual, auditory, and linguistic information, and the production of responses through text, speech, and goal-directed tool-mediated actions." Yet as of early 2026, the AI systems most commonly deployed — even the most powerful ones — are not omnimodal. They are either text-only LLMs with bolted-on modality adapters, separate vision-language models that can't handle audio, or speech systems that transcribe to text and then generate text responses independently. The integration is fragmented, and the capability gaps that result from this fragmentation are both practically important and theoretically revealing.

Why This Matters: Three Real-World Failure Modes

The paper's motivation crystallizes around three concrete failure modes in current systems, each with significant practical implications:

1. Passive perception without agency. Existing multimodal models "predominantly operate within passive perception-response paradigms" (Section 1). They can describe what they see or hear, but they cannot act — autonomously invoking tools, searching the web for real-time information, or writing and executing code in response to sensory input. This matters enormously for practical deployment. A home robot that can describe a spilled liquid but cannot autonomously look up cleaning instructions and plan a response is a toy, not a tool. A customer service system that can understand a user's complaint but cannot access their account history or process a refund is a chatbot, not an agent. The gap between perception and action is where most real-world value lies, and prior systems largely left this gap unfilled.

2. Real-time interaction that feels unnatural or is impossible. Speech interaction imposes hard real-time constraints: users expect responses to begin within a few hundred milliseconds, not seconds. Yet prior speech-enabled LLMs often suffered from instability and unnaturalness in streaming speech synthesis, which the paper attributes to a specific architectural problem: "encoding efficiency discrepancies between text and speech tokenizers" (Abstract). Text tokens represent words or subwords at a relatively consistent rate per unit of semantic content, while speech codec tokens must represent detailed acoustic features — pitch, timbre, rhythm — at a much higher and more variable rate per unit of content. When a model generates text and speech tokens in parallel streams (as in the dual-track Talker design of Qwen3-Omni), these mismatched rates create alignment failures: words get skipped, pronunciations become garbled, numbers are rendered ambiguously, and prosody breaks down, especially in languages with complex phonology or morphology. These aren't minor quality issues — they make systems unusable for extended conversation.

3. Scaling multimodal models without degrading text capabilities. A persistent challenge in the field is that training models on multimodal data often degrades their text-only performance — the infamous "forgetting" problem where an LLM fine-tuned for vision or audio tasks loses some of its reasoning, coding, or knowledge capabilities. The paper explicitly claims that Qwen3.5-Omni "maintains state-of-the-art performance on text and visual modalities without degradation relative to same-size single-model Qwen counterparts," and Section 5.1.1 provides data showing Qwen3.5-Omni-Plus achieving 85.9 on MMLU-Pro versus 86.8 for Qwen3.5-Plus-Instruct — a gap of less than 1 percentage point (Table 4). This matters because if adding multimodal capability required sacrificing core language skills, practitioners would face an impossible tradeoff. The paper's approach suggests this tradeoff is not fundamental but an artifact of training methodology.

Prior Approaches and Where They Fall Short

The pre-Qwen omni landscape. The paper positions itself against a backdrop of rapid but fragmented progress. By early 2025, the field had produced strong systems in individual modalities — Qwen3.5 for text (Team, 2026), Qwen2.5-VL for vision (Bai et al., 2025a), Qwen-Audio and Qwen2-Audio for audio understanding (Chu et al., 2023; 2024) — and several attempts at unified multimodal systems, most notably OpenAI's GPT-4o (2024), Google's Gemini series (Gemini Team, 2024; Comanici et al., 2025), and the Qwen-Omni line itself with Qwen2.5-Omni (Xu et al., 2025a) and Qwen3-Omni (Xu et al., 2025b). These systems demonstrated that omnimodal architectures were viable, but each had specific, well-documented limitations that Qwen3.5-Omni explicitly targets.

Limitation 1: The tradeoff between model capacity and inference efficiency. The Thinker–Talker architecture introduced in Qwen2.5-Omni established a clean separation: the Thinker handles understanding and text generation, and the Talker handles speech synthesis, receiving high-level representations from the Thinker. This separation was architecturally elegant but computationally expensive when both components used dense transformer architectures. As the paper notes (Section 2.1, bullet 1), Qwen3.5-Omni transitions both the Thinker and the Talker to a Hybrid Attention Mixture-of-Experts (MoE) design — not just one component, but both. This isn't a minor tweak; it's a fundamental capacity-efficiency tradeoff resolution. MoE architectures activate only a fraction of parameters per token, meaning a model can have hundreds of billions of total parameters (enabling broad knowledge and capability) while maintaining inference costs closer to a much smaller dense model. The paper doesn't explicitly state the parameter counts, but the "Plus" and "Flash" variants clearly occupy different points on this capacity-efficiency curve.

Limitation 2: Context length too short for real-world audio-visual content. Prior omni models — including Qwen3-Omni — supported context lengths in the tens of thousands of tokens. While adequate for text documents, this is severely limiting for audio-visual content. The paper provides concrete numbers (Section 2.1, bullet 2): Qwen3.5-Omni supports 256k tokens, enabling more than 10 hours of audio or 400 seconds of 720P video at 1 FPS. To put this in perspective, a 10-hour podcast, a multi-hour meeting, or a feature-length movie with descriptive audio all fit within a single context window. This isn't just a marketing number — it enables use cases that were structurally impossible before. You cannot summarize a full workday of meetings if your model can only hold 30 minutes of audio in context.

Limitation 3: Speech generation instability from rate mismatch. This is the problem that ARIA specifically addresses, and it's worth understanding the root cause in detail. In Qwen3-Omni's Talker design, text tokens and speech tokens were generated through dual tracks — essentially two parallel decoding streams. The text track produced discrete tokens like "The weather is nice today" while the speech track simultaneously produced RVQ (Residual Vector Quantization) codec tokens encoding the acoustic realization of those words. The problem is that a single text token (representing perhaps one word or subword) might correspond to dozens or even hundreds of speech codec frames, and this ratio varies dramatically across languages, speakers, and speaking rates. Without explicit alignment, the dual-track system would frequently drift: the speech track might "get ahead" and produce audio for words the text track hasn't generated yet, or "fall behind" and produce silence or filler while waiting. The result — skipped words, stuttering, incorrect pronunciation, and robotic prosody — was especially severe for languages with rich morphology or complex phonotactics, where the text-to-speech token ratio is less predictable.

The paper doesn't just claim this is a problem; it demonstrates it through comparison. Section 2.4 notes that Qwen3.5-Omni's Talker "adopts ARIA to dynamically align text and speech units before interleaving them," and Section 2.5 contrasts this with Qwen3-Omni's dual-track approach. The latency comparison in Table 2 shows that even with ARIA's alignment mechanism, the Talker maintains TTFC (Time-To-First-Chunk for audio) of 54-95ms for Plus and 56-81ms for Flash across concurrency levels — meaning the alignment doesn't introduce meaningful latency overhead.

Limitation 4: Long-sequence temporal perception degrades with sparse position IDs. This is a subtle but important architectural insight that the paper identifies in Qwen3-Omni's use of TMRoPE (Section 2.3 and Section 3). TMRoPE assigned temporal position IDs by mapping absolute time to integer positions — for example, a video frame at t=100 seconds might receive position ID 100 (if using per-second discretization). For long audio-video inputs, this produces excessively large and sparse temporal position IDs that undermine the model's ability to capture long-range temporal dependencies. The positions are sparse because neighboring video frames might be separated by large gaps in position space if the sampling rate is low, and they're large because encoding absolute seconds produces large integers that the position encoding wasn't designed to handle efficiently. Additionally, effective learning under TMRoPE required "large-scale and uniformly distributed sampling across different frame rates," which increased data construction costs.

Limitation 5: Fixed interleaving rates in speech generation cannot handle diverse languages. Beyond the dual-track problem, Qwen3-Omni's approach to text-speech interleaving used either MFA (Montreal Forced Aligner)-derived alignments or fixed interleaving rates. MFA requires language-specific acoustic models and phoneme inventories — it works for high-resource languages but is impractical for the 36 languages of speech output that Qwen3.5-Omni supports. Fixed rates — say, interleaving one text token every 8 speech frames — assume a constant text-to-speech token ratio across all languages and speakers, which is demonstrably false. ARIA replaces both with a simpler, more general constraint: "for any prefix of the generated sequence, the cumulative speech-to-text token ratio must not exceed the corresponding item-level global ratio" (Section 2.4). This is essentially a monotonic streaming constraint that allows flexible local variation while preventing global drift, and it works without language-specific aligners.

Limitation 6: Omnipotent understanding without omnipotent action. Even models that achieved strong multimodal understanding — like GPT-4o or Gemini-2.5 — largely stopped at generating text responses. They couldn't autonomously invoke tools, search the web, execute function calls, or generate code in response to audio-visual instructions. Qwen3.5-Omni explicitly positions itself as a "native omni agent model" that "not only perceives and reasons across all modalities, but also acts" (Section 1). The practical implications include WebSearch for real-time information retrieval, FunctionCall for API-based tool use, and what the paper calls Audio-Visual Vibe Coding — an emergent capability where the model "directly generates executable code from audio-visual instructions" (Section 1). This last capability is particularly striking because it represents a qualitatively new behavior that wasn't explicitly trained for but emerged from the combination of omnimodal understanding and code generation capabilities.

How This Paper Positions Itself

The paper does not frame Qwen3.5-Omni as a revolutionary departure but as a systematic evolution that addresses five specific, well-characterized limitations of its predecessor, Qwen3-Omni. The five key technical upgrades listed in Section 1 — (1) Hybrid MoE for both Thinker and Talker, (2) 256k long-context modeling, (3) multi-codebook speech representation for single-frame synthesis, (4) ARIA for dynamic text-speech alignment, and (5) massively expanded multilingual training — each directly maps to one of the limitations discussed above.

Two aspects of this positioning are worth highlighting:

First, the paper targets practical deployability, not just benchmark scores. The extensive latency analysis in Section 2.5 (Tables 1 and 2) is unusual for a technical report — it reports not just standard metrics but detailed per-component latency breakdowns (Thinker TTFT, Talker TTFC, per-token latencies, generation throughput, and real-time factor) across three concurrency levels (1, 4, and 8 concurrent streams). The paper explicitly frames these numbers in terms of serving costs and user experience: "In streaming audio-visual interaction scenarios, the first-packet latency is a critical factor affecting user experience, and the model's concurrency capability is key to reducing service costs and improving response speed." The Plus variant achieves 435ms first-packet latency for audio input and 651ms for video input at concurrency 1, with the Flash variant cutting these roughly in half. These are real numbers that practitioners can use to estimate deployment costs.

Second, the paper implicitly argues that omnimodal training is complementary to, not competitive with, modality-specific capabilities. Table 6 shows Qwen3.5-Omni-Plus achieving vision-to-text scores that are consistently within 1-2 percentage points of Qwen3.5-Plus-Instruct across STEM, general VQA, OCR, spatial intelligence, and video understanding tasks — and in some cases (RealWorldQA: 84.1 vs 79.1; MLVU: 86.8 vs 85.1; VideoMME: 81.9 vs 81.0) actually exceeding the text-only model. This is a strong empirical claim that multimodal training doesn't inherently degrade modality-specific performance, provided the architecture and training recipe are designed correctly. The three-stage training pipeline (Section 3) — encoder alignment, general multimodal training with 4 trillion tokens, and long-context fine-tuning — is presented as the key to achieving this parity.

The paper also positions against external competition, particularly Gemini-3.1 Pro. Table 5 shows Qwen3.5-Omni-Plus surpassing Gemini-3.1 Pro on MMAU (82.2 vs 81.1), MMSU (82.8 vs 81.3), RUL-MuchoMusic (72.4 vs 59.6), SongFormBench (81.1 vs 75.6 accuracy on HarmonixSet), VoiceBench (93.1 vs 88.9), and consistently lower WER across ASR benchmarks (Fleurs: 6.55 vs 7.32, LibriSpeech clean: 1.11 vs 3.36). For audio-visual understanding (Table 7), the picture is more nuanced — Qwen3.5-Omni-Plus leads on DailyOmni (84.6 vs 82.7) and Qualcomm IVD (68.5 vs 66.2) but trails on WorldSense (62.8 vs 65.5) and VideoMME with audio (83.7 vs 89.0). The paper is careful not to claim universal superiority, instead noting that performance "reaches the level of" Gemini-3.1 Pro (Abstract).

Finally, the paper makes a significant architecture-level claim about the necessity of explicit alignment mechanisms. ARIA is not presented as an incremental improvement over Qwen3-Omni's dual-track Talker but as a fundamentally different approach: "Rather than relying on MFA-derived alignments or fixed interleaving rates, ARIA enforces an adaptive rate constraint... Despite its simplicity, this design affords flexible text-speech alignment across languages, including those with relatively low encoding efficiency" (Section 2.4). The word "despite its simplicity" is telling — it suggests that the prior complexity (dual tracks, forced alignment, fixed rates) was unnecessary and possibly harmful, and that a simpler constraint-based approach is both more robust and more generalizable. This is a design philosophy claim as much as a technical one.

In summary, Qwen3.5-Omni positions itself as the answer to a specific, well-motivated question: given that we can now build models that understand multiple modalities, how do we make them efficient enough to deploy, natural enough to converse with, long-context enough to handle real-world content, and capable enough to act autonomously? The paper's answers — Hybrid MoE, ARIA, 256k context, and native agentic design — are presented not as isolated innovations but as an integrated solution to an integrated problem.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

Qwen3.5-Omni is an end-to-end AI system that can simultaneously hear, see, read, speak, and act — processing audio, video, images, and text as input and generating either text or natural-sounding speech as output, all while being able to autonomously invoke tools, search the web, or write code. The core problem it solves is that previous omnimodal models suffered from a fundamental mismatch between how fast text tokens and speech tokens need to be generated, causing stuttering, skipped words, and robotic prosody in real-time conversation; the solution is a unified streaming architecture with a dynamic alignment mechanism (ARIA) that lets text and speech generation proceed at their natural, different rates while keeping them synchronized, combined with a Mixture-of-Experts design that makes the entire system efficient enough to deploy at scale.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components connected in a pipeline with feedback:

1. Audio Transformer (AuT) — converts raw audio waveforms into compact 6.25 Hz token representations, handling up to 10 hours of continuous audio. It's trained from scratch on 40 million hours of audio-text data.

2. Vision Encoder (from Qwen3.5) — processes images and video frames into visual tokens, handling both static images and dynamic video at variable frame rates.

3. Text Tokenizer (Qwen3.5 tokenizer) — converts text into subword tokens using a 250k vocabulary byte-level BPE, improving encoding/decoding efficiency 10-60% across most languages.

4. Thinker (Hybrid MoE Transformer) — the "brain" that receives interleaved representations of text, audio tokens, and visual tokens as a unified sequence, performs understanding and reasoning, and generates text output tokens. It processes everything through a single large transformer with a 256k-token context window and uses explicit timestamps for temporal awareness.

5. Talker (Hybrid MoE Transformer + MTP module) — the "voice" that receives high-level representations from the Thinker (including the generated text and multimodal context) and produces streaming speech tokens using a multi-codebook prediction scheme. The ARIA mechanism interleaves text and speech tokens adaptively before the Talker processes them.

6. Code2Wav Renderer (causal ConvNet) — converts the Talker's predicted RVQ (Residual Vector Quantization) codec tokens into actual audio waveforms, operating frame-by-frame for streaming output.

Information flows as follows: raw input (text, audio, images, video) → modality-specific encoders produce token sequences → tokens are interleaved with explicit timestamps for temporal alignment → Thinker processes the unified 256k-token sequence and generates text → ARIA aligns the Thinker's text output with speech generation context → Talker predicts multi-codebook speech tokens autoregressively → Code2Wav renders the tokens to waveform incrementally. For real-time streaming, chunked prefilling allows the Thinker and Talker to begin generating before the full input is received.

3.3 Roadmap for the Deep Dive

• First, the Audio Transformer (AuT) — how raw audio is encoded, since it's the novel encoder trained from scratch and sets the temporal resolution that the rest of the system depends on.
• Second, how the Thinker integrates text, audio, images, and video into a unified representation — this is where the temporal alignment mechanism (replacing TMRoPE) and the 256k context handling live, and it's the foundation for all multimodal understanding.
• Third, the Talker's speech generation pipeline — from multi-codebook RVQ tokens through the MTP module to waveform rendering, since this is the generation counterpart that must run in streaming mode.
• Fourth, ARIA (Adaptive Rate Interleave Alignment) in detail — the paper's core architectural innovation that replaces the dual-track Talker design, since it's the mechanism that makes streaming speech natural and stable.
• Fifth, the streaming and concurrency optimizations — chunked prefilling, Hybrid MoE efficiency, and the latency architecture, since these determine whether the system is deployable.
• Sixth, the three-stage pretraining strategy — the data composition (4 trillion tokens across modalities) and the transition from 32k to 256k context, since this determines the model's multimodal capabilities.
• Seventh, the post-training pipeline — specialist distillation, on-policy distillation for audio quality, interaction-aligned RL, and the four-stage Talker training, since these are what bridge from pretrained capabilities to usable interaction quality.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems architecture paper with a central innovation — ARIA — whose core idea is that stable streaming speech synthesis requires dynamically aligning text and speech token generation rates rather than using fixed interleaving ratios or parallel dual-track decoding, and that this alignment can be achieved with a simple monotonic constraint rather than language-specific forced aligners.

Audio Transformer (AuT) — Audio Encoding

The AuT is the audio encoder that converts raw audio waveforms into compact, semantically meaningful token sequences at a consistent temporal resolution. It is a transformer-based encoder-decoder model trained from scratch specifically for Qwen3.5-Omni, not reused from a previous model.

Input preprocessing and architecture. The raw audio waveform is first resampled to 16 kHz (mono). Filter bank features are then extracted from the waveform and passed through a series of four Conv2D blocks that perform 16× temporal downsampling. The downsampled features are then fed into self-attention layers (the transformer encoder) to produce the final audio token sequence. The output token rate is 6.25 Hz, meaning each output token corresponds to approximately 160 ms of the original audio signal.

This 6.25 Hz rate is a critical design choice because it determines the temporal granularity at which the Thinker can access audio information. At 6.25 Hz, 10 hours of audio produces 10 × 3600 × 6.25 = 225,000 tokens, which fits within the 256k context window with room to spare for other modalities. A higher rate would provide finer temporal resolution but consume more of the context budget; a lower rate would be more efficient but lose temporal precision for tasks like speaker diarization, word-level alignment, or music analysis.

Training data and scale. The AuT encoder is trained on 40 million hours of audio-text paired data generated by Qwen3-ASR, a predecessor speech recognition model. This is a massive scale — 40 million hours is over 4,500 years of continuous audio — and it's critical because the encoder must learn robust representations that work across languages, acoustic conditions, and audio types (speech, music, environmental sounds). The paper notes that compared to Qwen3-Omni's encoder training, Qwen3.5-Omni's encoder "adapts more multilingual data of more than 20 languages," with the proportion of Chinese, English, and multilingual data set to 3.5 : 3.5 : 3 (Section 2.2). This balanced multilingual ratio is important because it prevents the encoder from overfitting to high-resource languages at the expense of low-resource ones.

Dynamic attention window training. A notable training detail: the AuT employs a "dynamic attention window size training mechanism" designed to ensure balanced performance between two deployment scenarios — real-time prefilling (where audio is processed in streaming chunks as it arrives, and the attention window grows with each new chunk) and offline audio understanding (where the full audio is available upfront and the attention window spans the entire sequence). Training with dynamic window sizes means the encoder learns to produce good representations regardless of how much context is available at inference time, which is essential for the streaming use case. Without this, an encoder trained only on full-sequence attention might perform poorly when forced to operate with limited lookahead during streaming.

Comparison with prior work. The paper explicitly contrasts this with Qwen3-Omni's encoder, which used less multilingual data and did not have this dynamic attention window training. The result, as shown in the ASR benchmarks (Table 5), is substantially lower word error rates: Qwen3.5-Omni-Plus achieves 6.55% WER on Fleurs (top 60 languages) compared to 7.32% for Gemini-3.1 Pro and 10.75% for Qwen3.5-Omni-Flash. On Cantonese specifically, Qwen3.5-Omni-Plus achieves 2.2% WER versus 6.3% for Gemini-3.1 Pro — a nearly 3× improvement on a language that was explicitly included in the expanded multilingual training data.

Temporal resolution implications. The 160 ms per token rate means the AuT produces one token for roughly every two spoken words in normal-speed English (approximately 150 words per minute). This is relatively coarse for tasks like phoneme-level analysis but adequate for word-level and utterance-level understanding, and it's the right tradeoff for a model that must handle 10-hour audio inputs. The paper doesn't explicitly discuss the temporal resolution tradeoffs, but the choice is implicit in the architecture: finer-grained audio understanding (like precise word timing) would come from the model's ability to attend across multiple tokens and use the explicit timestamps discussed in the next section, rather than from a higher encoder frame rate.

The Thinker — Unified Multimodal Perception and Reasoning

The Thinker is the central reasoning component that converts all input modalities into a unified representation and generates text output. It processes an interleaved sequence of tokens from text, audio (via AuT), and vision (via the Qwen3.5 vision encoder), using a single Hybrid MoE transformer with a 256k-token context window.

Text tokenization. Text is processed using the Qwen3.5 tokenizer, which the paper states "adopts byte-level byte-pair encoding with a vocabulary size of 250k (up from 150k), improving encoding and decoding efficiency by 10–60% across most languages" (Section 2.3). The 250k vocabulary is substantially larger than the 150k vocabulary used in Qwen3-Omni. A larger vocabulary means each token represents more text on average (higher compression), which is particularly important for the 256k context window — more content fits into the same number of tokens. The 10-60% efficiency improvement varies by language because BPE tokenizers are more efficient for languages with writing systems that decompose naturally into the byte pairs that BPE learns (e.g., Latin-alphabet languages) versus those where characters map less directly to bytes.

Audio token integration. Audio tokens from the AuT encoder are inserted into the Thinker's input sequence directly — there is no separate adapter or cross-attention mechanism. The audio is encoded at 6.25 Hz (one token per 160 ms), and these tokens are simply part of the 256k-token sequence that the Thinker processes. The paper doesn't specify whether audio tokens and text tokens share the same embedding space or use separate embedding tables, but the architecture diagram (Figure 2) shows a single unified transformer processing all modalities, which typically implies either shared embeddings or modality-specific embeddings that are summed or concatenated before entering the transformer.

Visual input processing. Images and videos are processed using the vision encoder from Qwen3.5, which is trained on a mixture of image and video data. For videos, frames are sampled at a dynamic frame rate — meaning the system can vary how many frames per second it extracts depending on the video content and the available context budget. This is important because fixed frame rate sampling would either waste tokens on static scenes or miss important fast action. The paper notes that this encoder "provides strong capabilities in both image understanding and video comprehension" (Section 2.3).

For silent video (video without an audio track), visual frames are treated as still images with temporal ordering. For audio-visual video (video with synchronized audio), the audio is extracted from the video file, resampled to 16 kHz, and encoded through the AuT as a separate stream that is interleaved with the video frames. This separation — processing visual and auditory streams independently and then fusing them in the Thinker — is fundamentally different from approaches that fuse audio and video at the encoder level.

The timestamps replacement for TMRoPE — a key architectural change. This is the most important change in the Thinker's perception pipeline relative to Qwen3-Omni. Qwen3-Omni used TMRoPE (Temporal Rotary Position Embedding), which assigned temporal position IDs by converting absolute time to integer positions. The paper identifies two critical problems with this approach (Section 2.3):

First, "directly encoding absolute time through temporal position IDs can lead to excessively sparse indices for visual patches from long video with audio inputs, which weakens long-range temporal modeling." The issue is that if you assign position ID 0 to an audio token at time t=0, and position ID 10 to the next audio token at t=1.6 seconds (because audio is at 6.25 Hz), a video frame at t=5 seconds might receive position ID 31 — and there's no mechanism for the position encoding to understand that tokens with widely separated position IDs should still have strong temporal relationships.

Second, "such a design often requires large-scale and uniformly distributed training samples across different frame rates, increasing data construction cost." Because the mapping from time to position ID depends on the frame rate, the model only learns to handle frame rates it was trained on. If you want to support variable frame rates, you need training data at all those rates, which is expensive.

The explicit timestamp solution. Instead of encoding time through position IDs, Qwen3.5-Omni "prepend[s] each video or audio-video temporal patch with an explicit timestamp represented as a formatted text string in seconds" (Section 2.3). For example, a video frame at t=5.3 seconds might have the text string "5.3" prepended to its visual tokens. This converts temporal information from an architectural mechanism (position encoding) into a semantic one (text the model can read and reason about), which has several advantages:

• Natural language understanding of time: The model can learn that "5.3" is close to "5.5" because it's trained on text representations, not because the position encoding happens to put them nearby. This is more robust and generalizes better to unseen temporal patterns.

• Arbitrary frame rates without retraining: Since the timestamp is just text, the model can handle any frame rate — it just reads whatever timestamp string is provided. There's no architectural dependency on the rate.

• Better long-range temporal modeling: When two events are 30 seconds apart, the model can see that one has timestamp "10.0" and the other has "40.0" and explicitly reason about the 30-second gap, rather than relying on position encoding to implicitly capture this through a multiple of the base frequency.

For audio sequences specifically, the paper additionally "insert[s] timestamps at random intervals to improve temporal alignment across modalities." This means that even within a continuous audio stream, the model occasionally sees timestamp markers like "12.5" inserted into the token stream. During training, this randomization teaches the model that timestamps can appear anywhere and must be interpreted relative to the surrounding audio content. During inference, this enables precise temporal reasoning about when things happen in the audio.

Handling positional conflicts across modalities. When multiple modalities are interleaved, there's a risk of positional conflicts — the position encoding might assign the same or overlapping positions to tokens from different modalities, confusing the model about ordering. The paper describes a specific resolution: "To prevent positional conflicts when processing multiple modalities, the position numbering is made contiguous, with each subsequent modality commencing from one plus the maximum position ID of the preceding modality" (Section 2.3). So if the text component uses positions 0–500, the audio component starts at position 501, and the video component starts after the audio ends. This is a simple but important detail: it ensures the model never confuses a text token with an audio token based on position alone.

Temporal ID alignment across modalities. The paper further specifies that "in the context of multimodal audio-visual streams, the audio component is encoded with a temporal ID for every 160 ms. The video is treated as a sequence of frames with monotonically increasing temporal IDs that are dynamically adjusted based on their actual timestamps to ensure a consistent temporal resolution of 160 ms per ID" (Section 2.3). This means both audio and video tokens are mapped to a shared temporal grid at 160 ms resolution, where each "temporal ID" represents an increment of 160 ms of real time. An audio token at t=3.2 seconds and a video frame at the same time would receive the same temporal ID, enabling the Thinker to directly associate them. This is a form of cross-modal temporal alignment that doesn't require explicit synchronization tokens — it's baked into the position numbering scheme.

Height and width position IDs. For visual frames, "height and width IDs for video frames are assigned in the same manner as for still images" — meaning the standard 2D position encoding for image patches is preserved. This is important because it means the vision encoder's spatial understanding (what's on the left vs. right, top vs. bottom) is not disrupted by the temporal encoding scheme. The model receives three types of positional information for each video patch: temporal (when), vertical (where in height), and horizontal (where in width).

The 256k context window. The Thinker supports a maximum context length of 262,144 tokens (256k). The paper provides concrete capacity numbers: this enables "more than 10 hours of audio and over 400 seconds of 720P audio-visual content at 1 FPS" (Section 2.1). The arithmetic: 10 hours of audio at 6.25 Hz = 225,000 audio tokens, leaving 37,144 tokens for text, video frames, and other content. For the 400 seconds of 720p video at 1 FPS, the 400 frames would produce some number of visual tokens (depending on the vision encoder's patch size), and the budget must be shared with the audio tokens from the same 400 seconds (2,500 audio tokens at 6.25 Hz). The fact that 10 hours of pure audio fits but only ~7 minutes of audio-visual content fits illustrates how much more token-intensive video is compared to audio.

Hybrid MoE architecture in the Thinker. Both the Thinker and Talker use the Hybrid Attention MoE architecture introduced in Qwen3.5. The paper doesn't provide detailed architectural specifications (number of layers, hidden dimensions, number of experts, expert capacity), but it does mention one notable component: the Gated Delta Net (GDN) module, which is "particularly effective for accelerating the modeling of long audio-video sequences" and "significantly reduces KV-cache I/O overhead in long-context inference, improving generation throughput and enabling higher serving concurrency" (Section 2.5). The GDN is a variant of linear attention that avoids the quadratic memory cost of standard self-attention over long sequences, which is critical for the 256k context window — standard attention would require storing 256k × d_model key-value pairs, which would be prohibitive for high-concurrency serving.

Talker — Streaming Speech Generation

The Talker is the speech synthesis component that converts the Thinker's multimodal representations and generated text into natural-sounding speech. It operates in a streaming manner, producing audio chunks as soon as they're available rather than waiting for the complete text response.

Speech representation — multi-codebook RVQ tokens. Qwen3.5-Omni uses a Residual Vector Quantization (RVQ) based speech representation, which the paper notes was "introduced in Qwen3-Omni" and "substantially improves inference efficiency" (Section 2.1). RVQ represents speech as a hierarchy of codebooks: the first codebook captures the most important acoustic features (coarse spectral envelope, fundamental frequency), the second codebook captures the residual — what the first codebook missed — and subsequent codebooks capture progressively finer acoustic details (breathiness, formant precision, high-frequency transients).

The specific tokenizer is Qwen3.5-Omni-Audio-Tokenizer, and the Talker "operates directly on the RVQ tokens" it produces. This is a multi-codebook representation, meaning each frame of audio is represented by multiple discrete tokens (one per codebook level), rather than a single token. For example, if there are 8 codebooks, each 160 ms frame of audio would be represented by 8 discrete tokens from 8 different vocabularies. The total bitrate is the codebook size × number of codebooks × 6.25 Hz, which is substantially higher than the text token rate but encodes much richer information.

Multi-Token Prediction (MTP) module. The Talker uses a multi-token prediction module to model the residual codebooks. The paper describes this as: "To model the residual codebooks, it employs a multi-token prediction (MTP) module, which enables fine-grained modeling and control of acoustic details" (Section 2.4). In MTP, at each decoding step, instead of predicting just one token (the first codebook for the current frame), the module predicts all codebooks for the current frame simultaneously — or more precisely, predicts the first codebook autoregressively and the subsequent codebooks in parallel conditioned on the first. This is what Table 1 refers to when it says "At each decoding step, an MTP module outputs the residual codebooks for the current frame." The advantage is that you get the fine-grained acoustic detail of multiple codebooks without the autoregressive decoding cost scaling linearly with the number of codebooks — you only pay the sequential penalty for the first codebook, and the rest are generated in parallel.

Talker conditioning on Thinker context. The Talker does not generate speech in isolation. It is "conditioned on the rich contextual information provided by the Thinker component, including historical text tokens, multimodal representations, and the streamed text of the current turn" (Section 2.4). This means the Talker receives:

• Historical text tokens: The text of previous turns in the conversation, providing linguistic context for appropriate prosody (e.g., continuing a topic with the same speaking style, or responding to a question with appropriate intonation).
• Multimodal representations: The Thinker's internal representations of the audio, visual, and text inputs for the current turn. This allows the Talker to modulate speech based on what the user's voice sounded like (e.g., matching emotional tone) or what was shown in a video (e.g., describing something exciting with increased energy).
• Streamed text of the current turn: As the Thinker generates text tokens one by one, the Talker receives these incrementally. This is what enables streaming — the Talker can begin generating speech for the first few words before the Thinker has finished generating the complete response.

The paper notes that "such conditioning allows Talker to dynamically modulate acoustic attributes — such as prosody, loudness, and emotion — in accordance with the evolving conversational context." This is a key capability for natural conversation: if the Thinker's text response expresses surprise, the Talker should increase pitch and energy; if the response is consoling, the Talker should soften and slow down. This conditioning mechanism is what makes these adjustments possible without explicit emotion or prosody tags in the text.

Talker system prompt for voice cloning and control. A significant new feature in Qwen3.5-Omni is the introduction of a "dedicated system prompt for Talker that specifies target voice characteristics, thereby enabling both zero-shot voice cloning and controllable speech generation" (Section 2.4). This is in contrast to "conventional speaker embeddings" — fixed vectors that represent a speaker's identity — which the paper argues are less flexible. The system prompt "can encode richer multimodal cues, including textual descriptions and codec sequences, providing substantially finer-grained control over acoustic realization."

For zero-shot voice cloning, the prompt might include a few seconds of the target speaker's audio encoded as RVQ tokens, allowing the Talker to reproduce that speaker's timbre, pitch range, and speaking style. For controllable speech generation, the prompt might include text descriptions like "speak in a calm, slow voice with a slight British accent" — the Talker learns to interpret these textual instructions as acoustic parameters.

This is philosophically similar to how the Thinker uses explicit text timestamps instead of position encoding: convert what was previously a fixed architectural mechanism (speaker embeddings) into a flexible semantic one (text prompts that the model interprets), gaining generalizability at the cost of slightly more tokens in the input.

Comparison with Qwen3-Omni's Talker. The paper highlights two key differences from Qwen3-Omni: (1) the system prompt for voice characteristics, and (2) ARIA replacing the dual-track generation paradigm. The dual-track design in Qwen3-Omni had the Thinker's text output and the Talker's speech output generated in parallel tracks with some fixed interleaving mechanism. This is the problem that ARIA is designed to solve, which we'll cover in the next section.

Code2Wav renderer. The final component in the speech pipeline is Code2Wav, a "causal ConvNet for waveform reconstruction" that converts the Talker's predicted RVQ codec tokens into actual audio waveforms that can be played through speakers. The ConvNet is "causal" meaning each output audio sample depends only on past and current codec tokens, never future ones — this is what makes streaming possible, since the model can output audio incrementally as tokens arrive. The paper notes in Table 1 that Code2Wav operates in streaming mode (✓) and that the codec decoding latency is approximately 3–5 ms, which is essentially negligible compared to the Thinker and Talker latencies.

ARIA — Adaptive Rate Interleave Alignment

ARIA is the paper's central technical innovation for speech generation. It addresses the fundamental instability in streaming speech synthesis that arises from the mismatched encoding efficiencies of text and speech tokenizers. To understand ARIA, we must first understand the problem it solves.

The root problem: text-speech rate mismatch. Consider what happens when a model generates spoken output for the sentence "The weather is nice today." In a typical subword tokenizer, this might be 5-7 text tokens (e.g., "The", "weather", "is", "nice", "today"). But in the RVQ speech codec at 6.25 Hz, these same words require many more speech tokens — at a normal speaking rate of ~3 words per second, a 7-word sentence takes about 2.3 seconds to speak, which at 6.25 Hz is roughly 14 speech frames (each with multiple codebook tokens). So the ratio of speech tokens to text tokens for this sentence is approximately 14:7 = 2:1.

But this ratio is not constant. For a fast speaker, the same sentence might take 1.5 seconds (9 speech frames, ratio 9:7 ≈ 1.3:1). For a slow, deliberate speaker, it might take 3 seconds (19 speech frames, ratio ≈ 2.7:1). For different languages, the ratio varies dramatically: a syllable-timed language like Spanish has a more regular text-to-speech ratio than a stress-timed language like English, and a mora-timed language like Japanese has an entirely different rhythm. Even within a single utterance, some words are spoken quickly and others are drawn out for emphasis.

In Qwen3-Omni's dual-track Talker, this variability caused systematic failures. The text track and speech track operated in parallel, with some fixed mechanism (either MFA-based forced alignment or a fixed interleaving rate like "interleave one text token every 8 speech frames") to keep them synchronized. When the actual text-to-speech ratio deviated from the assumed ratio — which it inevitably did — the tracks would drift apart. The speech track might start generating audio for words the text track hadn't produced yet (causing garbled or hallucinated speech), or the text track might be ready with new words but the speech track was still rendering previous words (causing pauses or filler sounds).

What ARIA does differently. ARIA replaces the dual-track paradigm with a unified interleaved single-stream formulation over text and speech tokens. Instead of generating text and speech in parallel and trying to align them after the fact, ARIA generates them in a single interleaved stream with an explicit constraint on the cumulative ratio:

"ARIA enforces an adaptive rate constraint: for any prefix of the generated sequence, the cumulative speech-to-text token ratio must not exceed the corresponding item-level global ratio." (Section 2.4)

This constraint deserves careful unpacking because it is both simple and powerful:

• "For any prefix of the generated sequence" means the constraint is checked continuously during generation — not just at the end of sentences or at alignment points. At every decoding step, the model considers what it has generated so far (the prefix) and checks whether it's allowed to generate another speech token or must generate text first.

• "The cumulative speech-to-text token ratio" is simply the total number of speech tokens generated so far divided by the total number of text tokens generated so far. If the model has generated 20 speech tokens and 10 text tokens, the cumulative ratio is 2.0.

• "Must not exceed the corresponding item-level global ratio" means this cumulative ratio cannot go above a pre-determined target for the current utterance. The "item-level global ratio" is the expected ratio for the entire response or turn — essentially, "for this response, text and speech will be generated at approximately this ratio." If the global ratio is, say, 2.5 (meaning each text token should roughly correspond to 2.5 speech tokens), then at any point during generation, the cumulative ratio must be ≤ 2.5.

How the constraint shapes generation. The practical effect of this constraint is that the Talker's decoding process alternates between generating text tokens and speech tokens based on the current cumulative ratio. If the cumulative ratio is below the global target (i.e., it has generated relatively more text than speech so far), it is allowed to generate speech tokens. If the cumulative ratio is at or above the global target, it must generate text tokens before it can generate more speech. This creates a natural "breathing" rhythm: the model generates some text, then some corresponding speech, then more text, then more speech, with the ratio constraint ensuring they never drift too far apart.

The key word is "adaptive" — the constraint doesn't force a fixed interleaving pattern. The model can generate 5 text tokens followed by 12 speech tokens (ratio 12:5 = 2.4) or 1 text token followed by 2 speech tokens (ratio 2:1 = 2.0) or any other pattern, as long as the cumulative ratio never exceeds the global target. This flexibility allows the model to adapt to local variations in speaking rate, word complexity, and prosodic emphasis while maintaining global alignment.

Why "must not exceed" rather than "must equal"? The inequality constraint (≤ rather than =) is important. It means the model can temporarily generate relatively more text than speech — the cumulative ratio can dip below the global target — which is natural at the beginning of a response when the model might want to plan several words ahead before starting to speak. It cannot, however, generate relatively more speech than text — this prevents the speech from "running ahead" of the text, which was the primary failure mode in the dual-track system. The constraint is asymmetric because the failure modes are asymmetric: speech getting ahead of text produces garbled output, while text getting ahead of speech just means the model has planned ahead (which is fine as long as the speech eventually catches up).

Comparison with prior approaches. The paper explicitly contrasts ARIA with two alternatives (Section 2.4):

• MFA (Montreal Forced Aligner)-derived alignments: These require running a separate speech-text alignment model that uses language-specific acoustic models and phoneme inventories. This works for high-resource languages but doesn't scale to the 36 languages of speech output that Qwen3.5-Omni supports. ARIA requires no language-specific components at all — the constraint is purely numerical.

• Fixed interleaving rates: These assume a constant ratio (e.g., 8 speech tokens per text token) for all utterances. This is brittle because the actual ratio varies with language, speaker, and content. ARIA allows flexible local variation while enforcing a global bound.

The paper's characterization of ARIA as having "despite its simplicity" is notable — it suggests that the prior approaches over-complicated the problem by trying to achieve perfect alignment at every moment, when a simple cumulative constraint was sufficient to prevent the catastrophic failure modes.

ARIA's role in streaming. In a streaming context, ARIA creates a unified token stream that the Talker can decode autoregressively. As the Thinker streams text tokens to the Talker, the Talker interleaves them with speech tokens according to the ARIA constraint, producing a single sequence like:

[text: "The"] [speech: frame1] [speech: frame2] [text: "weather"] [speech: frame3] [speech: frame4] [text: "is"] ...

This single-stream formulation eliminates the synchronization overhead of the dual-track design — there's only one decoder maintaining one state, one KV-cache, and one stream of output tokens. The paper notes this "reduces the synchronization overhead between separate generation tracks, enables more efficient token scheduling during decoding, and better matches the naturally incremental regime of streaming interaction" (Section 2.5).

How ARIA enables the latency characteristics. The theoretical latency numbers in Table 2 depend on ARIA's single-stream efficiency. The Talker's Time-To-First-Chunk (TTFC) — the time until the first playable audio packet is produced — is 54-95ms for the Plus variant and 56-81ms for Flash. These low latencies are possible because ARIA doesn't require waiting for the complete text response before starting speech generation; as soon as the first text token arrives from the Thinker, the Talker can begin generating corresponding speech tokens within the ARIA constraint. The table notation "Overall Latency cannot be obtained by simply summing several row values, but instead reflects the end-to-end critical path to the first playable audio packet" reinforces that this is a pipelined streaming system, not a sequential batch system.

Streaming and Concurrency Optimizations

The paper provides unusually detailed latency analysis for a technical report, reflecting the fact that real-time interaction is a core capability rather than an afterthought. Table 1 and Table 2 provide a comprehensive latency architecture.

Chunked prefilling. The Thinker uses "chunked-prefilling" — a mechanism where input is processed in chunks (portions of the total sequence) rather than waiting for the complete input. The paper notes this was "implemented in Qwen3-Omni and Qwen2.5-Omni" and "significantly reduces the Time-To-First-Token (TTFT) for both the Thinker and the Talker" (Section 2.5). The key insight is that for streaming audio or video, you don't need to wait for the full 10-hour audio to arrive before starting to process. The audio and vision encoders "are capable of outputting chunks along the temporal dimension," meaning they can produce partial encodings as data arrives. The Thinker can begin processing the first chunk while later chunks are still being encoded.

The chunked approach applies to both input modalities. For audio, the AuT encoder processes audio in chunks (presumably corresponding to some number of 160ms frames), each of which can be immediately passed to the Thinker. For video, frames can be processed as they arrive rather than waiting for the full video.

First-packet latency breakdown (Table 1). The paper distinguishes between audio and video scenarios:

• Audio input: First-packet latency is 435ms for Plus, 235ms for Flash. This is the time from the start of audio input to the first audio output packet — the complete end-to-end latency for the user to begin hearing the response. Flash's 235ms is approaching the ~200ms threshold that humans perceive as "immediate" in conversation.
• Video input: First-packet latency is 651ms for Plus, 426ms for Flash. The substantial increase over audio-only (216ms for Plus, 191ms for Flash) reflects the additional cost of processing video frames through the vision encoder and the Thinker.

Detailed latency decomposition (Table 2). Table 2 provides per-component latencies at three concurrency levels (1, 4, and 8 concurrent streams):

• Thinker TTFT (Time-To-First-Token): The time from receiving the input stream to the first text token generated by the Thinker. At concurrency 1, this is 80ms for audio and 255ms for video on Flash, and 162ms/377ms on Plus. This is the dominant component — the Thinker is the bottleneck, not the encoder or the Talker.

• Talker TTFC (Time-To-First-Chunk): The time until the Talker produces the first audio chunk. At concurrency 1, this is 56ms/61ms for Flash and 54ms/56ms for Plus (audio/video). These are remarkably low — the Talker is fast because it's generating from a pre-computed Thinker representation, not reasoning from scratch.

• Thinker TPOP (Time-Per-Output-Token): The per-token generation latency during steady-state text decoding. At concurrency 1, 5.6ms/5.9ms for Flash (audio/video), 17.4ms/18.5ms for Plus. The Plus is ~3× slower per token than Flash, reflecting the larger model size.

• Talker TPOP: Per-token latency for speech generation, including both the Talker backbone and the MTP module. At concurrency 1, 14.2ms/14.2ms for Flash, 14.9ms/14.9ms for Plus. Interestingly, the Talker latency is almost identical between Flash and Plus — suggesting the Talker may be the same (or similar) size in both variants, with the model capacity differences concentrated in the Thinker.

• Codec Decode: 3-5ms, essentially negligible.

• Overall Latency: The end-to-end critical path to the first playable audio packet, which at concurrency 1 is 235ms/426ms for Flash (audio/video) and 435ms/651ms for Plus. The paper warns that "since ARIA organizes text and speech generation in a unified interleaved stream, Overall Latency cannot be obtained by simply summing several row values" — the pipelining means that Thinker text generation and Talker speech generation overlap.

• Thinker TPS (Tokens Per Second): Generation throughput for text. At concurrency 1, 177/171 tokens/sec for Flash (audio/video), 57/54 for Plus. These are relatively low because at concurrency 1, the model is not fully utilizing the hardware.

• Talker TPS: 70/70 for Flash at concurrency 1, 67/67 for Plus. The speech generation throughput scales well with concurrency (reaching 389/388 for Flash at concurrency 8), suggesting efficient batching.

• Generation RTF (Real-Time Factor): The ratio of computation time to audio duration. At concurrency 1, 0.178 for Flash, 0.187 for Plus. Both are well below 1.0, meaning the system can generate speech faster than real-time — a critical requirement for streaming. At concurrency 8, RTF increases to 0.257 (Flash) and 0.334 (Plus), still below 1.0, meaning 8 simultaneous streams can all run in real-time.

Concurrency scaling behavior. A notable pattern in Table 2: as concurrency increases from 1 to 8, overall latency roughly doubles (235→352ms for Flash audio, 435→955ms for Plus) rather than scaling linearly. This indicates effective batching — processing multiple streams together amortizes overhead. The Thinker TPS also scales dramatically with concurrency (177→942 for Flash, 57→266 for Plus), showing that the Hybrid MoE architecture efficiently handles batched inference.

Hybrid MoE and GDN for efficiency. The paper attributes this efficiency to two architectural choices. The Hybrid MoE architecture activates only a fraction of parameters per token, reducing the effective FLOPs per token. The Gated Delta Net (GDN) module "is particularly effective for accelerating the modeling of long audio-video sequences" and "significantly reduces KV-cache I/O overhead in long-context inference" (Section 2.5). KV-cache I/O is often the bottleneck for long-context inference because the key-value pairs for every previous token must be loaded from memory for each new token — for a 256k context, this is a massive memory bandwidth problem. The GDN module's linear attention mechanism presumably avoids storing and loading the full quadratic attention matrix, enabling the high concurrency numbers in Table 2.

Deployment infrastructure details. The latency numbers are evaluated "on internal vLLM with torch.compile and CUDA Graph acceleration enabled for the MTP module and codec decoder." vLLM is a popular LLM serving framework; torch.compile provides JIT compilation optimizations; CUDA Graph acceleration reduces kernel launch overhead for the small, repetitive operations in the MTP and codec decoder. The paper notes that "due to the substantial scale difference between Qwen3.5-Omni-Flash and Qwen3.5-Omni-Plus, the two variants adopt different deployment-time resource allocation and parallelization strategies; therefore, their latency and throughput numbers are not intended for strict horizontal comparison" — meaning you shouldn't directly compare Flash and Plus latencies as if they're on identical hardware, because they're deployed differently optimized for their respective scales.

Pretraining Strategy — Three Stages with Massive Multimodal Data

The pretraining of Qwen3.5-Omni follows a carefully structured three-stage process designed to progressively build multimodal capabilities without degrading the base LLM's text understanding.

Stage 1: Encoder Alignment. In this initial phase, "the LLM component of Qwen3.5-Omni is initialized with parameters from Qwen3.5" — so the text reasoning capabilities start from a state-of-the-art text-only model. The vision encoder is adopted from Qwen3.5 (pre-trained on images and videos), and the audio encoder is initialized from the AuT pre-training. Crucially, "the LLM parameters are locked" — the base language model is frozen, and only the encoders and their adapters are trained. The training uses "a vast corpus of audio-text and image-text pairs to enhance semantic understanding within the LLM" (Section 3, Stage 1).

The training proceeds in two sub-steps within Stage 1: first, the adapters (projection layers between encoder outputs and the LLM's embedding space) are trained; then, the encoders themselves are unfrozen and trained. This graduated approach prevents the randomly initialized adapters from producing noisy gradients that could destabilize the pre-trained encoders. The goal is to align the audio and vision representations with the LLM's existing semantic space without disturbing the LLM's text capabilities — the encoders learn to produce representations that the LLM can "understand" in terms of its existing text-based knowledge.

Stage 2: General Multimodal Training. This is the main pretraining phase, where "all parameters are unfrozen" and the model is trained on "a wider range of multimodal data for more comprehensive learning with a sequence length of 32,768" (Section 3, Stage 2). The dataset composition is specified precisely:

• Total: approximately 4 trillion tokens
• Text: 0.92 trillion tokens (23% of total)
• Audio: 1.99 trillion tokens (49.8% of total — the largest single modality)
• Image: 0.95 trillion tokens (23.8% of total)
• Video: 0.14 trillion tokens (3.5% of total)
• Video-Audio: 0.29 trillion tokens (7.3% of total, though these may overlap with the audio and video categories above, totaling more than 100%)

The heavy emphasis on audio (nearly half the training data) reflects the model's intended strength in speech and audio understanding. The relatively small proportion of video data (3.5% for silent video, 7.3% for video with audio) reflects the higher cost and lower availability of video data compared to text, images, and audio. The 32,768-token sequence length at this stage is 8× longer than many standard LLM pretraining contexts (typically 4,096), allowing the model to learn medium-range temporal dependencies in audio and video content.

Stage 3: Long Context Extension. In the final pretraining phase, "we increased the maximum token length from 32,768 to 262,144 and also raised the proportion of long audio and long video in the training data." The 8× jump in context length (32k → 256k) is achieved through continued training with longer sequences and more long-form data. The paper states that "experimental results indicate that these adjustments lead to significant improvements in the model's ability to understand long sequence data" — though no specific ablation numbers are provided in the paper.

The mechanism for the context extension isn't specified (e.g., whether it uses RoPE frequency scaling, position interpolation, or simply continuing training with longer sequences), but the explicit timestamps approach (prepending time as text) may help here: since temporal relationships are encoded as text rather than position IDs, the model can potentially generalize to longer sequences more easily because it reads explicit time values rather than relying on position encoding extrapolation.

Text preprocessing with natural language prompts. The paper notes that "following Qwen3-Omni, we employ a wider range of natural language prompts to enhance both the generalization ability and instruction-following capabilities" (Section 3). This means the training data is constructed with diverse prompt templates — not just "Describe this audio" but variations like "What sounds do you hear in this recording?", "Can you transcribe what's being said?", "Summarize the key events in this audio clip" — which prevents the model from overfitting to specific prompt formats and improves generalization.

Multilingual training scope. Table 3 provides the comprehensive list of supported languages and dialects:

• Text: 201 languages (inherited from Qwen3.5)
• Speech input (ASR): 74 languages + 39 Chinese dialects = 113 total varieties. The languages range from high-resource (English, Chinese, Spanish) to lower-resource (Afrikaans, Asturian, Cebuano, Kyrgyz, Tajik). The inclusion of 39 Chinese dialects is notable — these are typically underrepresented in multilingual speech systems.
• Speech output (TTS): 29 languages + 7 Chinese dialects = 36 total varieties. This is a subset of the input languages, reflecting the greater difficulty of generating natural speech (which requires modeling prosody and accent) compared to recognizing it.

Post-Training Pipeline for the Thinker

The Thinker's post-training employs a three-stage strategy designed to "preserve the model's capabilities across all modalities without degradation, ensure high response quality under audio queries, and optimize the overall interaction experience" (Section 4.1). The training uses the ChatML format for structuring conversational data.

Stage 1: Specialist Distillation. The idea is to train separate expert models for different domains (text reasoning, vision, audio), then distill their capabilities into a single unified model. All teacher models are fine-tuned from the pre-trained Qwen-3.5 base checkpoint using Supervised Fine-Tuning (SFT) and reinforcement learning (RL). The paper specifies that "beyond text-related tasks, including agentic, coding, and foundational reasoning tasks, we also train specialized teacher models for vision and audio" (Section 4.1).

The specialist teachers are used to generate "domain-specific data" — for example, the vision teacher might produce high-quality visual descriptions, the audio teacher might produce detailed audio captions, the coding teacher might produce correct code solutions — and these are then used as training targets for the unified model. This is a form of knowledge distillation where the student sees diverse, high-quality outputs and learns to replicate them. The advantage over training directly on a mixture of data is that the specialist teachers can produce higher-quality outputs in their domains than a single model could, and distilling from them transfers this quality.

Stage 2: On-Policy Distillation (OPD) — the key quality alignment step. Despite the specialist distillation, "a substantial gap remains between the quality of responses conditioned on audio queries and that of responses conditioned on text queries, particularly in speech dialogue" (Section 4.1). This is the core problem: the model reasons better from text than from audio, even when the content is the same. A user speaking a math problem gets a lower-quality response than a user typing the same problem.

OPD addresses this by leveraging the model's own text-conditioned capabilities. For each audio-text paired query:

1. The model is given the text version of the query (e.g., a transcription of what the user said).
2. The model generates a response conditioned on this text input, which typically has higher quality.
3. This high-quality text-conditioned response is used as the distillation target for the audio-conditioned version of the same query.
4. The model is trained (via SFT) to produce the text-conditioned-quality response when given only the audio input.

The paper describes this as "on-policy" because it uses the model's own current policy (i.e., its own generations under text conditioning) as targets, iteratively improving. The training signal is: "when you hear this audio, respond as if you had read it as text." Over many training examples, this gradually closes the gap between audio-conditioned and text-conditioned response quality.

Stage 3: Interaction-Aligned RL. Even with specialist distillation and OPD, models trained primarily on single-turn or short multi-turn data exhibit problems in extended conversations: "unintended language code-switching, persona inconsistency, and degraded instruction-following over extended contexts" (Section 4.1). These are specifically interaction-quality issues that don't show up in benchmark evaluations but dramatically affect user experience.

To address these, the paper introduces Interaction-Aligned RL, where multi-turn conversation trajectories are constructed and reward signals are designed around interaction quality objectives. The paper doesn't specify the exact reward functions, but the description suggests they target:

• Language consistency: Penalizing code-switching (e.g., suddenly responding in a different language than the conversation was in).
• Persona consistency: Rewarding the model for maintaining a coherent personality, role, or style across turns.
• Instruction-following stability: Rewarding the model for continuing to follow complex instructions even in later turns of long conversations.

By explicitly optimizing for these interaction-level objectives through RL, the model learns to produce more stable, consistent, and aligned behavior in prolonged interactions. The paper claims this "improves the model's overall usability in practical conversational scenarios."

Post-Training Pipeline for the Talker

The Talker's training follows a four-stage pipeline to enable natural, contextually appropriate spoken responses.

Stage 1: General Stage. The Talker is initially trained on "more than 20 million hours of multilingual speech data paired with multimodal context" (Section 4.2). This is distinct from the AuT encoder's 40 million hours — the AuT encoder learns audio-to-text mapping, while the Talker learns text/context-to-speech mapping. The training goes "beyond a simple monotonic mapping from multimodal representations to speech" by introducing diverse tasks, such as "instruction-following speech generation," which requires the Talker to modulate its output based on linguistic instructions (e.g., "speak more slowly" or "emphasize the word 'important'").

Stage 2: Long-Context Stage. After initial training, the model undergoes "data quality stratification through a dedicated curation pipeline" and "continual pre-training (CPT) on high-quality subsets" (Section 4.2). This is critical because the initial 20 million hours of data inevitably contains noisy samples — misaligned text-speech pairs, background noise, transcription errors — that can introduce hallucinations in generated speech (e.g., pronouncing words that weren't in the text, or generating unnatural prosody for certain word sequences).

To mitigate this, the paper augments the curation with "Qwen3-Omni-Captioner," which presumably generates high-quality captions or transcriptions for speech data, enabling better filtering and alignment. The stage also extends the maximum context length to 64k tokens, allowing the Talker to handle longer and more complex Thinker outputs.

Stage 3: Reinforcement Learning Stage. This stage aligns Talker behavior with human preferences through two complementary approaches:

• Direct Preference Optimization (DPO): "Multilingual preference pairs based on human annotations" are constructed — pairs of speech outputs where human annotators preferred one over the other — and the model is optimized to prefer the chosen output over the rejected one using DPO's closed-form preference loss.

• Group Sequence Policy Optimization (GSPO): The paper additionally incorporates "rule-based rewards and adopt[s] GSPO (Zheng et al., 2025) to further improve overall capability and training stability across diverse tasks." GSPO is a variant of policy optimization that operates on groups of sequences (rather than individual pairs), which can be more stable for training across diverse, multi-objective tasks. The rule-based rewards likely include metrics like pronunciation accuracy (measured by ASR WER on the generated speech), prosodic naturalness, and speaker similarity — objectives that can be automatically computed without human annotation.

The combination of DPO (using human preferences for subjective quality) and GSPO (using rule-based rewards for objective metrics) allows the Talker to optimize for both perceived naturalness and measurable accuracy.

Stage 4: Speaker Fine-Tuning. The final stage is "lightweight speaker fine-tuning on top of the base model, enabling Qwen3.5-Omni to faithfully capture target speaker characteristics while further improving the naturalness, expressiveness, and controllability of its speech outputs" (Section 4.2). This is what enables the custom-voice speech generation evaluated in Table 12, where the model can clone a speaker's voice from just a few seconds of sample audio. The paper notes that Qwen3.5-Omni "is fine-tuned only on monolingual data" for this stage, yet "demonstrates strong cross-lingual generalization in custom-voice speech generation" — meaning the model can reproduce a speaker's voice characteristics in languages it never heard that speaker speak. This is a non-trivial generalization capability that suggests the Talker has learned to disentangle speaker identity (timbre, pitch range, speaking style) from linguistic content.

Summary of Design Choices and Their Justifications

• Explicit text timestamps over TMRoPE: Treating time as readable text rather than encoded in position IDs avoids sparse position problems, eliminates the need for frame-rate-uniform training data, and enables more robust long-range temporal reasoning.
• ARIA over dual-track Talker: The cumulative ratio constraint provides a language-independent, computationally simple mechanism for text-speech alignment that prevents catastrophic drift without requiring language-specific aligners.
• Hybrid MoE with GDN for both Thinker and Talker: The efficient attention mechanism and selective parameter activation enable the 256k context window and high-concurrency serving without prohibitive computational cost.
• On-policy distillation for audio quality: Using the model's own text-conditioned responses as training targets for audio-conditioned inputs closes the modality quality gap without requiring external high-quality audio-labeled data.
• The 6.25 Hz audio token rate: Balances temporal resolution (160ms granularity) with context efficiency (10 hours fits in 256k tokens), enabling long-form audio understanding.
• Three-stage pretraining (locked LLM → full unfreeze → long context): Gradually introduces multimodal and long-context capabilities without destabilizing the base LLM's text understanding.
• Monolingual custom-voice fine-tuning with cross-lingual generalization: Testing the model's ability to clone voices in unseen languages validates that speaker identity has been disentangled from language-specific phonetics during training.

4. Key Insights and Innovations

Innovation 1: The Text–Speech Rate Mismatch as the Diagnostic Concept for Streaming Speech Instability

The paper's most conceptually distinctive contribution is not ARIA itself — a specific alignment mechanism — but the diagnostic framing that identifies why streaming speech generation in omnimodal models has been unstable and unnatural. Prior work treated speech generation quality as a data-scale or model-capacity problem: train on more data, use a bigger model, and speech will sound better. The dual-track Talker in Qwen3-Omni represented this philosophy — generate text and speech in parallel with some synchronization mechanism, and hope the alignment works out.

What this paper does differently is name a specific architectural pathology: text tokenizers and speech codec tokenizers operate at fundamentally different encoding densities per unit of semantic content, and this ratio varies dramatically across languages, speakers, and even within single utterances. A single text token might encode "the" or "antidisestablishmentarianism" — spans of vastly different acoustic duration — while speech codec tokens encode fixed ~160ms frames regardless of linguistic content. The consequence is that parallel-track generation with fixed-rate interleaving produces systematic alignment failures — skipped words, garbled pronunciations, ambiguous number rendering — that are not data-quality issues but architectural ones.

This framing is significant because it redirects the research conversation. Before this paper, improving streaming speech meant scaling up: more training data, bigger models, better codecs. After this paper, the question becomes: how do you architecturally handle the fact that text and speech propagate at different rates through the model? This is a qualitative shift from a data-scale framing to an architectural-alignment framing, and it opens a new class of solutions (adaptive interleaving constraints, dynamic ratio tracking) that were not naturally motivated by the prior framing.

The evidence for this diagnostic insight is indirect but cumulative across the paper. The ARIA mechanism is explicitly motivated by the rate-mismatch problem (Section 2.4). The comparison with Qwen3-Omni's dual-track design highlights what changed. The multilingual speech generation results (Tables 9, 10, 11), showing strong performance across 29 languages with widely varying phonological structures, provide the empirical validation: if the rate mismatch is the core pathology and ARIA addresses it in a language-independent way, then strong cross-lingual generalization is expected. The cross-lingual voice cloning results (Table 12) further support this — the model can maintain speaker identity and content fidelity even in languages it was never fine-tuned on for that speaker, suggesting the alignment mechanism doesn't overfit to language-specific phonetic patterns.

This is a fundamental conceptual contribution rather than an incremental improvement. Prior work in speech-language models (CosyVoice 2/3, MaskGCT, F5-TTS, Spark TTS) treated text-speech alignment as a pre-processing or post-processing step external to the model architecture. ARIA internalizes the alignment constraint into the generation process itself, which represents a different category of solution entirely.

Innovation 2: Explicit Text Timestamps as a Semantic Alternative to Positional Encoding for Temporal Awareness

The paper makes a subtle but philosophically significant architectural choice that represents a different way of thinking about what kinds of information should be encoded in model architecture versus model input. Prior approaches to temporal awareness in multimodal models — including Qwen3-Omni's TMRoPE — embedded temporal information in the position encoding mechanism: the rotary position embeddings were modulated by absolute or relative time, making temporal proximity a property of how position IDs were assigned rather than something the model explicitly reads and reasons about.

Qwen3.5-Omni inverts this: temporal information is removed from the architecture and placed in the input as formatted text strings ("5.3" prepended to a video frame, random timestamp insertions in audio streams). This seems like a simple implementation detail, but it represents a conceptual stance: temporal reasoning is not fundamentally different from any other kind of symbolic reasoning the model does, so it should use the same mechanism — reading and interpreting text tokens — rather than requiring a special-purpose architectural feature.

The significance of this shift extends beyond temporal awareness. It is an instance of a broader design principle: whenever possible, encode information as text tokens that the model can read rather than as architectural modifications that the model must learn to interpret implicitly. The paper makes the same move with speaker characteristics in the Talker, replacing "conventional speaker embeddings" (fixed architectural vectors) with a "dedicated system prompt" (text the model reads). This is not a coincidence — it's a consistent philosophy of moving semantic information from architecture to input, trading slightly longer context length for dramatically more flexible and generalizable processing.

The practical benefits are substantial and grounded in specific limitations the paper identifies in TMRoPE (Section 2.3): sparse position IDs for long video undermine temporal modeling; effective learning requires expensive uniform frame-rate sampling; extrapolation to unseen temporal patterns is unreliable. By moving timestamps to text, all three problems disappear simultaneously — the model reads explicit time values, so it can handle any frame rate, any temporal spacing, and any sequence length without architectural modification.

This is a fundamental architectural insight that generalizes beyond this specific model. The explicit timestamp approach demonstrates that positional encoding — long considered an essential architectural component of transformers — may be the wrong mechanism for certain kinds of structured relational information. When the relationship is simple and symbolic (two events are 30 seconds apart), letting the model read "30.0" as text is more robust than trying to encode that relationship through sinusoidal frequencies in position embeddings. This is a contribution to the broader debate about what belongs in architecture versus what belongs in data, and it has implications for any multimodal system that needs to reason about structured relationships (spatial coordinates, speaker identities, scene graphs) alongside unstructured perceptual inputs.

Innovation 3: On-Policy Distillation as a Modality Transfer Strategy Rather Than a Data Augmentation Strategy

The paper's on-policy distillation (OPD) technique, described in Section 4.1 Stage 2, represents a conceptually novel approach to a problem that is typically framed as a data collection challenge. The standard framing would be: "audio-conditioned responses are lower quality because we don't have enough high-quality audio-conditioned training data, so we need to collect more." Solutions would involve human annotation, synthetic data generation, or cross-modal data augmentation — all approaches that treat the gap as a data deficiency.

OPD reframes the problem entirely: the model already knows how to produce high-quality responses — it just does so under text conditioning, not audio conditioning. The gap is not a capability gap ("the model can't reason well enough") but a modality transfer gap ("the model has good reasoning, but it's not equally accessible through all input modalities"). The solution is not to collect more audio training data but to use the model's own text-conditioned reasoning as a teacher signal for its audio-conditioned reasoning — essentially, "teach the audio pathway to produce the same outputs the text pathway already can."

This is conceptually significant because it identifies a specific failure mode in multimodal training that is not addressed by standard approaches like data mixture balancing or multi-task learning. Even when a model is trained on massive amounts of audio data (1.99 trillion tokens, per Stage 2 pretraining), there remains a systematic quality gap between modalities. This suggests the gap is not simply a matter of data quantity — it may reflect fundamental asymmetries in how the model processes different modalities, where text input provides a more direct path to the model's reasoning capabilities than audio input does. OPD addresses this asymmetry directly by creating an explicit training signal that says "when you receive this query through audio, produce the same response you would produce through text."

The evidence for this conceptual reframing comes from the text-to-text benchmark parity (Table 4), where Qwen3.5-Omni-Plus achieves 85.9 on MMLU-Pro versus 86.8 for the text-only Qwen3.5-Plus-Instruct — a gap of less than 1 percentage point. This near-parity is difficult to achieve if multimodal training systematically degrades text capabilities; it suggests that the training pipeline, including OPD, successfully preserves the base model's reasoning while extending it to new modalities. The paper also notes that "OPD and interaction-aligned RL have a positive effect on improving the instruction-following capabilities of an omni-model LLM" (Section 5.1.1), observing that instruction-following scores are sometimes higher for the omni model than the text-only baseline.

This is an incremental but conceptually important innovation. OPD is not the first distillation method for multimodal models, but its specific formulation — using the model's own on-policy text-conditioned generations as targets for audio-conditioned training — targets a specific pathology (modality transfer gap) that prior multimodal training recipes did not explicitly address. It provides a template for future multimodal systems: anytime a model exhibits a quality gap between modalities, consider whether the gap can be closed by distilling from the stronger modality to the weaker one, rather than collecting more data for the weaker modality.

Innovation 4: Omnimodal Training Does Not Inherently Degrade Modality-Specific Performance — When the Architecture and Training Are Designed Correctly

This is a negative-result-as-insight contribution. A persistent assumption in the multimodal AI community has been that training on multiple modalities necessarily involves a tradeoff: you gain multimodal capabilities at the cost of degrading performance in each individual modality — the so-called "forgetting" or "catastrophic interference" problem. This assumption has been so widespread that it shaped research agendas: papers either accepted the tradeoff as inevitable or developed complex continual learning, elastic weight consolidation, or modality-specific regularization techniques to mitigate it.

Qwen3.5-Omni provides strong empirical evidence that this tradeoff is not fundamental — it is an artifact of specific architectural and training choices. The evidence is systematic:

• Text-to-text (Table 4): Qwen3.5-Omni-Plus scores within 1-2 points of Qwen3.5-Plus-Instruct across knowledge, reasoning, coding, and agentic benchmarks. On IFBench (instruction following), it actually exceeds the text-only baseline (52.6 vs 51.1). The average gap across 14 benchmarks is approximately 1-2 percentage points.

• Vision-to-text (Table 6): Qwen3.5-Omni-Plus achieves scores comparable to or exceeding the text-only model on visual understanding benchmarks. On RealWorldQA (84.1 vs 79.1), MLVU (86.8 vs 85.1), and VideoMME (81.9 vs 81.0), the omni model outperforms the text-only model. On MMMU (80.1 vs 81.0) and MathVista (86.1 vs 86.9), the gaps are negligible.

• Audio-to-text (Table 5): Qwen3.5-Omni-Plus achieves state-of-the-art on MMAU (82.2), MMSU (82.8), VoiceBench (93.1), and ASR across languages, demonstrating that adding vision and agentic capabilities does not degrade audio performance.

The conceptual significance is that this finding shifts the burden of proof. Before this paper, the default assumption was that multimodal training causes forgetting, and the research question was "how do we mitigate it?" After this paper, the question becomes "under what specific conditions does multimodal training not cause forgetting, and how do we replicate those conditions?" The paper's implicit answer — revealed through its training recipe — includes three key conditions: (1) locked LLM parameters during initial encoder alignment (Stage 1 pretraining), preventing early-stage multimodal gradients from disturbing the base LLM's text representations; (2) on-policy distillation (OPD) to close modality-specific quality gaps without degrading the source modality; and (3) interaction-aligned RL that optimizes for multimodal interaction quality without backpropagating through the modality encoders in a way that would degrade their individual capabilities.

This is a fundamental empirical finding with significant practical implications. If the tradeoff is not fundamental, then the path to omnimodal models is much simpler than previously thought: start with a strong text-only model, carefully add modality encoders with a frozen backbone, use distillation to transfer capabilities across modalities, and apply RL for interaction quality. The paper does not claim this recipe is optimal or universally applicable, but it demonstrates that the ceiling — where multimodal training causes no degradation — is achievable, which changes what researchers and practitioners should aim for.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a broad constellation of benchmarks rather than a single primary dataset, spanning five input-output modalities. For text→text, the suite includes MMLU-Pro, MMLU-Redux, SuperGPQA, C-Eval, IFEval, IFBench, AA-LCR, LongBench v2, GPQA, LiveCodeBench v6, HMMT Nov 25, IMOAnswerBench, BFCL-V4, and TAU2Bench — all standard academic benchmarks of knowledge, reasoning, and agent capability with fixed test splits. For audio→text, evaluation covers four domains: audio understanding (MMAU, MMAR, MMSU, RUL-MuchoMusic, SongFormBench), end-to-end speech dialogue (VoiceBench, URO-Bench-pro, SpeechRole, WildSpeech-Bench), speech-to-text translation (Fleurs top 59 languages into English and Chinese), and automatic speech recognition (Fleurs, Common Voice, LibriSpeech, WenetSpeech, KeSpeech, Opencpop-test, MIR-1K). For vision→text, benchmarks span mathematical/STEM reasoning (MMMU, MMMU-Pro, MathVista, MathVision, DynaMath, ZEROBench), general VQA (RealWorldQA, MMStar, HallusionBench, SimpleVQA), document understanding (CharXiv, CC-OCR, AI2D, MMLongBench-Doc, OCRBench), spatial intelligence (ERQA, CountBench, RefCOCO, ODInW13, EmbSpatialBench), video understanding (Video-MME, MLVU, MVBench, LVBench, MMVU, MME-VideoOCR), and medical VQA (SLAKE, PMC-VQA, MedXpertQA-MM). For audio-visual→text, text-query benchmarks include DailyOmni, WorldSense, AVUT, AV-SpeakerBench, and VideoMME with audio; audio-query evaluation uses Qualcomm IVD; captioning uses OmniCloze; tool use uses OmniGAIA. For X→speech, zero-shot TTS is evaluated on SEED, multilingual TTS on public and internal test sets built from Fleurs and Zhang et al. (2025a), cross-lingual TTS on CV3-Eval, and custom-voice TTS on the same multilingual test sets. The paper reports benchmark scores on standard test or validation splits as defined by each benchmark, without indicating any custom splits beyond the internal multilingual test set.

• Base model(s). Two variants of Qwen3.5-Omni are evaluated: Qwen3.5-Omni-Plus (hundreds of billions of parameters — the paper does not report exact parameter counts, but notes the "substantial scale difference" between variants in Section 2.5) and Qwen3.5-Omni-Flash (a smaller, faster variant). Both are instruct models initialized from the Qwen3.5 base checkpoint, with the Thinker inheriting Qwen3.5 parameters, the vision encoder adopted from Qwen3.5, and the audio encoder trained from scratch as AuT. The specific parameter counts and detailed architecture dimensions are not disclosed, which limits the ability to do precise FLOPs accounting or to compare scaling behavior across model sizes. The paper argues the model is representative of frontier omnimodal capabilities and positions it primarily against Gemini-3.1 Pro for audio and audio-visual tasks, and against Qwen3.5-Plus-Instruct (the text-only counterpart) for text and vision tasks. For speech generation, comparisons include both academic TTS systems (CosyVoice 2/3, F5-TTS, MaskGCT, Spark TTS) and commercial APIs (ElevenLabs, MiniMax-Speech, Gemini-2.5 Pro-Preview-TTS, GPT-Audio).

• Metrics. The paper employs a heterogeneous set of metrics depending on the benchmark. For knowledge, reasoning, and VQA tasks (MMLU-Pro, MMMU, MathVista, etc.), the primary metric is accuracy (↑, higher is better). For ASR tasks (Fleurs, Common Voice, LibriSpeech, etc.), the metric is Word Error Rate (WER, ↓, lower is better), with Character Error Rate (CER, ↓) used for italicized languages in Table 13 where word boundaries are ambiguous (e.g., Chinese, Japanese, Cantonese). For speech-to-text translation, the metric is BLEU (↑). For speech generation, content consistency is measured by WER or CER on ASR-transcribed generated speech (lower is better), and speaker similarity is measured by cosine similarity between speaker embeddings of generated and reference speech (SIM, ↑). For the SongFormBench benchmark, three metrics are reported: accuracy (acc), hit rate at 0.5 F1 (hr.5f), and hit rate at 3-frame tolerance (hr3f). For the URO-Bench-pro dialogue benchmark, the paper reports three separate dimensions: Understanding (U), Reasoning (R), and Oral Conversation (O). Some benchmarks (ZEROBench, OmniGAIA) have custom scoring protocols (e.g., judge-model evaluation using DeepSeek-V3.2-Thinking for OmniGAIA). The paper does not report confidence intervals, standard deviations, or statistical significance tests for any metric, which is standard for technical reports of this scale but limits the ability to assess whether small differences (e.g., 85.9 vs 86.8 on MMLU-Pro) are statistically meaningful.

• Baselines. The paper compares against a diverse set of models, varying by evaluation domain. For text→text, the primary baseline is Qwen3.5-Plus-Instruct (Team, 2026), the text-only counterpart at the same scale — this is the key comparison for testing whether multimodal training degrades text capabilities. For audio→text, the primary baseline is Gemini-3.1 Pro (Comanici et al., 2025), with Gemini-3-Flash and GPT-4o-Transcribe as secondary comparisons for ASR and translation tasks. For vision→text, the baseline is again Qwen3.5-Plus-Instruct, with other vision-language models not systematically compared. For audio-visual→text, the baseline is Gemini-3.1 Pro, with OmniGAIA additionally evaluated against an unspecified baseline (the paper only reports absolute scores, not relative gains). For zero-shot speech generation, baselines include a wide range of academic TTS systems: Seed-TTS (ICL and RL variants; Anastassiou et al., 2024), MaskGCT (Wang et al., 2024c), E2 TTS (Eskimez et al., 2024), F5-TTS (Chen et al., 2024c), Spark TTS (Wang et al., 2025b), CosyVoice 2 (Du et al., 2024b), CosyVoice 3 (Du et al., 2025), MiniMax-Speech (Zhang et al., 2025a), MiMo-Audio-7B-Instruct (Zhang et al., 2025c), Qwen2.5-Omni-7B (Xu et al., 2025a), and Qwen3-Omni-30B-A3B (Xu et al., 2025b). For multilingual speech generation, baselines are MiniMax-Speech and ElevenLabs (commercial systems accessed through their APIs). For custom-voice speech generation, baselines include ElevenLabs Multilingual v2, Gemini-2.5 Pro-Preview-TTS (Achernar), GPT-Audio-2025-08-28 (Alloy), and MiniMax-Speech-2.8-HD, all accessed through official APIs in March 2026.

• Generation budget / compute accounting. The paper does not standardize on a single compute metric across evaluation domains. For text, vision, and audio understanding tasks, models are evaluated at whatever generation budget each benchmark protocol specifies (typically single greedy or sampled generation, though exact decoding parameters are not specified for most benchmarks). For latency and throughput analysis (Section 2.5, Tables 1 and 2), compute is measured in milliseconds (first-packet latency, per-token latency), tokens per second (throughput), and Generation RTF (Real-Time Factor = computation time / audio duration). For speech generation, the paper does not control for total FLOPs or parameters across compared models — academic TTS models range from hundreds of millions to tens of billions of parameters, and commercial APIs like ElevenLabs and GPT-Audio are black boxes with unknown parameter counts and inference budgets. This makes the speech generation comparisons essentially capability benchmarks rather than efficiency-matched comparisons — the paper demonstrates that Qwen3.5-Omni achieves strong speech generation quality, but does not claim it does so with less compute than prior systems.

• Cross-validation / statistical protocol. The paper reports no cross-validation, statistical significance testing, or confidence intervals. For the compute-optimal strategy selection that was central to Section 3's methodology, no analogous cross-validation is described for the evaluation-stage comparisons (e.g., the model selection that produced the specific Plus and Flash checkpoints being evaluated was not done via cross-validation on a held-out split). The only exception is the OmniGAIA benchmark, where the paper specifies that results are "evaluated using DeepSeek-V3.2-Thinking as the judge" and evaluated "without a thinking prompt and without <answer> formatting" — these are evaluation protocol details, not statistical protocols. For the internal multilingual test sets, no details are provided about construction, size, or speaker distribution, making replication impossible. The latency numbers in Table 2 are described as "theoretical first-packet latency" (emphasis added) and "evaluated on internal vLLM" — they represent measured performance on specific infrastructure rather than guaranteed performance, and the paper explicitly warns that "due to the substantial scale difference between Qwen3.5-Omni-Flash and Qwen3.5-Omni-Plus, the two variants adopt different deployment-time resource allocation and parallelization strategies; therefore, their latency and throughput numbers are not intended for strict horizontal comparison."

Main Quantitative Results

Text→Text: Omnimodal Training Preserves Text Capabilities

The central claim for text performance is that Qwen3.5-Omni-Plus achieves capabilities "on par with its text-only counterpart across multiple dimensions" (Section 5.1.1). Table 4 provides the evidence, comparing Qwen3.5-Omni-Plus and Qwen3.5-Omni-Flash against Qwen3.5-Plus-Instruct across 14 benchmarks grouped into six categories. The headline numbers:

• Knowledge: MMLU-Pro 85.9 vs 86.8 (gap: −0.9 points), MMLU-Redux 94.2 vs 94.3 (−0.1), SuperGPQA 66.4 vs 67.4 (−1.0), C-Eval 92.0 vs 92.3 (−0.3). The average gap across knowledge benchmarks is approximately 0.6 percentage points.
• Instruction Following: IFEval 89.7 vs 89.7 (tied), IFBench 52.6 vs 51.1 (+1.5 points). The omni model equals or exceeds the text-only baseline on both benchmarks, with a notable advantage on IFBench.
• Long Context: AA-LCR 57.0 vs 62.0 (−5.0), LongBench v2 59.6 vs 60.2 (−0.6). The largest gap appears on AA-LCR, where the omni model trails by 5 points.
• STEM: GPQA 83.9 vs 85.9 (−2.0), a moderate gap on this graduate-level science benchmark.
• Reasoning: LiveCodeBench v6 65.6 vs 67.1 (−1.5), HMMT Nov 25 84.4 vs 86.2 (−1.8), IMOAnswerBench 65.5 vs 68.3 (−2.8). The gaps are consistent but modest, with the largest on the math competition benchmark.
• General Agent: BFCL-V4 63.3 vs 66.1 (−2.8), TAU2Bench 81.0 vs 82.7 (−1.7).

Across all 14 benchmarks, Qwen3.5-Omni-Plus trails the text-only baseline on 12 of 14, ties on 1, and exceeds on 1. The average gap is approximately 1.5 percentage points. This is remarkably small for a model that adds full audio, vision, and speech generation capabilities — it supports the claim that multimodal training does not substantially degrade text capabilities. However, the consistency of the direction (trailing on 12/14 benchmarks) suggests there is a small but systematic cost to multimodal training, even if the magnitude is practically negligible for most applications.

The Flash variant shows substantially larger gaps, trailing by roughly 5-12 points across most benchmarks (e.g., MMLU-Pro 79.9 vs 86.8, GPQA 76.4 vs 85.9, LiveCodeBench 56.6 vs 67.1), reflecting the expected capacity tradeoff of a smaller model.

Audio→Text: SOTA on Understanding, Dialogue, Translation, and ASR

Table 5 provides the comprehensive audio-to-text comparison against Gemini-3.1 Pro, with ASR results also compared against GPT-4o-Transcribe and Gemini-3-Flash in the detailed Table 13 (Appendix). The paper claims Qwen3.5-Omni-Plus "surpass[es] Gemini-3.1 Pro in key audio tasks" (Abstract) and achieves SOTA results. The evidence:

Audio Understanding: Qwen3.5-Omni-Plus leads Gemini-3.1 Pro on 5 of 6 reported benchmarks/metric clusters:

• MMAU: 82.2 vs 81.1 (+1.1)
• MMSU: 82.8 vs 81.3 (+1.5)
• RUL-MuchoMusic: 72.4 vs 59.6 (+12.8 — the largest margin in this category)
• SongFormBench-HarmonixSet: 81.1 | 72.9 | 85.3 vs 75.6 | 46.8 | 77.9 (all three metrics substantially higher)
• SongFormBench-CN: 87.1 | 65.7 | 84.2 vs 78.1 | 43.2 | 71.9
• MMAR: 80.0 vs 83.7 (−3.7 — the only benchmark where Gemini-3.1 Pro leads)

The Flash variant is competitive but generally trails both Plus and Gemini-3.1 Pro on understanding metrics.

Dialogue: Qwen3.5-Omni-Plus leads on 3 of 4 benchmarks:

• VoiceBench: 93.1 vs 88.9 (+4.2)
• URO-Bench-pro: 66.3 | 86.3 | 99.8 vs 69.1 | 84.0 | 99.2 (trails on Understanding by 2.8, leads on Reasoning by 2.3 and Oral by 0.6)
• SpeechRole: 123.5 vs 124.2 (−0.7, essentially tied)
• WildSpeech-Bench: 75.4 vs 76.3 (−0.9, essentially tied)

S2TT (Speech-to-Text Translation): Qwen3.5-Omni-Plus leads across all three aggregate metrics:

• Fleurs xx↔zh (top 59): 30.2 vs 29.5 (+0.7 BLEU)
• Fleurs xx↔en (top 59): 35.4 vs 34.6 (+0.8 BLEU)
• Fleurs xx↔zh/en (top 59): 32.8 vs 32.1 (+0.7 BLEU)

The margins are consistent but small — approximately 0.7-0.8 BLEU across all three aggregates. The detailed Tables 14 and 15 (Appendix) provide per-language breakdowns, where the pattern is more nuanced: Qwen3.5-Omni-Plus leads on 40 of 59 en2xx pairs (Table 14), but the average BLEU advantage (33.8 vs 31.8) is driven partly by very large margins on specific Asian languages like Cantonese (40.1 vs 25.5, +14.6 BLEU) and Javanese (26.8 vs 16.4, +10.4 BLEU). On many European language pairs, the models perform similarly or Gemini-3.1 Pro leads slightly (e.g., Greek en2xx: 30.0 vs 30.0 tied; Swedish xx2en: 47.2 vs 46.6). This pattern supports the paper's claim of strong performance on Asian languages specifically, but the overall average advantage is partly an artifact of very large wins on a few languages rather than consistent small wins across all languages.

ASR: Qwen3.5-Omni-Plus achieves an average WER of 6.6% across the Fleurs top-60 languages (Table 5, with detailed per-language results in Table 13), versus 7.3% for Gemini-3.1 Pro, 10.4% for GPT-4o-Transcribe, and 10.8% for Qwen3.5-Omni-Flash. The model achieves the best WER on the majority of individual languages in Table 13. Particularly notable results include:

• Cantonese: 2.2% vs 6.3% for Gemini-3.1 Pro (−4.1 points absolute, a 65% relative reduction)
• KeSpeech (Chinese dialects): 3.46% vs 23.67% for Gemini-3.1 Pro (−20.21 points absolute)
• WenetSpeech: 4.30% | 5.84% (net|meeting) vs Gemini-3.1 Pro's 11.53% | 14.21%

The Flash variant achieves 10.8% average WER on Fleurs, which is competitive with Gemini-3-Flash (10.5%) while dramatically outperforming it on specific languages like Cantonese (3.1% vs 10.8%) and KeSpeech (4.47% vs not reported, but GPT-4o-Transcribe gets 28.1% on a subset).

Summary assessment: The audio→text results strongly support the claim that Qwen3.5-Omni-Plus achieves SOTA on audio understanding and ASR. The margins are largest on multilingual and dialectal speech, consistent with the paper's emphasis on expanded multilingual training data. On speech translation and dialogue, the advantages are present but smaller, and on some metrics (MMAR, URO-Bench-pro Understanding, WildSpeech-Bench) Gemini-3.1 Pro maintains a lead or ties.

Vision→Text: Comparable to Text-Only Counterpart, With Video Understanding Advantages

Table 6 compares Qwen3.5-Omni-Flash and Qwen3.5-Omni-Plus against Qwen3.5-Plus-Instruct across 28 vision benchmarks grouped into six categories. The headline pattern mirrors the text→text results: small gaps on most benchmarks, with Plus occasionally exceeding the text-only baseline.

STEM and Puzzle: Qwen3.5-Omni-Plus trails Qwen3.5-Plus-Instruct on 6 of 7 benchmarks/metrics, with an average gap of about 1-2 points (MMMU 80.1 vs 81.0, MathVista 86.1 vs 86.9, DynaMath 83.8 vs 84.2) and one tie on ZEROBench (5 vs 6 — both extremely low on this "impossible" benchmark). The gap is largest on MathVision (73.0 vs 73.6) and MMMU-Pro (73.9 vs 73.8, which is essentially tied). ZEROBench_sub, a more granulated version of ZEROBench, actually shows a plus for the omni model (34.4 vs 31.1, +3.3 points).

General VQA: Qwen3.5-Omni-Plus leads on 2 of 5 benchmarks and trails on 3:

• RealWorldQA: 84.1 vs 79.1 (+5.0 points — the largest advantage in the vision table)
• MMStar: 79.4 vs 80.3 (−0.9)
• MMBenchEN-DEV-v1.1: 92.8 vs 93.8 (−1.0)
• SimpleVQA: 65.3 vs 66.1 (−0.8)

The large RealWorldQA advantage is notable because this benchmark tests understanding of real-world scenes — a domain where audio-visual training might provide complementary information that pure vision models lack.

Text Recognition and Document Understanding: Qwen3.5-Omni-Plus trails or ties on all 5 benchmarks, with gaps of approximately 0-2 points: CharXiv 72.5 vs 74.2 (−1.7), CC-OCR 83.4 vs 83.0 (+0.4), AI2D 91.2 vs 92.1 (−0.9), MMLongBench-Doc 57.5 vs 59.7 (−2.2), OCRBench 91.3 vs 91.4 (−0.1).

Spatial Intelligence: Mixed results — Plus leads on 2 of 5, ties on 2, and trails on 1: CountBench 95.1 vs 95.1 (tie), ERQA 54.8 vs 53.8 (+1.0), RefCOCO 95.0 vs 95.2 (−0.2, tie), ODInW13 49.5 vs 50.3 (−0.8), EmbSpatialBench 85.4 vs 83.4 (+2.0).

Video Understanding: This is where the omni model shows a consistent advantage. Qwen3.5-Omni-Plus leads on all 6 video benchmarks:

• VideoMME (w/o subtitles): 81.9 vs 81.0 (+0.9)
• MLVU (M-Avg): 86.8 vs 85.1 (+1.7)
• MVBench: 79.0 vs 76.7 (+2.3)
• LVBench: 71.2 vs 68.6 (+2.6)
• MMVU: 67.5 vs 67.1 (+0.4)
• MME-VideoOCR: 77.0 vs 74.2 (+2.8)

The paper attributes this to "the effectiveness of joint video-audio training paradigms" and posits that "audio-visual streams constitute the most naturalistic representation of real-world phenomena, wherein visual and auditory modalities are intrinsically coupled" (Section 5.1.3). This is a plausible explanation — training on video with synchronized audio may provide richer learning signals than silent video alone — but the paper provides no ablation separating the effect of joint training from other factors like the 256k context window or the explicit timestamp mechanism.

Medical VQA: Qwen3.5-Omni-Plus leads on SLAKE (84.7 vs 82.8, +1.9), ties on PMC-VQA (62.7 vs 62.4, +0.3), and trails slightly on MedXpertQA-MM (54.7 vs 55.3, −0.6).

Summary assessment: The vision results support the claim that multimodal training does not substantially degrade vision capabilities, and may actually improve them for video understanding specifically. The average gap across all 28 numbers (treating ZEROBench sub-scores separately) is approximately 0.5 percentage points in favor of the text-only model, but with the video advantage pulling the average close to parity. The lack of comparison against other vision-language models (beyond Qwen's own text-only model) limits the ability to assess whether Qwen3.5-Omni-Plus is competitive with standalone vision-language models like GPT-4V or Gemini-2.5-Pro on vision tasks — the paper only demonstrates it's comparable to Qwen's own text-only model.

Audio-Visual Video→Text: Strong but Not Dominant

Table 7 compares Qwen3.5-Omni-Plus against Gemini-3.1 Pro across 9 audio-visual benchmarks. The results are more mixed than the audio-only comparisons:

Text-Query QA: Qwen3.5-Omni-Plus leads on 2 of 5, trails on 3:

• DailyOmni: 84.6 vs 82.7 (+1.9)
• AVUT: 85.0 vs 85.6 (−0.6, effectively tied)
• WorldSense: 62.8 vs 65.5 (−2.7)
• AV-SpeakerBench: 71.3 vs 75.1 (−3.8)
• VideoMME with audio: 83.7 vs 89.0 (−5.3 — the largest gap in the table)

The VideoMME-with-audio gap is particularly notable given that Qwen3.5-Omni-Plus outperformed the text-only baseline on VideoMME without subtitles by 0.9 points (Table 6). The addition of audio to the VideoMME evaluation appears to favor Gemini-3.1 Pro substantially, suggesting that Gemini's audio-visual fusion may be stronger on this particular benchmark, even though Qwen3.5-Omni leads on other audio-visual metrics.

Audio-Query QA: Qwen3.5-Omni-Plus leads on Qualcomm IVD (68.5 vs 66.2, +2.3), demonstrating strength in the interactive scenario where questions are asked verbally about a shared visual context.

Captioning: Qwen3.5-Omni-Plus leads on Omni-Cloze (64.8 vs 57.2, +7.6), a substantial margin on captioning quality.

Tool Use: Qwen3.5-Omni-Plus trails Gemini-3.1 Pro significantly on OmniGAIA (57.2 vs 68.9, −11.7 points). The paper notes this result is "evaluated without a thinking prompt and without <answer> formatting" — suggesting the evaluation may disadvantage Qwen3.5-Omni-Plus relative to a model that was specifically optimized for this benchmark's format. The Flash variant scores only 33.9 on OmniGAIA, indicating tool use is a capacity-dependent capability that degrades substantially with model size.

Summary assessment: The audio-visual results are the most mixed in the paper. On DailyOmni, QUALCOMM IVD, and Omni-Cloze, Qwen3.5-Omni-Plus leads. On WorldSense, AV-SpeakerBench, VideoMME-with-audio, and OmniGAIA, Gemini-3.1 Pro leads, with the margins on VideoMME and OmniGAIA being substantial (5.3 and 11.7 points, respectively). The paper's claim that Qwen3.5-Omni-Plus "matches [Gemini-3.1 Pro] in comprehensive audio-visual understanding" (Abstract) is reasonable on some metrics but overstates on others — a more precise characterization would be "competitive on most audio-visual benchmarks with specific advantages on captioning and audio-query QA, but with notable gaps on video understanding with audio and tool use."

X→Speech: State-of-the-Art Content Fidelity Across Languages

The speech generation evaluation covers four scenarios: zero-shot TTS, multilingual TTS, cross-lingual TTS, and custom-voice TTS.

Zero-Shot Speech Generation (Table 8): On the SEED-TTS benchmark, Qwen3.5-Omni-Plus achieves WER of 0.99 (test-zh) and 1.26 (test-en). This places it among the top systems:

• CosyVoice 3: 0.71 | 1.45 (leads on test-zh, trails on test-en)
• MiniMax-Speech: 0.83 | 1.65 (leads on test-zh, trails on test-en)
• Qwen3-Omni-30B-A3B: 1.07 | 1.39 (trails on both)
• Seed-TTS (RL): 1.00 | 1.94 (trails on both)
• Qwen2.5-Omni-7B: 1.42 | 2.33 (trails substantially)

Qwen3.5-Omni-Plus achieves the best test-en WER among all compared systems (1.26) and is second-best on test-zh after CosyVoice 3 (0.99 vs 0.71). The combination of strong performance on both splits suggests robust zero-shot speech generation, though the lack of speaker similarity metrics for the SEED evaluation means we cannot assess voice cloning fidelity — the evaluation only captures whether the generated speech says the right words, not whether it sounds like the target speaker.

Multilingual Speech Generation (Tables 9 and 10): The paper evaluates on two test sets: a public multilingual TTS test set (20 languages, Table 9) and an internal multilingual test set (9 additional languages, Table 10). Against MiniMax-Speech and ElevenLabs (commercial systems):

• Content consistency (WER): Qwen3.5-Omni-Plus achieves the best WER in 21 of 29 evaluated languages across both tables. Particularly large advantages appear on languages where competitor systems struggle dramatically: Chinese (0.695 vs 2.252 MiniMax, 16.026 ElevenLabs), Thai (2.170 vs 2.701 MiniMax, 73.936 ElevenLabs — ElevenLabs essentially fails on Thai with 74% WER), Vietnamese (1.143 vs 0.880 MiniMax, 73.415 ElevenLabs), Arabic (2.602 vs 1.665 MiniMax, 1.666 ElevenLabs — MiniMax leads slightly here), and several European languages where all systems perform well (German: 0.447 vs 1.906 MiniMax, 0.572 ElevenLabs). For languages where Qwen3.5-Omni-Plus does not lead, it is typically competitive (e.g., Polish: 1.427 vs ElevenLabs' 0.766, where ElevenLabs leads).

• Speaker similarity (SIM): Qwen3.5-Omni-Plus achieves the best speaker similarity in the majority of languages, outperforming both MiniMax-Speech and ElevenLabs on most rows. The SIM scores for Qwen3.5-Omni-Plus typically range from 0.730 to 0.859, indicating strong but not perfect voice cloning fidelity.

A striking pattern: ElevenLabs fails catastrophically on several Asian languages — WER of 73.936 on Thai, 73.415 on Vietnamese, 114.813 on Thai (custom-voice, Table 12), 102.018 on Hebrew (custom-voice, Table 12), 82.849 on Vietnamese (custom-voice, Table 12) — while Qwen3.5-Omni-Plus maintains WERs in the 1-7% range on these same languages. This suggests ElevenLabs' system may not support these languages natively and may be attempting to generate them using phoneme inventories from other languages, resulting in largely unintelligible output. Qwen3.5-Omni-Plus's consistent performance across all 29 languages indicates genuine multilingual speech generation capability rather than English-centric generation with accent adaptation.

Cross-Lingual Speech Generation (Table 11): On the CV3-Eval cross-lingual benchmark, Qwen3.5-Omni-Plus is compared against CosyVoice 2, CosyVoice 3, and Qwen3-Omni-30B-A3B across 12 source-target language pairs. The metric is mixed error rate (WER for English-target, CER for other targets, both ↓):

• Qwen3.5-Omni-Plus achieves the best result on 10 of 12 pairs.
• The largest advantage is on zh-to-ko (Chinese-to-Korean): 4.03 vs 14.4 for CosyVoice 3 (a 72% relative reduction, as highlighted in the Abstract) and 24.8 for CosyVoice 2. The en-to-ko pair shows a similar pattern: 3.72 vs 5.87 (CosyVoice 3) and 21.9 (CosyVoice 2).
• On ja-to-zh (Japanese-to-Chinese), CosyVoice 3 leads slightly (3.05 vs 3.55).
• On en-to-zh (English-to-Chinese), Qwen3.5-Omni-Plus leads (4.86 vs 5.09 CosyVoice 3, 13.5 CosyVoice 2).

The cross-lingual task is particularly demanding because the model must preserve speaker identity while generating speech in a language the reference speaker may never have spoken. Qwen3.5-Omni-Plus's strong performance, especially on Korean-target pairs where prior systems struggled dramatically (CosyVoice 2: 24.8 CER on zh-to-ko), suggests effective cross-lingual voice transfer.

Custom-Voice Speech Generation (Table 12): After lightweight speaker fine-tuning, Qwen3.5-Omni-Plus is compared against four commercial systems (ElevenLabs, Gemini-2.5 Pro, GPT-Audio, MiniMax) on the multilingual test set (29 languages). Content consistency (WER) results:

• Qwen3.5-Omni-Plus achieves the best WER in 10 of 29 languages.
• On the remaining languages, it is typically competitive, though specific commercial systems lead on some: ElevenLabs leads on Polish (0.733 vs Qwen's 1.462), Turkish (0.551 vs 1.309), Czech (1.916 vs 3.769); GPT-Audio leads on Chinese (0.829 vs 0.785 — a very small gap).
• ElevenLabs again fails catastrophically on Thai (114.813) and Vietnamese (82.849), where Qwen3.5-Omni-Plus achieves 1.653 and 1.320 respectively.

The paper notes that Qwen3.5-Omni-Plus was "fine-tuned only on monolingual data" for the custom-voice stage, yet "demonstrates strong cross-lingual generalization." This is indeed the important result — the model can clone a speaker's voice and reproduce it in languages it never saw that speaker speak during fine-tuning. The mechanism (presumably disentangled speaker identity representations learned during the earlier Talker training stages) is not directly tested, but the generalization result itself is strong evidence for effective voice cloning.

Summary assessment: The speech generation results are the strongest in the paper, with Qwen3.5-Omni-Plus consistently ranking among the top systems on content fidelity across zero-shot, multilingual, cross-lingual, and custom-voice settings. The gap between Qwen3.5-Omni-Plus and Qwen3-Omni-30B-A3B on zero-shot TTS (test-en WER: 1.26 vs 1.39, test-zh WER: 0.99 vs 1.07) suggests genuine progress from the previous generation. The dramatic gaps between Qwen3.5-Omni-Plus and commercial systems on Asian languages (Thai, Vietnamese, Cantonese) suggest that the expanded multilingual training — particularly the 3.5:3.5:3 ratio of Chinese:English:multilingual audio data (Section 2.2) — has produced genuine multilingual speech capability rather than English-centric generation.

Ablation Studies and Robustness Checks

Comparison with text-only counterpart across modalities: The most extensive robustness check is the systematic comparison against Qwen3.5-Plus-Instruct across text (Table 4) and vision (Table 6) benchmarks. The consistent finding — gaps of approximately 0-2 percentage points across most benchmarks — provides evidence that multimodal training did not catastrophically degrade text or vision capabilities. This is not a formal ablation (no variant of Qwen3.5-Omni was trained without the multimodal components), but it serves as a strong sanity check.

Scale variants (Plus vs Flash): The paper evaluates two model variants (Plus and Flash) across most benchmarks. The consistent performance gap between them — Flash trailing by roughly 3-8 points on most text and vision benchmarks, and by similar margins on audio — provides an implicit scaling curve. The gap is largest on the most capacity-dependent tasks: OmniGAIA (33.9 Flash vs 57.2 Plus, a 23.3-point gap), MMLU-Pro (79.9 vs 85.9, a 6-point gap), and LiveCodeBench v6 (56.6 vs 65.6, a 9-point gap). This suggests that agentic and complex reasoning capabilities are the most sensitive to model scale.

Explicit timestamp vs. TMRoPE (Section 2.3): The paper claims that replacing TMRoPE with explicit text timestamps (1) avoids sparse position IDs for long video, (2) eliminates the need for uniform frame-rate sampling, and (3) improves temporal modeling. However, no ablation experiment is reported comparing the two approaches quantitatively. The evidence is purely qualitative and mechanistic — the paper explains why TMRoPE fails (sparse IDs) and why explicit timestamps should work better (natural language understanding of time), but does not report accuracy on temporal reasoning benchmarks with and without the change. This is a significant omission for what the paper presents as a key architectural improvement.

ARIA vs. dual-track Talker (Section 2.4): Similarly, ARIA is presented as the solution to the text-speech rate mismatch problem, but no direct comparison between Qwen3.5-Omni with ARIA and an otherwise-identical model with the Qwen3-Omni dual-track design is reported. The paper compares Qwen3.5-Omni-Plus against Qwen3-Omni-30B-A3B on speech generation (Tables 8 and 11), which shows improvement, but these models differ in many respects beyond the Talker design (Hybrid MoE architecture, 256k context, expanded multilingual training, improved post-training). It is impossible to attribute the speech quality improvements specifically to ARIA versus these other factors. The paper's argument for ARIA is strong mechanistically (the rate mismatch is a real problem, and ARIA's constraint is a principled solution), but the empirical validation is weak: we see that speech quality improved from Qwen3-Omni to Qwen3.5-Omni, but we cannot tell how much of that improvement came from ARIA versus the other upgrades.

On-policy distillation (OPD) effect on instruction following (Section 4.1, Stage 2): The paper claims that "OPD and interaction-aligned RL have a positive effect on improving the instruction-following capabilities of an omni-model LLM" (Section 5.1.1), citing the IFBench score where Qwen3.5-Omni-Plus exceeds the text-only baseline (52.6 vs 51.1). This is suggestive but circumstantial — no ablation is reported with OPD removed to isolate its effect. The improved instruction following could equally be attributed to the expanded training data, the interaction-aligned RL, or the specialist distillation stage.

Cross-lingual voice cloning generalization (Table 12): The paper makes a strong claim that "Qwen3.5-Omni is fine-tuned only on monolingual data" yet "demonstrates strong cross-lingual generalization in custom-voice speech generation." This is supported by the Table 12 results, where the model achieves competitive WER in languages for which it had no custom-voice fine-tuning data. However, this is not a controlled ablation — it's an observation about the model's behavior. The paper does not compare against a model fine-tuned on multilingual custom-voice data to determine whether monolingual fine-tuning is actually sufficient (perhaps multilingual fine-tuning would be even better) or whether the generalization is fragile in some way.

Multilingual ASR per-language breakdown (Table 13): The detailed 60-language ASR table serves as an effective robustness check for the claim of strong multilingual speech recognition. The pattern is clear: Qwen3.5-Omni-Plus achieves the best WER on the majority of languages, with the largest advantages on Asian languages and Chinese dialects. The Flash variant, while trailing overall, achieves competitive or superior performance on specific languages (Chinese 2.9 vs 2.6 for GPT-4o-Transcribe, Cantonese 3.1 vs 10.8 for Gemini-3-Flash), suggesting that even the smaller model benefits from the multilingual training data.

Concurrency scaling latency (Table 2): The latency numbers at different concurrency levels (1, 4, 8) provide an implicit robustness check for the streaming architecture. Key observations:

• Overall latency approximately doubles from concurrency 1 to 8 (e.g., Flash audio: 235→352ms, Plus audio: 435→955ms), rather than scaling linearly by 8×. This indicates effective batching amortization.
• Thinker TPS scales substantially (Flash: 177→942, Plus: 57→266), confirming the Hybrid MoE architecture's batching efficiency.
• Generation RTF stays well below 1.0 even at concurrency 8 (Flash: 0.257, Plus: 0.334), confirming real-time feasibility.

The paper does not report latency numbers beyond concurrency 8, so the ceiling of the batching efficiency is unknown — the system might degrade sharply at higher concurrency if KV-cache memory becomes the bottleneck.

ZEROBench performance as a capability ceiling detector: Both Qwen3.5-Omni-Plus and Qwen3.5-Plus-Instruct score extremely low on ZEROBench (5 and 6, respectively). This benchmark is designed to be "impossible" for current models, so the uniformly low scores serve as a sanity check: neither model is exploiting benchmark-specific shortcuts that would inflate scores on standard benchmarks while failing on genuinely hard ones. The ZEROBench_sub scores (34.4 for Omni vs 31.1 for text-only) are more informative and suggest the omni model may have a slight advantage on certain sub-categories, though the absolute scores remain low.

Critical Assessment

The paper makes several central claims. I walk through each and assess whether the reported experiments genuinely support them, where the evidence is strong, and where it is incomplete.

Claim from the Abstract: "Qwen3.5-Omni-Plus achieves SOTA results across 215 audio and audio-visual understanding, reasoning, and interaction subtasks and benchmarks, surpassing Gemini-3.1 Pro in key audio tasks and matching it in comprehensive audio-visual understanding."

The "215 subtasks and benchmarks" claim is difficult to verify independently because the paper doesn't enumerate all 215. Looking at the reported tables: Table 5 (audio) has roughly 20 aggregate metrics but the detailed Appendix tables (13, 14, 15) contain per-language breakdowns that collectively account for hundreds of individual language-benchmark pairs. The "surpassing Gemini-3.1 Pro in key audio tasks" is well-supported: Qwen3.5-Omni-Plus leads on MMAU (82.2 vs 81.1), MMSU (82.8 vs 81.3), VoiceBench (93.1 vs 88.9), and ASR across most languages. The "matching it in comprehensive audio-visual understanding" is a weaker claim justified by the mixed Table 7, where Qwen3.5-Omni-Plus leads on DailyOmni, AVUT (effectively tied), QUALCOMM IVD, and Omni-Cloze, but trails on WorldSense, AV-SpeakerBench, and VideoMME-with-audio by margins of 2.7–5.3 points. A more precise characterization would acknowledge the specific benchmarks where Gemini-3.1 Pro maintains a clear lead, particularly VideoMME-with-audio (89.0 vs 83.7, a 5.3-point gap) and OmniGAIA (68.9 vs 57.2, an 11.7-point gap). The claim is supported on the subset of audio-visual benchmarks that test captioning and audio-query understanding but overstated on video understanding with audio and tool use, where Gemini-3.1 Pro leads substantially.

Claim from Section 5.1.1: "Qwen3.5-Omni-Plus demonstrates text capabilities that are on par with its text-only counterpart across multiple dimensions."

This claim is largely supported by Table 4. Across 14 benchmarks, the average gap is approximately 1.5 percentage points in favor of the text-only model. This is close enough to "on par" for most practical purposes, and the one benchmark where the omni model leads (IFBench) partially offsets the small deficits elsewhere. However, three caveats are worth noting. First, there is a systematic direction to the gaps — the omni model trails on 12 of 14 benchmarks, which suggests multimodal training exacts a small but consistent cost even if the magnitude is small. Second, the largest gaps appear on the most reasoning-intensive benchmarks (IMOAnswerBench: −2.8, BFCL-V4: −2.8, GPQA: −2.0), suggesting that complex reasoning may be more affected than knowledge recall. Third, and most importantly, these comparisons are against Qwen's own text-only model, not against other frontier text-only models. We cannot conclude from Table 4 that Qwen3.5-Omni-Plus is competitive with, say, Claude 3.5 Sonnet or GPT-4o on text tasks — only that it hasn't lost much relative to Qwen's own text-only baseline.

Claim from Section 5.1.3: "Qwen3.5-Omni-Plus achieves performance comparable to that of Qwen3.5-Plus-Instruct, while demonstrating stronger results on video understanding tasks."

The second half of this claim — stronger video understanding — is the best-supported claim in the vision section. Qwen3.5-Omni-Plus leads on all 6 video benchmarks (VideoMME, MLVU, MVBench, LVBench, MMVU, MME-VideoOCR), with margins of 0.4–2.8 points. This is consistent and suggestive of a genuine benefit from joint video-audio training. However, the mechanism is not empirically isolated: the omni model also has the explicit timestamp mechanism, a 256k context window, and different training data composition. Any of these could contribute to the video advantage independent of the audio modality. A clean ablation would compare against a model trained with the same architecture and data but without the audio stream during video training — this would isolate whether the audio signal itself provides the advantage or whether other factors explain it. Without this ablation, the claim that "audio-visual streams constitute the most naturalistic representation" (Section 5.1.3) is a plausible hypothesis rather than a demonstrated finding.

Claim about ARIA from Abstract and Section 2.4: "ARIA dynamically aligns text and speech units, significantly enhancing the stability and prosody of conversational speech with minimal latency impact."

The latency impact claim is supported by the TTFC numbers (Table 2): 54-56ms for Plus, 56-61ms for Flash, which are low enough to be compatible with real-time conversation. However, the stability and prosody enhancement claims are not directly tested. The paper compares final speech generation quality (Tables 8-12) against prior systems, which shows improvement, but cannot attribute this to ARIA specifically because too many variables changed simultaneously between Qwen3-Omni and Qwen3.5-Omni (architecture, training data, post-training, model scale). To genuinely test ARIA's contribution, the paper would need to report an ablation comparing Qwen3.5-Omni with ARIA against an otherwise-identical model using Qwen3-Omni's dual-track Talker design. No such experiment is reported. The mechanistic argument for ARIA is strong, but the empirical validation is indirect and confounded.

Claim about explicit timestamps from Section 2.3: replacing TMRoPE with explicit text timestamps enables "more precise and robust temporal perception, especially when extrapolating long-context multimodal inputs."

Like ARIA, this claim is mechanistically well-motivated but empirically untested. No benchmark is reported that specifically tests temporal reasoning with long audio-visual inputs. The video understanding advantage (Table 6) is consistent with better temporal perception, but could equally reflect other factors. The paper would be strengthened by a dedicated temporal reasoning benchmark (e.g., "At what time did the speaker mention X?" or "How long after the door closed did the dog bark?") with a direct comparison against the TMRoPE variant.

Claim about multilingual generalization from Section 5.2.2 and 5.2.4: the model generalizes across languages for speech generation, including languages not seen during fine-tuning.

This is the best-supported cross-domain generalization claim in the paper. The evidence is strong:

• Multilingual TTS (Tables 9, 10): The model achieves the best WER on 21 of 29 languages, with dramatic advantages on specific languages where competitor systems fail.
• Cross-lingual TTS (Table 11): The model leads on 10 of 12 language pairs, with a 72% relative reduction on zh-to-ko compared to CosyVoice 3.
• Custom-voice TTS (Table 12): After monolingual fine-tuning, the model generalizes to 29 languages, achieving competitive or best WER on most.

The large number of languages (29) and the consistency of the advantage across diverse language families (Indo-European, Sino-Tibetan, Japonic, Koreanic, Tai-Kadai, Austronesian, Afroasiatic, Uralic, Turkic, Dravidian) provide robust evidence of genuine multilingual capability rather than language-specific optimization. The failure of commercial systems on specific Asian languages (ElevenLabs: 73-114% WER on Thai and Vietnamese) further highlights that Qwen3.5-Omni's multilingual speech generation is not merely competitive but addresses a genuine gap in existing systems.

A note on what's missing: The paper does not evaluate the model on several dimensions that would be important for practitioners:

• ASR in noisy conditions: All ASR benchmarks use relatively clean speech (LibriSpeech, Common Voice, Fleurs). Real-world deployment requires robustness to background noise, overlapping speech, and acoustic variability.
• Long-form audio understanding beyond ASR: While the 10-hour audio context is highlighted, the paper doesn't evaluate tasks that require reasoning over very long audio spans (e.g., summarizing a multi-hour meeting, finding specific moments in a long podcast, analyzing musical structure over an entire symphony).
• Audio-visual speech recognition: The combination of lip reading and audio for ASR in noisy conditions — a natural capability for an omnimodal model — is not evaluated.
• Emotion recognition and generation controllability: Despite claiming "human-like emotional nuance" and controllable emotion in speech generation (Section 1), no quantitative evaluation of emotion recognition accuracy or emotion generation fidelity is reported.
• Human evaluation for speech naturalness: All speech evaluations are automatic metrics (WER, CER, speaker similarity). The paper does not report subjective human evaluations of naturalness, which would be the gold standard for speech generation quality.
• Failure mode analysis: Beyond the acknowledgement that ReST-EM-trained revision models degraded (from the prior sections), the paper provides no systematic analysis of when or why Qwen3.5-Omni fails — what kinds of inputs cause hallucination, what accents or dialects are poorly handled, what types of reasoning break down.

In summary, the experimental evidence strongly supports the paper's claims about multilingual speech generation quality and the preservation of text/vision capabilities. The evidence is weaker for the specific architectural innovations (ARIA, explicit timestamps, OPD), where causal attribution is confounded by multiple simultaneous changes and the absence of controlled ablations. The audio-visual understanding claims are partially supported but overstated on the benchmarks where Gemini-3.1 Pro leads. The paper would be substantially strengthened by component-level ablations isolating the contribution of each architectural change, dedicated long-form temporal reasoning benchmarks, human evaluation of speech naturalness, and systematic failure analysis.

6. Limitations and Trade-offs

No Controlled Ablations for the Core Architectural Claims

The assumption or constraint. The paper presents three major architectural innovations — ARIA for dynamic text-speech alignment (Section 2.4), explicit text timestamps replacing TMRoPE (Section 2.3), and on-policy distillation (OPD) for closing the audio-text quality gap (Section 4.1) — as key contributors to Qwen3.5-Omni's performance. However, none of these innovations is evaluated through a controlled ablation experiment that isolates its contribution by holding all other variables constant. The paper compares Qwen3.5-Omni against Qwen3-Omni on speech generation (Tables 8 and 11) and shows improvement, but these models differ along multiple axes simultaneously: Hybrid MoE architecture (replacing dense transformers), 256k context window (replacing 32k), expanded multilingual training data (113 languages for ASR vs. the prior scope, though the exact Qwen3-Omni number is unspecified), ARIA (replacing dual-track Talker), explicit timestamps (replacing TMRoPE), OPD (new post-training stage), and interaction-aligned RL (new post-training stage). From this comparison, it is impossible to determine how much of the improvement comes from ARIA versus, say, the expanded multilingual training data or the larger model scale.

The consequence. Practitioners who want to adopt specific components of Qwen3.5-Omni's architecture — for instance, implementing ARIA in a different omnimodal system — cannot assess whether the mechanism provides benefits independently of the other changes. The paper's mechanistic arguments are strong: the text-speech rate mismatch is a real architectural pathology, and ARIA's cumulative ratio constraint is a principled solution. But engineering decisions require knowing the magnitude of the benefit, not just its existence. If ARIA contributes only a 0.1 WER improvement on speech generation while the multilingual data expansion contributes a 1.0 WER improvement, then a team with already-strong multilingual data might prioritize other investments. Without ablations, the paper's implicit claim — that each architectural innovation mattered — remains a hypothesis rather than a demonstrated finding.

What evidence exists in the paper. The speech generation benchmarks (Tables 8–12) show Qwen3.5-Omni-Plus outperforming Qwen3-Omni-30B-A3B substantially: on SEED test-en, WER improves from 1.39 to 1.26 (Table 8); on cross-lingual TTS, error rates drop dramatically (e.g., zh-to-ko from 5.13 CER to 4.03, Table 11). The latency analysis (Table 2) shows that ARIA's single-stream formulation maintains low Talker TTFC (54–95ms), consistent with the claim that the alignment mechanism does not introduce meaningful latency overhead. However, these results show that the whole system improved, not that ARIA specifically caused the improvement. The paper provides no comparison of, for example, Qwen3.5-Omni's ARIA-based Talker against an otherwise-identical model using Qwen3-Omni's dual-track design. Similarly, no temporal reasoning benchmark is reported that isolates the effect of explicit timestamps relative to TMRoPE.

Mitigation status. The paper acknowledges none of these missing ablations. It does not flag the confounded comparison as a limitation. The architectural innovations are presented as contributions, and the downstream performance improvements are offered as implicit validation — but a causal link is never established. This is a significant methodological gap for a paper whose primary claims center on novel architectural mechanisms.

Cross-Modal Quality Gaps Remain Despite OPD

The assumption or constraint. On-policy distillation (OPD, Section 4.1) is designed to close the quality gap between audio-conditioned and text-conditioned reasoning by using the model's own text-conditioned generations as training targets for audio-conditioned inputs. The paper presents this as an effective technique and notes that it "gradually aligns its audio-conditioned outputs with its text-conditioned behavior." However, the paper does not report any direct measurement of the residual gap after OPD — i.e., how much worse are audio-conditioned responses than text-conditioned responses for the same queries in the final model? The text-to-text benchmark parity (Table 4) demonstrates that multimodal training did not substantially degrade the model's text capabilities, but parity on text-input benchmarks says nothing about whether audio-input reasoning has caught up to text-input reasoning.

The consequence. In practical deployment, if a user speaks a complex math problem to the model rather than typing it, do they get a measurably worse answer? If OPD significantly narrowed but did not close the gap, then audio-input users — who are the primary users of a voice-interactive system — are receiving systematically lower-quality reasoning than text-input users. This matters enormously for fairness and user experience: a voice-first assistant that reasons worse than its text-input counterpart is a degraded product, even if its text capabilities match state-of-the-art. Without quantifying the residual gap, practitioners cannot assess whether the system is suitable for high-stakes voice applications (medical consultation, financial advice, educational tutoring) where the quality difference between spoken and typed queries could have real consequences.

What evidence exists in the paper. The only indirect evidence comes from the overall audio understanding benchmarks (Table 5), which show strong performance on tasks like MMAU (82.2), MMSU (82.8), and VoiceBench (93.1). However, these benchmarks measure general audio comprehension, not the specific comparison of audio-conditioned vs. text-conditioned reasoning on identical content. The paper does not report, for instance, accuracy on a math benchmark where the same problems are presented as text vs. as spoken audio, which would directly measure the OPD effectiveness. The claim that "OPD and interaction-aligned RL have a positive effect on improving the instruction-following capabilities" (Section 5.1.1) is supported by the IFBench score (52.6 vs. 51.1 for text-only), but this is instruction-following on text inputs — it does not measure whether audio instruction-following has caught up.

Mitigation status. The paper does not acknowledge this as a residual limitation. OPD is presented as a solution rather than a partial mitigation, and no measurement of the remaining gap is provided. Section 4.1 notes that "a substantial gap remains" before OPD is applied, but does not characterize what gap remains afterward. For future work, a systematic study measuring accuracy parity between text-conditioned and audio-conditioned responses across reasoning benchmarks would be essential to validate OPD's effectiveness and characterize any remaining modality asymmetry.

Difficulty Estimation Is Entirely Absent — No Mechanism for Adaptive Resource Allocation

The assumption or constraint. Qwen3.5-Omni provides two model variants (Plus and Flash) and reports theoretical latency at different concurrency levels (Table 2), but the model has no mechanism for estimating task difficulty and adapting its computational strategy accordingly. Every query receives the same processing pipeline regardless of whether it is a trivial greeting ("Hello, how are you?") that could be handled with minimal compute or a complex reasoning task ("Explain the proof of the Riemann hypothesis") that would benefit from deeper processing. The paper's focus is on achieving uniformly strong performance across benchmarks; it does not address the problem of when to spend less compute or how to trade off latency against accuracy per-query.

The consequence. In production deployment, uniform resource allocation is economically wasteful and can degrade user experience. Easy queries that could be answered with a small, fast model variant or with fewer generation steps still incur the full cost of the Plus variant with its higher latency and compute budget. Hard queries that might benefit from additional test-time computation (multiple sampling, verification, or chain-of-thought expansion) receive only the standard single-pass generation. The Flash and Plus variants provide a coarse two-level tradeoff, but there is no finer-grained or per-query adaptation. The concurrency scaling data (Table 2) shows that latency nearly doubles from concurrency 1 to 8 (Flash audio: 235ms → 352ms), meaning that under load, all queries slow down equally regardless of their urgency or complexity. For a service handling diverse user requests — from simple chitchat to multi-step function calling — this uniform degradation means simple requests are unnecessarily delayed while complex requests receive no additional computational resources.

What evidence exists in the paper. The paper provides no evidence regarding difficulty estimation or adaptive allocation because it does not study these questions. The latency and throughput numbers (Table 2) characterize system behavior under uniform load but do not explore differentiated service. The two model variants (Plus vs. Flash) represent a static architectural choice rather than a dynamic allocation policy — a deployer chooses one variant for their entire deployment rather than routing queries adaptively between them. The paper does not report any correlation between input characteristics (query length, modality composition, estimated complexity) and the computational resources actually needed to produce a correct response.

Mitigation status. Not addressed. The paper does not discuss adaptive compute allocation, difficulty estimation, or dynamic model selection as future work. This is a significant gap for a system that positions itself as a deployable real-time interactive agent — real-world services almost universally employ some form of adaptive resource allocation (request prioritization, model cascades, early exit) to balance cost, latency, and quality. The paper's latency results are therefore best understood as worst-case or uniform-service numbers rather than the performance achievable under an optimized deployment policy.

Video Understanding with Audio Trails the Primary Competitor on Several Benchmarks, and the Failure Mode Is Unexplored

The assumption or constraint. The paper claims that Qwen3.5-Omni-Plus "matches [Gemini-3.1 Pro] in comprehensive audio-visual understanding" (Abstract). However, Table 7 reveals that this "matching" is uneven: Qwen3.5-Omni-Plus leads on DailyOmni (+1.9 points) and audio-query QA (+2.3 on Qualcomm IVD), but meaningfully trails on three of the nine audio-visual benchmarks — WorldSense (−2.7), AV-SpeakerBench (−3.8), and VideoMME with audio (−5.3) — and trails dramatically on tool use (OmniGAIA: −11.7). The 5.3-point gap on VideoMME with audio is particularly notable because Qwen3.5-Omni-Plus outperforms the text-only baseline on VideoMME without audio (+0.9, Table 6). This suggests that Qwen3.5-Omni-Plus's audio-visual fusion is not universally strong — on certain tasks, the addition of audio degrades relative performance compared to a competitor whose audio-visual integration may be more effective.

The consequence. For practitioners selecting an audio-visual model, the specific weaknesses matter. WorldSense evaluates "real-world omnimodal understanding" and VideoMME is one of the largest and most established video-understanding benchmarks. A 5.3-point deficit on VideoMME with audio — versus a model (Gemini-3.1 Pro) that is a direct commercial competitor — is practically meaningful. If a deployment involves video content where the audio track carries critical information (e.g., lecture videos, surveillance footage with audio cues, video conferencing), the weaker audio-visual fusion could lead to worse understanding than using a competitor model or even than using Qwen3.5-Omni-Plus on the video alone with a separate ASR transcription. The paper provides no analysis of why the gap exists: is it a training data composition issue (too little synchronized audio-visual data relative to Gemini), an architectural limitation in how the Thinker fuses audio and visual tokens, or a benchmark-specific artifact (e.g., Gemini was explicitly optimized for VideoMME)? Without this diagnostic, practitioners cannot predict whether the gap will persist or reverse on their specific audio-visual tasks.

What evidence exists in the paper. Table 7 provides the raw scores but no analysis. The paper does not discuss the WorldSense, AV-SpeakerBench, or VideoMME-with-audio deficits. The OmniGAIA gap is partially attributed to evaluation protocol: "OmniGAIA is evaluated without a thinking prompt and without <answer> formatting" — implying the gap might shrink with different prompting — but this is speculative and not tested. No ablation is reported to determine whether specific architectural choices (e.g., the explicit timestamp mechanism vs. alternative temporal fusion methods) contribute to the audio-visual fusion gap.

Mitigation status. The paper does not acknowledge these deficits as limitations. The Abstract's language — "matching it in comprehensive audio-visual understanding" — is not contradicted by the data (the model does match or exceed on some benchmarks) but is selectively optimistic given the gaps on WorldSense, AV-SpeakerBench, and VideoMME-with-audio. Future work should systematically characterize the conditions under which Qwen3.5-Omni's audio-visual fusion underperforms and identify whether the root cause is data, architecture, or benchmark-specific optimization.

The Model's Parameter Count, Training Compute, and Architecture Dimensions Are Undisclosed

The assumption or constraint. The paper describes Qwen3.5-Omni-Plus as having "hundreds of billions of parameters" and using a Hybrid MoE architecture, but does not disclose the exact parameter count, the number of experts, the expert capacity, the hidden dimension, the number of layers, the attention head count, or the total pretraining FLOPs. Section 2.5 notes the "substantial scale difference" between Flash and Plus variants, and Table 2 shows Plus Thinker TPOP is approximately 3× higher than Flash (17.4ms vs. 5.6ms at concurrency 1), but the precise scale is unspecified. The pretraining data composition is provided in token counts (Section 3: 4 trillion total tokens with per-modality breakdowns), but the training compute is not reported. Speech generation comparisons (Tables 8–12) include models ranging from 7B parameters (MiMo-Audio-7B-Instruct) to commercial black-box systems of unknown scale, without any FLOPs-matched or parameter-matched controls.

The consequence. Practitioners cannot make informed deployment decisions without approximate parameter counts. Memory requirements, serving costs, quantization feasibility, and hardware compatibility all depend on model scale. A "hundreds of billions" parameter MoE might have an effective inference cost similar to a much smaller dense model (if expert activation is sparse) or a much larger one (if many experts are activated). Without knowing the architecture, a team evaluating whether to deploy Qwen3.5-Omni-Flash on consumer hardware or Qwen3.5-Omni-Plus on a multi-GPU server cannot estimate resource requirements. The speech generation comparisons are also weakened: Qwen3.5-Omni-Plus achieves better WER than CosyVoice 3 on cross-lingual tasks (Table 11), but if Qwen3.5-Omni-Plus has 10× or 100× more parameters, the comparison is not one of algorithmic efficiency but of brute-force scale. Researchers cannot determine whether the architectural innovations (ARIA, explicit timestamps) provide benefits per parameter or simply reflect the advantages of a much larger model.

What evidence exists in the paper. The paper provides latency and throughput numbers (Tables 1 and 2) that partially proxy for computational cost: at concurrency 1, Plus Thinker TPOP is 17.4ms (audio) vs. Flash's 5.6ms, roughly a 3.1× ratio. Plus overall latency is 435ms vs. Flash's 235ms for audio, roughly a 1.85× ratio (the ratio is less than 3.1× because Talker latency and codec decode are similar across variants). These numbers characterize inference cost but not model scale — a 3× slower model could be 3× larger or could have similar parameters but worse hardware utilization. The paper does not report training compute, carbon footprint, or hardware configuration for training.

Mitigation status. The paper does not acknowledge the lack of scale disclosure as a limitation. The description "hundreds of billions of parameters" provides a rough order of magnitude but is insufficient for any precise analysis. For future releases, even approximate parameter counts and a FLOPs estimate for pretraining would substantially improve the report's utility for practitioners and researchers seeking to understand the cost-performance tradeoffs. The Qwen team's prior reports (Qwen2.5-Omni-7B, Qwen3-Omni-30B-A3B) disclosed parameter counts; the omission here represents a regression in transparency.

7. Implications and Future Directions

How This Work Changes the Landscape

Qwen3.5-Omni is best understood not as a paradigm shift but as a reframing of the omnimodal AI problem from one of data scale to one of architectural alignment. The paper's central diagnostic contribution — that streaming speech instability arises from a specific, nameable pathology (text-speech rate mismatch) rather than from insufficient training data or model capacity — changes how researchers should think about building voice-interactive AI systems. Before this work, the dominant approach to improving speech generation quality was to scale up: more training data, larger models, more sophisticated codecs. The dual-track Talker in Qwen3-Omni represented this philosophy — generate text and speech in parallel, hope alignment works out, and scale up until it does. Qwen3.5-Omni argues, through its architecture and results, that this approach hits a ceiling that scaling alone cannot breach, and that what is needed instead is an explicit mechanism for handling the fundamental asymmetry between text and speech token densities.

This reframing is significant because it redirects research investment. If the rate mismatch is the bottleneck, then marginal dollars are better spent on alignment mechanisms (adaptive interleaving constraints, dynamic ratio tracking, streaming-aware decoding schedules) than on simply collecting more speech training data. The paper's empirical results substantiate this: Qwen3.5-Omni achieves the best content fidelity across 21 of 29 languages (Tables 9, 10) and reduces cross-lingual error rates by 72% on zh-to-ko relative to CosyVoice 3 (Table 11), despite CosyVoice 3 being a strong, well-resourced academic TTS system. If scale alone were the answer, CosyVoice 3 — which also benefits from substantial data and compute — would not trail by such margins on specific language pairs. The pattern of results (largest advantages on languages with complex text-speech mappings, like Korean and Thai) is exactly what the rate-mismatch diagnostic predicts.

The paper also makes a second conceptual move with practical consequences: it demonstrates that encoding semantic information as text tokens rather than architectural mechanisms generalizes better across languages and modalities. The explicit timestamp approach (Section 2.3) and the Talker system prompt for voice characteristics (Section 2.4) both follow this principle. The former improves temporal reasoning by letting the model read timestamps as text rather than encoding them in position embeddings; the latter enables zero-shot voice cloning through textual descriptions rather than fixed speaker embedding vectors. The cross-lingual voice cloning results (Table 12) — where monolingual speaker fine-tuning generalizes to 29 languages — provide the strongest evidence that these textual representations capture disentangled, transferable features. This principle (model reads semantic information as text rather than learning it implicitly through architecture) is a design philosophy with implications beyond this specific system: it suggests that future multimodal architectures should default to putting structured information in the token stream unless there is a specific reason to embed it in weights or embeddings.

A third shift is more empirical than conceptual: the paper provides the strongest-yet evidence that omnidirectional multimodal training does not inherently degrade modality-specific performance, provided the training recipe is designed with this goal in mind. Tables 4 and 6 show Qwen3.5-Omni-Plus within 0-2 percentage points of its text-only counterpart across 42 text and vision benchmarks, with occasional improvements (RealWorldQA: +5.0, IFBench: +1.5). This finding matters because the assumption of a capability tradeoff has shaped research agendas — papers have developed complex regularization, elastic weight consolidation, and modality-specific replay techniques to prevent "forgetting." Qwen3.5-Omni suggests these may be unnecessary if the base model is frozen during early multimodal alignment and if distillation-based post-training (OPD) is used to transfer capabilities across modalities. This doesn't close the book on the forgetting problem, but it raises the ceiling for what practitioners should expect from a well-designed multimodal training pipeline.

The paper also resolves a latent contradiction in the speech synthesis literature. Prior systems (CosyVoice 2, F5-TTS, MaskGCT) demonstrated strong zero-shot TTS on English and Chinese, but often degraded dramatically on low-resource or phonologically distant languages — a pattern visible in the ElevenLabs failures on Thai (73.9% WER, Table 9) and Vietnamese (73.4% WER, Table 9). The implicit narrative was that multilingual TTS requires either massive per-language data or sophisticated language-specific adapters. Qwen3.5-Omni demonstrates that a single model, trained with balanced multilingual data (3.5:3.5:3 Chinese:English:multilingual for audio, Section 2.2) and an alignment mechanism that is inherently language-agnostic (ARIA's cumulative ratio constraint), can match or exceed per-language-optimized systems across 29 diverse languages. The contradiction is resolved not by denying the difficulty of multilingual TTS but by identifying the specific bottleneck (rate mismatch + data imbalance) and addressing it directly.

Finally, the paper's agentic capabilities — particularly Audio-Visual Vibe Coding (generating code from audio-visual instructions, Section 1) — represent an emergent capability that was not explicitly trained for but arose from the combination of omnimodal understanding and code generation. While the paper provides minimal evaluation of this capability (OmniGAIA is the only agentic benchmark, with mixed results: 57.2 for Plus vs. 68.9 for Gemini-3.1 Pro), the very fact of its emergence suggests that native omnimodal training may unlock qualitatively new behaviors that modular architectures (separate ASR, vision, and code models) cannot produce. This shifts the research question from "can we make separate modality systems work together?" to "what new capabilities emerge when all modalities are processed by a single model end-to-end?", which is a more ambitious framing.

Follow-Up Research This Work Enables

Controlled ablation of ARIA versus dual-track Talker at matched parameter count. The paper's strongest mechanistic claim — that ARIA's adaptive rate constraint, rather than general scale improvements, is responsible for the speech generation quality gains over Qwen3-Omni — is untested. A high-priority follow-up would train two versions of the Talker at identical scale (e.g., both with the same Hybrid MoE architecture, same training data, same 256k context): one using ARIA's unified interleaved single-stream with cumulative ratio constraint, and one using Qwen3-Omni's dual-track design with fixed interleaving. The evaluation should specifically target the failure modes ARIA is designed to prevent: word skipping rate (measured by alignment between generated text and ASR output on the generated speech), pronunciation error rate on numbers and rare words, and prosodic stability across long utterances (e.g., does intonation drift over the course of a 60-second generated response?). The cross-lingual speech generation benchmarks from Tables 9-11 should be the primary testbed, with particular attention to languages where Qwen3.5-Omni dramatically outperforms competitors (Korean, Thai, Japanese) — the rate-mismatch hypothesis predicts that the ARIA variant should show the largest advantages on languages with the most variable text-to-speech token ratios.

Systematic measurement of the residual audio-text quality gap after OPD. On-policy distillation (Section 4.1) is described as a technique that "gradually aligns [the model's] audio-conditioned outputs with its text-conditioned behavior," but no measurement of the remaining gap is provided. A necessary follow-up study would construct a controlled evaluation set where identical reasoning problems (mathematical word problems, logical puzzles, multi-step instructions) are presented to Qwen3.5-Omni-Plus in both text and spoken audio form, with matched content. The primary metric would be the accuracy delta between modalities across difficulty levels (binned by the model's text-conditioned accuracy on each problem). The hypotheses to test: (1) Hard problems show a larger audio-text gap than easy problems (suggesting OPD transfers surface-level patterns but not deep reasoning); (2) The gap persists even after OPD, consistent with the finding in Table 4 that text reasoning benchmarks trail the text-only baseline by ~1-2 points; (3) Languages with less representation in the OPD training data show larger gaps. This study would also inform whether OPD needs to be iterated (multiple rounds of on-policy distillation) or supplemented with RL-based audio conditioning.

Temporal reasoning benchmark comparing explicit timestamps against TMRoPE. The paper replaces TMRoPE with explicit text timestamps (Section 2.3) but provides no quantitative evaluation of the claimed improvements in "precise and robust temporal perception, especially when extrapolating long-context multimodal inputs." A diagnostic benchmark should be constructed with three task categories: (1) absolute temporal localization — "At what time did the speaker first mention Paris?" in 10-hour audio recordings; (2) relative temporal reasoning — "How long after the doorbell rang did the dog start barking?" in synchronized audio-visual video; and (3) temporal extrapolation — testing on sequences 50-100% longer than the maximum training sequence length from Stage 2 pretraining (32,768 tokens). The experiment would compare Qwen3.5-Omni against a variant using TMRoPE (with appropriate frequency scaling for long context), with both models otherwise identical. The explicit-timestamp hypothesis predicts that absolute and relative temporal accuracy should be higher for the timestamp variant, and that the TMRoPE variant should degrade more sharply on extrapolated-length sequences due to sparse position IDs.

Human evaluation of speech naturalness and emotion expressiveness. The paper evaluates speech generation entirely through automatic metrics (WER, CER, speaker embedding cosine similarity), but its claims include "human-like emotional nuance" (Section 1) and "naturalness, expressiveness, and controllability" (Section 4.2). A rigorous human evaluation is essential to validate these qualitative claims. A well-designed study would: (1) use a Mean Opinion Score (MOS) protocol for naturalness on a diverse test set spanning all 29 speech output languages and both preset and custom voices; (2) specifically ablate the Talker's conditioning on Thinker multimodal representations (Section 2.4, paragraph 3) by generating speech with and without access to the emotional/prosodic context from the Thinker, and having raters judge which sounds more contextually appropriate; (3) evaluate emotion controllability by instructing the model to generate the same text with different specified emotions (happy, sad, urgent, calm) and measuring both raters' emotion recognition accuracy and the perceived naturalness of the emotional expression. This would provide the missing subjective validation for what is currently only an objectively measured capability.

Failure mode taxonomy for audio-visual understanding versus Gemini-3.1 Pro. The audio-visual results (Table 7) are the most mixed in the paper, with Qwen3.5-Omni-Plus trailing Gemini-3.1 Pro by 5.3 points on VideoMME with audio and 11.7 points on OmniGAIA. A systematic failure analysis is needed: on what kinds of audio-visual inputs does Qwen3.5-Omni underperform, and why? This could be approached by (1) stratifying VideoMME questions by the degree to which the audio track is necessary for correct answering (questions answerable from video alone vs. questions requiring audio-visual integration); (2) measuring whether the gap on audio-necessary questions is larger, indicating a specific audio-visual fusion weakness; (3) comparing Qwen3.5-Omni's video-only accuracy (from Table 6, VideoMME without subtitles: 81.9) against its video-with-audio accuracy (Table 7, 83.7) to determine whether adding audio actually helps — if the gain from adding audio is smaller for Qwen3.5-Omni than for Gemini-3.1 Pro (which scores 89.0 on the audio version), this would indicate weaker audio-visual integration rather than weaker video understanding generally. This analysis would directly inform whether the fix requires architectural changes to audio-visual fusion or simply more synchronized audio-visual training data.

Long-form audio reasoning beyond ASR accuracy. The paper's 256k context window enables "more than 10 hours of audio" (Section 2.1), but all audio evaluations focus on understanding benchmarks with short clips (MMAU, MMSU) or ASR on sentence-level utterances (Fleurs, Common Voice). The capability enabled by long-context audio — reasoning over multi-hour recordings — is completely unevaluated. A natural follow-up would test: (1) meeting summarization on 2-4 hour multi-speaker recordings (e.g., AMI corpus extended with synthetic multi-hour meetings); (2) podcast question-answering where answers require synthesizing information from widely separated segments (e.g., "Summarize the guest's views on climate policy across her three appearances on the show"); (3) musical structure analysis over full symphonies or albums, requiring tracking themes across 30-60 minute spans. These evaluations would test whether the 256k context and explicit temporal timestamps genuinely enable long-range reasoning or merely allow the model to keep audio in memory without effectively utilizing it — a distinction that matters enormously for the practical value of the long-context capability.

Practical Applications and Downstream Use Cases

Multilingual voice assistants for under-served languages. The paper's multilingual speech generation results (Tables 9-12) are the strongest in the report, with Qwen3.5-Omni-Plus achieving best-in-class WER on 21 of 29 evaluated languages and maintaining performance on languages where commercial systems fail catastrophically (ElevenLabs: 73.9% WER on Thai, 73.4% on Vietnamese; Qwen3.5-Omni-Plus: 2.17% and 1.14% respectively, Table 9). This directly enables voice assistant deployment in markets that have been poorly served by existing systems. A company building a voice interface for Thai or Vietnamese users does not need to train a language-specific TTS system or accept garbled output — they can deploy Qwen3.5-Omni and get content fidelity comparable to English-language systems (Thai custom-voice WER: 1.65%, Table 12). The cross-lingual voice cloning capability further enables use cases where a brand voice or celebrity persona (recorded in one language) can be extended to all 29 supported languages without per-language fine-tuning — reducing the cost of launching a consistent branded voice experience across international markets from months of per-language recording and model training to a single audio sample and API deployment.

Real-time video conferencing with agentic capabilities. The combination of 256k context (supporting 400 seconds of 720p video at 1 FPS, Section 2.1), streaming audio-visual processing, and native tool use (WebSearch, FunctionCall) enables a concrete deployment scenario: a meeting assistant that continuously processes a video conference stream, answers questions about meeting content in real time, and autonomously takes actions. The first-packet latency numbers (435ms audio, 651ms video for Plus at concurrency 1, Table 2) are within acceptable bounds for conversational interaction — the system can respond to a spoken question within about half a second of the question ending. The 8-concurrency latency (955ms for Plus audio, Table 2) supports simultaneous processing of up to 8 meeting streams on a single deployment, and the Generation RTF stays below 0.35 at all concurrency levels, meaning speech output keeps pace with real-time conversation. The Audio-Visual Vibe Coding capability (Section 1) adds the ability for the assistant to generate and execute code in response to meeting content — for example, a team discussing a data analysis could ask the assistant to "plot the sales numbers we just talked about for the last three quarters," and the model could generate the visualization code, execute it, and share the result, all within the meeting context.

Automated audio-visual content captioning at scale for accessibility and search. The paper reports strong captioning performance on Omni-Cloze (64.8 vs. 57.2 for Gemini-3.1 Pro, Table 7) and describes the model's ability to generate "script-level structured captions with precise temporal synchronization and automated scene segmentation" (Abstract). This directly enables automated captioning pipelines for video archives. A media company with a library of 100,000 hours of video content could process this content through Qwen3.5-Omni to generate (1) time-synchronized descriptive captions (including speaker identification, sound event description, and scene-level summaries), (2) searchable metadata enabling natural language retrieval ("find scenes where someone is playing piano while explaining music theory"), and (3) accessibility-oriented captions for deaf and hard-of-hearing viewers that include non-speech audio descriptions. The 10-hour audio context means feature-length films can be processed in a single pass without the segment-boundary artifacts that plague shorter-context systems. The explicit timestamps (Section 2.3) provide frame-level temporal alignment without requiring a separate alignment model, reducing the complexity of the captioning pipeline.

When to Prefer This Method

The paper does not explicitly articulate a decision rule for choosing Qwen3.5-Omni over alternative omnimodal systems or for choosing between architectural approaches. It positions the model primarily against Gemini-3.1 Pro on benchmarks and against commercial TTS APIs on speech generation, but these comparisons are capability demonstrations rather than prescriptive tradeoff analyses. The Plus and Flash variants represent a cost-quality spectrum internal to the Qwen3.5-Omni family rather than distinct methodological alternatives. Because the paper does not frame a clear "use this architecture when X, use that architecture when Y" tradeoff against named alternatives (e.g., "ARIA is preferable to dual-track decoding when the target language has a text-to-speech token ratio exceeding 3:1"), I omit a forced decision matrix. The evidence in the paper does suggest, but does not claim, that the approach is particularly well-suited to multilingual speech scenarios where content fidelity in Asian and low-resource languages is critical — but this is an observation about benchmark results, not a principle the paper itself establishes as a decision rule.