LongCat-Flash-Omni Technical Report

🎯 Pitch

A 560B-parameter open-source model achieves real-time, low-latency audio-visual interaction while simultaneously matching or beating proprietary systems like Gemini-2.5-Pro on omni-modal benchmarks—a feat made possible by a curriculum training strategy that preserves unimodal performance even as new modalities are integrated. Crucially, its novel modality-decoupled parallelism sustains over 90% of text-only training throughput during massive multimodal training, proving that heterogeneous omni-modal training need not be prohibitively inefficient.

1. Executive Summary

This technical report introduces LongCat-Flash-Omni, a 560-billion-parameter open-source omni-modal model that unifies text, image, video, and audio understanding with real-time speech generation within a single framework. Built on the LongCat-Flash Shortcut-connected Mixture-of-Experts (ScMoE) architecture with zero-computation experts (activating 27B of 560B parameters on average), the model adopts a curriculum-inspired progressive training strategy that sequentially incorporates modalities from simpler to more complex — beginning with text-only pretraining, then adding speech, images, video, and finally continuous audio features — while employing a modality-decoupled parallelism scheme that sustains over 90% of text-only training throughput during multimodal training. LongCat-Flash-Omni achieves state-of-the-art performance among open-source omni-modal models — attaining 61.4 on OmniBench (vs. 58.5 for Qwen3-Omni-Instruct), 60.9 on WorldSense (vs. 52.0), and 78.2 on VideoMME with audio (vs. 73.0) — while enabling millisecond-level response latency in streaming audio-visual interaction through a chunk-wise feature interleaving mechanism and an asynchronous serving pipeline, establishing that a large-scale open-source model can deliver competitive real-time multimodal interaction against proprietary systems only when the training pipeline carefully balances unimodal preservation with cross-modal fusion across difficulty-ordered training stages.

2. Context and Motivation

The Core Problem: Building a Unified Model That Doesn't Sacrifice Any Modality

The fundamental challenge this paper tackles is deceptively straightforward to state but extraordinarily difficult to execute: how do you build a single model that simultaneously excels at understanding text, images, video, and audio while also generating speech in real time — without any one capability degrading the others? The field has produced excellent text models, capable vision-language models, competent speech recognition systems, and natural-sounding text-to-speech — but these have largely been separate systems stitched together, each optimized in isolation. The promise of an omni-modal model is that these capabilities can reinforce each other: understanding a cooking video involves reading the recipe text, watching the chef's hand movements, and hearing the sizzling sounds simultaneously, with each modality providing complementary information that the others miss.

The difficulty is not merely an engineering challenge of combining components. It arises from genuine cross-modal heterogeneity — the fundamental structural differences between modalities that make them resist unified representation. Text is a highly compressed symbolic representation with roughly 3–4 semantically dense tokens per second of spoken content. Speech is the acoustic realization of those same concepts but operates at 12.5+ tokens per second (using the paper's audio tokenizer at 16.67 Hz), enriched with paralinguistic information — speaker timbre, emotion, prosody, accent — that text entirely lacks. Images encode spatial structure in parallel rather than sequentially, requiring the model to reason about spatial relationships, object configurations, and visual semantics. Video compounds this by adding temporal dynamics on top of spatial structure, demanding the model track objects across frames, understand causal sequences, and maintain coherence over extended temporal horizons.

Each modality also comes with dramatically different computational requirements. As Table 2 of the paper quantifies, in a single micro-batch during supervised fine-tuning, the vision encoder's computational cost ranges from 0.08 to 400 TFLOPs, the audio encoder from 0.01 to 110 TFLOPs, while the LLM decoder dominates at 1920 to 4668 TFLOPs. The token length distributions across modalities are equally heterogeneous (Figure 7), with text, image, and audio sequences exhibiting fundamentally different statistical properties that make batching and load balancing difficult.

The paper's central claim is not just that an omni-modal model is possible, but that it can be done in the open — as a fully open-source model with competitive performance against proprietary systems — and that the key to success is a specific ordering of training stages combined with careful data curation and an infrastructure design that treats the modality heterogeneity as a first-class constraint rather than something to be averaged away.

Why This Matters: From Capability Silos to Natural Interaction

The importance of solving this problem extends in several directions:

Real-time audio-visual interaction is the next frontier for human-AI interfaces. The paper explicitly positions itself relative to conversational AI products: "Humans are inherently omni-modal beings, capable of efficiently perceiving and integrating diverse forms of information, including visual and auditory inputs" (Section 1). Current voice assistants process speech through a separate ASR pipeline, then reason over text, then convert text back to speech through a TTS system — a chain of separate models that introduces latency, loses paralinguistic information at each stage boundary, and cannot reason jointly over what is seen and heard simultaneously. An end-to-end omni-modal model that processes raw audio and video directly, reasons over both modalities jointly, and generates speech tokens natively, eliminates these information bottlenecks. The paper demonstrates this concretely through the qualitative analysis in Table 16, where LongCat-Flash-Omni achieves 91.5% on paralinguistic understanding — "actively interpreting user emotions from both facial expressions and vocal cues" — something impossible for text-only models or pipeline systems that discard acoustic nuance early.

The open-source community lacks competitive omni-modal models. At the time of this report, proprietary systems like Gemini-2.5 (from Google) and GPT-4o (from OpenAI) have demonstrated impressive omni-modal capabilities — processing text, images, audio, and video within unified architectures with natural spoken interaction. However, these models are closed-source, accessible only through limited APIs, and their architectures and training procedures remain largely undisclosed. The open-source community has produced strong text-only models (the DeepSeek family, Qwen, LongCat-Flash itself) and capable vision-language models, but omni-modal models that integrate audio understanding, speech generation, and real-time interaction have lagged significantly. As the paper notes in Section 1, "research on omni-modal models has attracted broad attention, with many subsequent efforts proposed in the community" — citing Qwen3-Omni, M2-Omni, Baichuan-Audio, Ola, and Vita-1.5 — but none of these had achieved the comprehensive capability profile that the paper demonstrates, particularly on difficult cross-modal benchmarks like WorldSense and OmniBench, or on real-time audio-visual interaction tasks.

Training efficiency for multimodal models is poorly solved, creating a barrier to entry. The computational cost of pretraining a 560B-parameter model is enormous even for text-only data. Adding vision, audio, and video modalities multiplies this challenge because the models and data are heterogeneous — GPUs processing vision encoders sit idle while audio encoders run, the LLM decoder dominates FLOP consumption but must wait for encoder outputs, and data loading patterns differ across modalities. The paper's modality-decoupled parallelism (MDP) scheme and the claim of "over 90% of the throughput achieved by text-only training" is significant because it demonstrates that the infrastructure challenge is solvable: multimodal training need not be dramatically more expensive than text-only training if the parallelism strategy is designed with modality heterogeneity in mind. This lowers the barrier for other teams attempting similar models.

There is a genuine question about whether multimodal training degrades unimodal capabilities. A persistent concern in multimodal model development is catastrophic forgetting — as a model acquires new modalities, it may lose previously learned capabilities in its original modality. This is particularly acute for text, since text-only models have reached impressive levels of reasoning, coding, and factual knowledge. If adding vision and audio to an LLM causes its text capabilities to deteriorate, the trade-off may not be worth it. The paper directly addresses this concern through its text evaluation results (Tables 12 and 13), showing that LongCat-Flash-Omni not only maintains but sometimes improves upon the text performance of its LongCat-Flash foundation model. This preservation is attributed to the progressive training strategy and the carefully balanced data mixture (text:vision:speech ratio of 2:1:1 maintained across stages 2–4).

Where Prior Approaches Fall Short

The paper identifies several categories of limitations in existing work:

Pipeline architectures discard cross-modal information and introduce latency. The dominant approach in deployed speech interaction systems (voice assistants, customer service bots) chains together separate models: ASR → LLM → TTS. Each stage operates on a different representation (audio waveform → text → text → audio waveform), and the intermediate text representation discards all paralinguistic information — emphasis, emotion, uncertainty, speaker identity — that the original speech signal contained. The paper's approach contrasts sharply: by having the LLM backbone directly process continuous audio features from the audio encoder and directly generate discrete speech tokens through multiple audio prediction heads, paralinguistic information can flow through the entire reasoning process. This is what enables the model to simultaneously respond to what the user says and how they say it, and to generate speech with appropriate emotional tone and prosody rather than flat synthesized text.

Existing omni-modal models treat modalities as add-ons rather than first-class citizens. The paper cites several recent omni-modal efforts — Qwen3-Omni (from Alibaba), M2-Omni, Baichuan-Audio, Ola, Vita-1.5 — but positions LongCat-Flash-Omni as qualitatively different in two respects. First, the scale: at 560B parameters (27B activated), it is substantially larger than most open-source omni-modal models, enabling deeper cross-modal reasoning. Second, the integration depth: rather than grafting audio capabilities onto a predominantly vision-language model, LongCat-Flash-Omni incorporates audio from the earliest stages of multimodal training (Stage-1, immediately after text-only pretraining), treating speech as a foundational modality on par with text and vision. The paper argues this early-fusion approach — where audio, text, and vision features are interleaved and processed jointly by the LLM backbone — leads to deeper cross-modal representations than late-fusion approaches that handle modalities in separate encoders before combining them only at the output level.

Real-time streaming interaction imposes constraints that offline models ignore. Most multimodal evaluation focuses on offline tasks: given an image and a question, produce an answer; given a video, summarize it. But real-time interaction introduces fundamentally different requirements that the paper explicitly addresses:

Temporal synchronization: audio and video streams must be time-aligned at the feature level so the model can reason about their correspondence (e.g., "what made that sound?").
Speculative computation: the model cannot wait until the user finishes speaking to begin processing — it must prefill audio-visual features as they arrive to minimize first-packet latency. The paper's speculative prefill-decode switching strategy in Section 6.2 explicitly handles this: the LLM begins decoding at a speculative "end of turn" point before the VAD system confirms the user has stopped, reducing perceived latency.
Barge-in handling: users can interrupt the model's speech output mid-response, requiring the system to gracefully terminate generation, truncate at a natural breakpoint, and switch back to listening mode — a capability the paper describes in its audio delivery and interruption mechanism.
Long-term memory across turns: multi-turn dialogue requires the model to remember visual and auditory context from earlier turns, including what was shown and said minutes ago in the conversation, not just the current utterance.

These streaming-specific challenges are not addressed by the offline evaluation protocols common in the multimodal literature, and the paper's real-time audio-visual interaction evaluation (Section 7.4.2) — with 200 dialogue sessions per model rated by 250 independent annotators — represents a deliberate extension beyond standard benchmarks.

Training infrastructure for heterogeneous multimodal models is immature. The paper's discussion of training infrastructure (Section 5) reveals a problem that is easy to overlook: standard distributed training strategies assume model homogeneity — all parts of the model have similar computational requirements, memory footprints, and data dependencies. This assumption fails catastrophically for omni-modal models. Table 2 shows a 4000× range in computational cost across components within a single micro-batch. A naive approach — applying the same parallelism strategy to all components — would leave most GPUs idle most of the time (the vision encoder finishes in milliseconds while the LLM decoder runs for seconds). The paper identifies two prior approaches to this problem: FSDP-based approaches (like OrchMLLM and veOmni) that shard parameters but avoid pipeline parallelism bubbles, and decoupling approaches (like DistTrain, PipeWeaver, and Optimus) that separate parallelization strategies across components. The paper's MDP scheme builds on the Optimus approach of completely decoupling modality encoders and the LLM at the distributed level, enabling independent scheduling of encoder and decoder computations.

How This Paper Positions Itself

The paper's positioning can be understood along four axes:

1. A comprehensive open-source omni-modal model, not a novel architecture. LongCat-Flash-Omni is primarily a systems and training methodology contribution rather than a fundamentally new architectural proposal. The vision encoder (LongCat-ViT) was introduced in prior work by the same team (Qiao et al., 2025), the audio tokenizer/decoder (LongCat-Audio-Codec) is from prior work (Zhao et al., 2025a), and the LLM backbone (LongCat-Flash with ScMoE) is from prior work (Meituan, 2025a). The paper's novel contributions are in the integration: the multi-stage curriculum that determines when and how each modality is introduced, the data curation strategies that balance modalities while preserving unimodality strength, the infrastructure designs that make training feasible at scale, and the serving pipeline that enables real-time interaction. This is a valid and important type of contribution — engineering integration at this scale, with these performance results, represents genuine research — but the paper is clear that it stands on the shoulders of its component models rather than claiming to have invented them.

2. A deliberate contrast with proprietary systems through openness. The paper's abstract emphasizes: "We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community." This is not just a boilerplate commitment to open-source — it is a deliberate positioning against Gemini and GPT-4o, which are closed, and against other open-source efforts that may not release training details. The paper devotes significant space to training data curation (Section 3.1, broken into seven subsections covering audio, image-text, OCR/grounding/GUI, STEM, multi-image, video, and long-context data), training stages (Section 3.2 with six sub-stages), post-training (Section 4 with SFT and DPO details), and infrastructure (Section 5 with explicit memory optimization breakdowns in Table 4). This level of detail is unusual in industry technical reports and serves a clear purpose: enabling reproduction and building community trust that the reported results are achievable by others.

3. A focus on real-time interaction as a distinguishing capability. While many multimodal models handle offline tasks (image QA, video understanding, audio captioning), the paper explicitly highlights real-time audio-visual interaction as a core capability that sets LongCat-Flash-Omni apart. Section 2.4.2 describes the streaming audio-visual interaction mechanism in detail — the chunk-wise feature interleaving, the sparse-dense sampling strategy for turn-taking, and the architectural components that enable low-latency processing. Section 6.2 describes the asynchronous streaming pipeline with VAD, speculative prefill-decode switching, and audio delivery with interruption handling. And Section 7.4.2 provides the real-time interaction evaluation against both proprietary products (Doubao, GPT-4o, iFlytek Spark, StepFun, ChatGLM) and open-source models (Qwen2.5-Omni, Qwen3-Omni). This evaluation — with human conversationalists conducting multi-turn dialogues and 250 independent annotators rating naturalness — goes well beyond standard benchmark evaluations and reflects a commitment to demonstrating practical deployment capability rather than just leaderboard performance.

4. An explicit argument that progressive, curriculum-inspired training is the key to preserving unimodal strength. The paper's central methodological claim is that the order of modality introduction matters enormously. The progression is: Stage-0 (text-only, 16T tokens, building strong language and reasoning foundations), Stage-1 (text + speech, 5.1T tokens, adding the structurally-similar sequential modality), Stage-2 (text + speech + images, >3T tokens, adding spatial reasoning), Stage-3 (text + speech + images + video, 0.33T tokens, adding temporal dynamics with annealing), Stage-4 (context extension to 128K tokens), Stage-5 (continuous audio encoder alignment to replace discrete tokens with richer acoustic features). This ordering is not arbitrary — it follows a deliberate "simpler to more complex" curriculum: text (abstract symbols with strong pretraining baselines) → speech (sequential like text but with acoustic richness) → images (spatial, fundamentally non-sequential) → video (spatial + temporal, the most complex modality). The paper argues this ordering prevents the catastrophic forgetting that might occur if video were introduced too early (overwhelming the model with temporal complexity before it has mastered spatial reasoning) or if text were deemphasized in favor of multimodal data (undermining the reasoning capabilities that underpin all modalities).

The paper also positions its data curation as a first-class contribution rather than an afterthought. Each data category (Sections 3.1.1 through 3.1.7) receives detailed treatment of collection, filtering, and quality control procedures. The vision-speech QA data construction — pairing visual inputs with TTS-synthesized spoken prompts — represents a deliberate bridge between offline vision-language understanding and real-time speech interaction, allowing capabilities learned on text-based vision QA to transfer to spoken dialogue settings. The audio-visual interaction data pipeline, with its model-driven automation followed by human-in-the-loop verification (Section 4.1.2), addresses the chicken-and-egg problem of needing interaction data to train an interaction model but needing an interaction model to generate interaction data — solving it through a semi-automated cycle with human quality control at the bottleneck.

In summary, this paper addresses the gap between specialized unimodal models that excel in isolation and the need for integrated omni-modal systems that can perceive, reason, and interact across modalities in real time. It positions itself not as a theoretical breakthrough but as an engineering and methodological milestone: demonstrating that, with the right training curriculum, data strategy, and infrastructure design, it is possible to build a 560B-parameter open-source model that competes with proprietary systems across the full spectrum of omni-modal capabilities while enabling practical real-time deployment.

3. Technical Approach

3.1 Reader Orientation

LongCat-Flash-Omni is a single neural network — one massive 560-billion-parameter transformer — that can simultaneously see images and video, hear audio, read text, reason about all of them jointly, and produce responses as either text or natural speech in real time. The problem it solves is that existing models treat modalities as separate pipelines (speech recognition → text model → text-to-speech), which discards crucial information — like the emotional tone in someone's voice or the visual context of what they're pointing at — at each translation step. The "shape" of the solution is a curriculum-driven integration: start with a strong text-only model, then progressively teach it speech (the closest modality to text), then images, then video (the most complex), each time carefully balancing the data mixture so that new capabilities are added without old ones deteriorating — all while designing the training infrastructure to handle the fact that processing a second of video costs 4000× more FLOPs than processing a second of audio within the same training batch.

3.2 Big-Picture Architecture (Diagram in Words)

Imagine five major processing blocks connected in a pipeline, with the largest block — the language model — sitting at the center:

Vision Encoder (LongCat-ViT, ~600M parameters): Takes raw images or video frames as input and outputs a sequence of "visual tokens" — compact numerical vectors that encode what the model sees. It supports native-resolution inputs (no squashing images to squares) and handles both single images and video with the same architecture.
Audio Encoder (~600M parameters): Takes raw audio waveforms, converts them to 80-dimensional Fbank spectral features, and outputs a sequence of "audio tokens" — continuous vectors representing 80ms chunks of sound. It uses a streaming architecture so it can process speech as it arrives rather than waiting for the utterance to finish.
Audio Tokenizer/Decoder (LongCat-Audio-Codec): On the input side (during early training stages), it converts speech into four parallel streams of discrete tokens — one for semantic content, three for acoustic details — at 16.67 Hz. On the output side, it takes the four streams of discrete tokens generated by the LLM and reconstructs them into a natural-sounding audio waveform.
The LLM Backbone (LongCat-Flash, 560B total, 27B activated): This is the brain. It receives interleaved sequences of text tokens, visual tokens, and audio tokens (or continuous audio features), processes them through 64 transformer layers with Shortcut-connected Mixture-of-Experts (ScMoE) modules that dynamically route each token to a subset of expert networks, and produces two parallel output streams: text tokens (for text responses) and four-codebook audio tokens (for speech responses). The "zero-computation experts" mean that for any given token, many experts are skipped entirely — only 27B of the 560B parameters are actually used per forward pass on average.
Modality Projectors: Small two-layer MLPs that act as translators — one converts vision encoder outputs into the LLM's internal representation space, another converts audio encoder outputs into the same space. They ensure that an image patch and a sound chunk and a text word all end up as vectors in the same 7168-dimensional space where the LLM can reason about them jointly.

Information flows as follows: raw audio and video arrive simultaneously → the audio encoder processes speech chunk-by-chunk in real time while the vision encoder processes video frames at 2 FPS → the projectors map both into the LLM's token space → these tokens are interleaved in time-synchronized order (second 1's video tokens, then second 1's audio tokens, then second 2's video tokens, etc.) → the LLM processes the entire multimodal sequence through its transformer layers → the LLM generates text tokens (fed to a text decoder) and four parallel streams of audio tokens (fed to the audio decoder) → the audio decoder reconstructs the waveform for the user to hear, while text appears on screen.

3.3 Roadmap for the Deep Dive

First, the progressive training curriculum (Section 3.2 of the paper): The six-stage pipeline that introduces modalities in order of increasing complexity — text, speech, images, video, long context, continuous audio — because this ordering determines when and how each architectural component is trained, and understanding the data ratios and loss weighting at each stage is essential before examining any single component in isolation.
Second, the data curation strategies across seven categories (Section 3.1): What data is used, how it is filtered and augmented, and how the 2:1:1 text-to-vision-to-speech ratio is maintained — because data quality and diversity are what make the progressive training work rather than degenerate into catastrophic forgetting.
Third, the vision encoder (LongCat-ViT, Section 2.1): Its native-resolution design, 2D-RoPE position encoding, progressive resolution adaptation during contrastive pretraining, and how it handles the video-specific compression pipeline — since the vision encoder is the computational bottleneck for video processing.
Fourth, the audio components (Section 2.2): The audio tokenizer with its four-codebook design at 16.67 Hz, the streaming audio encoder with FSMN layers and one-frame lookahead, and the causal audio decoder — because the transition from discrete speech tokens (stages 1–4) to continuous audio features (stage 5) is a key design decision that affects both training efficiency and acoustic fidelity.
Fifth, the LLM backbone with ScMoE (Section 2.3): Multi-head Latent Attention, shortcut-connected experts, zero-computation experts, and variable per-token computation — because the 27B activated parameter count is what makes streaming real-time interaction feasible at this scale.
Sixth, the streaming interaction mechanism (Section 2.4): Chunk-wise audio-visual interleaving, sparse-dense sampling during turn-taking, textual timestamps, and hierarchical video token compression — because this is what distinguishes real-time omni-modal interaction from offline multimodal understanding.
Seventh, the post-training pipeline (Section 4): Supervised fine-tuning with vision-speech QA data, audio-visual interaction data construction with human-in-the-loop verification, and the joint text-speech DPO objective — because the foundation model's capabilities must be shaped into usable interactive behaviors.
Eighth, the training infrastructure (Section 5): Modality-decoupled parallelism, the ModalityBridge for cross-component communication, and the memory optimization strategies that enable fitting 560B parameters across 80GB GPUs — because without these, the training would be infeasible.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a carefully ordered, curriculum-inspired progressive training pipeline — combined with modality-decoupled distributed training infrastructure and lightweight streaming-optimized encoders — enables a 560B-parameter open-source model to achieve competitive omni-modal performance while maintaining real-time interaction latency.

The Progressive Training Curriculum: Six Stages in Order of Modality Complexity

The most fundamental design decision in LongCat-Flash-Omni is not which architectures to use for each modality — those are largely inherited from prior work (LongCat-ViT, LongCat-Audio-Codec, LongCat-Flash) — but rather the sequence and conditions under which each modality is introduced during pretraining. The paper explicitly frames this as a curriculum: "we adopt a curriculum-inspired, progressive training strategy that gradually transitions from simpler to more complex sequence modeling tasks" (Section 3.2). The progression logic is as follows:

Stage-0: Text Pre-Training (16 trillion tokens). This stage is identical to the initial phase of LongCat-Flash (the team's prior text-only model). The model is trained on a high-quality, diverse text corpus with a constant learning rate, and the proportion of reasoning-heavy data (STEM, code) is progressively increased. The output is a 560B-parameter language model with strong text understanding, reasoning, and code generation capabilities. This stage matters because it establishes a robust "knowledge backbone" — factual knowledge, linguistic competence, and reasoning patterns — that all subsequent modalities will build upon. The paper's implicit argument is that starting multimodal training from a random initialization (or from a weak text model) would lead to worse results because the model would have to simultaneously learn language structure, world knowledge, and cross-modal alignment — an optimization problem likely to result in mediocre performance across all dimensions.

Stage-1: Text-Speech Continued Pre-Training (5.1 trillion tokens). Speech is introduced before vision because, despite being a different modality, speech is structurally the most similar to text: both are sequential signals where temporal order matters, and speech is fundamentally the acoustic realization of the same linguistic content that text captures symbolically. This structural similarity means the model can reuse much of its sequence modeling capability from text pretraining when learning to process speech. The training data consists of three components: pure text data (to preserve text capabilities), speech-text interleaved data (where transcribed audio is broken into short segments and either the audio or text portion of each segment may be randomly masked), and a small proportion of ASR-style data (pure speech-to-text transcription). All speech is discretized into four-codebook token sequences by the audio tokenizer — the first codebook captures semantic content (what words are being said), while the other three codebooks capture acoustic details (how they sound — pitch, timbre, speed).

The model architecture at this stage is shown in Figure 6: text embeddings and audio embeddings (one per codebook, with acoustic codebooks offset by one timestep relative to the semantic codebook) are fused and fed into the LLM decoder. The LLM has four parallel output heads: one text head and three audio heads (one for each codebook beyond the semantic one). The training objective is a weighted sum of four loss terms:

$\mathcal{L}_{\text{total}} = a \mathcal{L}_{\text{pure-text}} + b \mathcal{L}_{\text{audio}} + c \mathcal{L}_{\text{audio-text}} + d \mathcal{L}_{\text{first-audio}}$

where a = 1.75, b = 0.25, c = 1.5, d = 0.1 are the loss weights, $\mathcal{L}_{\text{pure-text}}$ is the standard next-token prediction loss on pure text sequences, $\mathcal{L}_{\text{audio}}$ is the next-token prediction loss on audio token sequences (for all four codebooks), $\mathcal{L}_{\text{audio-text}}$ is the text-generation loss term within interleaved audio-text sequences, and $\mathcal{L}_{\text{first-audio}}$ is an additional loss term applied specifically to the semantic audio token (the first codebook) to strengthen its role as the primary carrier of linguistic content.

What this loss computes: It is a weighted linear combination of four cross-entropy losses, each measuring how well the model predicts the next token in different contexts. The pure-text term (a = 1.75, the largest weight) ensures that the model does not forget text capabilities while learning speech. The interleaved audio-text term (c = 1.5) encourages the model to learn cross-modal connections between spoken words and their written transcriptions. The audio-only term (b = 0.25, a small weight) provides basic speech modeling signal. The first-audio term (d = 0.1, the smallest weight) specifically targets the semantic codebook to improve early convergence on speech content.

Why these weights: The paper states these values were determined by "carefully hyperparameter tuning and extensive preliminary experiments" and that they achieve "the best balance between preserving text capabilities and enhancing audio performance." The heavy weight on pure text (a = 1.75) reflects the prioritization of preventing catastrophic forgetting, while the moderate weight on interleaved audio-text (c = 1.5) supports cross-modal alignment without overwhelming the text signal. The small audio-only weight (b = 0.25) prevents the lower semantic density of speech (12.5 tokens per second for speech vs. ~3–4 meaningful text tokens per second of spoken content) from dominating the optimization.

The text-to-audio token ratio is 2:1 (twice as many text tokens as audio tokens), consuming 5.1 trillion tokens total with slight learning rate decay.

Stage-2: Multimodal Continued Pre-Training (>3 trillion tokens). Images are introduced for the first time. The vision encoder (LongCat-ViT) is initialized from the contrastively pretrained checkpoint (trained on 14.6 billion image-text pairs from scratch, as described in Section 2.1) and the vision projector is randomly initialized. The training data now includes image caption data and interleaved image-text data, while maintaining the same text-to-audio ratio (2:1) from Stage-1 and setting the text-to-vision ratio also at 2:1. The total data mixture ratio is text:vision:audio = 2:1:1. The vision encoder, vision projector, and LLM decoder are all jointly trained — no freezing of vision components — with a nearly constant learning rate. The vision-related loss weight is set at 0.25 in addition to the Stage-1 loss weights.

Stage-3: Multimodal Annealing Training (0.33 trillion tokens). This is the quality-over-quantity phase. The model continues training with "curated higher-quality data under an annealed learning rate." Video data (captioning and QA datasets) is introduced for the first time, along with a broader range of image-related data: OCR, grounding, GUI, multi-image, and STEM datasets. The text:vision:speech token ratio remains 2:1:1. The total data volume is smaller (0.33T tokens) but higher quality, and the learning rate is annealed (decayed to zero) to achieve superior final performance. A critical innovation at this stage is the PPL-gap-based data sampling strategy: the training corpus is segmented into subsets by semantics and task. A reference vision-language model computes per-sample perplexity (PPL) for each subset as the "expected convergence level." During training, if a subset's PPL lags behind the expected level, its sampling weight is dynamically increased — the model automatically spends more training on the data it is struggling to learn. This addresses the "heterogeneity in content distribution, learning difficulty, and scale" that characterizes vision data. Samples where "PPL convergence is consistent with downstream performance" are isolated into independent subsets to prevent their signal from being diluted by aggregation with less informative data.

Stage-4: Context-Length Extension Training (120 billion tokens). The context window is expanded from 8K to 128K tokens in two steps: first to 32K tokens using 100B training tokens while increasing RoPE's base frequency from 1M to 5M, then to 128K tokens using 20B training tokens while increasing the RoPE base frequency to 10M. This progressive extension — rather than jumping directly to 128K — prevents the positional encoding from becoming distorted, since RoPE's base frequency controls how well the model can distinguish between nearby positions (higher base frequency = better local discrimination at long ranges). The training data mixture incorporates 25% additional long-context multimodal data (from Section 3.1.7) on top of the Stage-3 data, maintaining the 2:1:1 text:vision:speech ratio.

Stage-5: Audio Encoder Alignment Training. This is the transition from discrete speech tokens to continuous audio features. During stages 1–4, the LLM processes speech as four parallel streams of discrete tokens from the audio tokenizer — a necessary simplification for efficient large-scale training, but one that "hinders the model's ability to capture fine-grained acoustic details." In Stage-5, the LLM parameters are frozen and only the audio encoder (the streaming architecture with FSMN layers described in Section 2.2) is trained. The audio encoder converts raw speech into continuous audio features, which are then projected into the LLM's semantic space. The audio encoder weights are initialized from the speech-recognition-trained checkpoint, while the audio projector is randomly initialized. Training data is formatted as "Task Prompt + Speech Input + LLM Response" with the loss computed only on the response tokens. The paper notes that "empirical results showed negligible performance differences" between pre-training the audio projector separately and direct end-to-end audio encoder training, so the simpler direct approach is used. This stage achieves two objectives: (1) preserving the LLM's established multimodal processing capabilities (since the LLM is frozen), and (2) replacing the lossy discrete token representation with rich continuous features that capture finer acoustic nuance.

Data Curation: Seven Categories with Systematic Quality Control

The paper devotes Section 3.1 (seven subsections) to data curation, treating data quality and diversity as first-class design decisions rather than implementation details. The total pretraining corpus exceeds 2.5 trillion tokens.

Audio Data (Section 3.1.1). The speech-text interleaved data pipeline processes tens of millions of hours of audio: (1) Voice activity detection (VAD) splits long audio into speaking segments and removes non-speech regions. (2) Two proprietary ASR models independently transcribe each segment; segments with significant transcription discrepancies are discarded — this dual-model cross-validation catches cases where either ASR system made errors. (3) A multilingual speech aligner performs forced alignment, yielding precise timestamps for each word. (4) Segments where the speech-duration-to-text-length ratio falls outside the 0.5th to 99.5th percentile range are discarded — this removes unusual cases like extremely fast speech or long silences without words. (5) Adjacent speaking segments separated by silences shorter than 10 seconds are merged into training samples. (6) Each merged sample is split at punctuation marks into fragments (audio chunk + transcribed text). Some fragments have their audio masked (keeping only text), others have their text masked (keeping only audio), creating interleaved training sequences where the model must predict missing modalities based on context. For audio understanding data, the paper curates a comprehensive set covering audio captioning, semantic audio understanding, paralinguistic analysis, acoustic scene/event detection, audio QA, and music understanding, combining open-source datasets with in-house data and applying text translation and pseudo-label generation for data augmentation.

Generic Image-Text Data (Section 3.1.2). Image captions undergo multi-stage cleaning: (1) Text-level filtering removes extremely short captions. (2) Image-level filtering removes abnormal resolutions and poor-quality images. (3) Pair-level filtering uses a SigLIP similarity threshold to discard image-text pairs with low semantic alignment. (4) Re-captioning with multiple open-source vision-language models generates dense, fine-grained captions while preserving world knowledge from original annotations. (5) Additional filtering removes mixed-language captions, repetitions, and truncations. Crucially, after these quality filters, the paper implements a diversity preservation step: image-text pairs are clustered by joint image-text embeddings, and samples are resampled from each cluster to ensure long-tail content representation. The paper's experiments reveal that "preserving diversity at this stage is more beneficial than overly strict quality filtering" — an important empirical insight that counterbalances the natural tendency to filter aggressively. Furthermore, a concept-based resampling strategy inspired by MetaCLIP expands the vocabulary with a 200K-scale Chinese lexicon to achieve broader concept coverage. For interleaved image-text data, a two-stage pipeline first filters noisy tokens, sensitive content, and overly complex samples, then uses density-based pruning and semantic clustering to sample a diverse, evenly distributed subset, reducing the raw dataset by approximately 74% while maintaining quality.

OCR, Grounding and GUI Data (Section 3.1.3). OCR data covers scene text, structured documents, handwriting, and mathematical expressions, with synthesized multi-page and region-level samples (2–6 pages each) to teach the model document-level reasoning. Grounding data uses open-source detection datasets (Visual Genome, COCO, Objects365) to construct two types of QA: localization data (given a description, output coordinates) and region-captioning data (given coordinates, describe what's there), with all coordinates normalized to 0–1000 in JSON format. GUI data encompasses screenshots from diverse PC and mobile applications, with three task types: perception (OCR, VQA, captioning on GUI screenshots), grounding (identifying UI elements and their locations), and planning (multi-step navigational paths with observation, summarization, reasoning, and action components).

STEM Data (Section 3.1.4). The paper constructs a large-scale multimodal STEM dataset of 15 million image-text pairs spanning K–12 to advanced university levels across a wide range of disciplines. The data is structured into multiple-choice and open-ended generative QA formats, with "rigorous multi-stage filtering to ensure factual accuracy, eliminate ambiguity, and standardize formatting." This dataset is positioned as essential for deep conceptual understanding and robust reasoning.

Multi-Image Data (Section 3.1.5). A capability taxonomy is constructed with fine-grained categories like "emotion recognition," "vehicle identification," and "time calculation with clocks." Multi-image QA data is collected through careful selection and augmentation from open-source datasets, as well as synthesis using dedicated image generation tools. The purpose is to teach the model to compare, contrast, and reason across multiple images — a capability particularly important for the multi-image benchmarks where LongCat-Flash-Omni shows strong performance (e.g., 77.1 on MuirBench vs. 62.1 for Qwen3-Omni).

Video Data (Section 3.1.6). The video data pipeline consists of three components: (1) High-quality video caption data: videos are divided into scenes using a scene detection algorithm, clustered, sampled for diversity, and captioned by a proprietary model that describes event sequences and overall context. (2) Temporally grounded video QA data: rule-based transformations convert annotations from temporal action detection, segmentation, video summarization, and temporal sentence grounding tasks into QA pairs, with a proprietary model generating more challenging follow-up questions. (3) Action recognition video QA data: video recognition datasets (Kinetics-700, Something-Something v2) are transformed into multiple-choice and open-ended QA pairs to strengthen fine-grained action understanding.

Long-Context Multimodal Data (Section 3.1.7). This category addresses the challenge of reasoning over extended sequences. It includes: (1) Image-text interleaved data created by concatenating single-image data on the same topic, or rendering text segments within long documents as images — teaching the model needle-in-a-haystack retrieval and in-context learning. (2) Long video QA data constructed by segmenting long videos into clips, generating rich captions for each clip, analyzing scene continuity across clips, concatenating and refining continuous clips into longer-form video captions, and constructing temporally grounded QA pairs. This data specifically targets "long-sequence cross-modal modeling and memory retention."

Vision Encoder (LongCat-ViT): Native-Resolution Encoding with Unified Image-Video Architecture

The vision encoder is a 637M-parameter Vision Transformer (LongCat-ViT) with 32 layers, 16 attention heads, hidden size 1280, intermediate size 5184, and patch size 14×14 pixels. The key innovations over a standard ViT are:

2D Rotary Position Embeddings (2D-RoPE). Standard ViTs use 1D learned position embeddings that assign a unique vector to each patch position in a fixed grid — this works for fixed-resolution images but breaks when images have arbitrary resolutions or aspect ratios, because patch positions beyond the training grid have no learned embedding. 2D-RoPE, in contrast, encodes position using sinusoidal functions of the (row, column) coordinates, generalizing the RoPE mechanism (Su et al., 2024) from the 1D case (token position in a sequence) to the 2D case (spatial position in an image grid). The result is that the encoder can process images of any resolution and aspect ratio without requiring a pre-defined grid size — the positional signal is computed on the fly from the spatial coordinates of each patch. The detailed hyperparameters are in Table 1.

Native Resolution Encoding. Conventional ViTs like CLIP and SigLIP resize all input images to a fixed square (e.g., 224×224 or 384×384), which "can result in substantial information loss, particularly for images with extreme aspect ratios or high native resolutions" (Section 2.1). LongCat-ViT instead encodes images at their native resolutions. For each image or video frame, if the number of patches falls within a predefined range (576–5832 during training), only minimal resizing is applied to ensure both dimensions are divisible by 112 (the patch size of 14 times a downsampling factor). If outside this range, the image is rescaled to fit within the patch count limits while preserving aspect ratio. The 2× pixel-unshuffle operation applied along the spatial dimension before the main transformer reduces the sequence length by a factor of 4, mitigating the quadratic computational complexity of self-attention on high-resolution inputs.

Unified Image and Video Encoding. LongCat-ViT uses a "unified patchification module" that treats videos as sequences of frames, each frame independently patchified. This means the same model processes both still images and video frames without architectural modifications — a video is simply a longer sequence of visual tokens. The patch embeddings for all frames are computed by the same ViT, ensuring consistent visual representations across image and video tasks.

Contrastive Vision-Language Pretraining with Progressive Resolution Adaptation. The vision encoder is not trained from scratch during multimodal pretraining. Instead, it undergoes a separate contrastive pretraining phase on 14.6 billion image-text samples. This phase uses a progressive training scheme: (1) Start with fixed low-resolution training (e.g., 224×224) to establish basic visual-textual alignment at lower computational cost. (2) Transition to native-resolution fine-tuning once the model has learned basic visual features — this curriculum prevents the model from being overwhelmed by the complexity of high-resolution images before it has learned fundamental visual concepts. (3) Introduce video data only in the final stage of contrastive training to reduce computational overhead. To facilitate early convergence, feature distillation from a frozen pretrained vision model is used as an auxiliary objective, with the distillation weight gradually reduced in later stages.

Architectural Enhancements. Beyond 2D-RoPE and native resolution, LongCat-ViT incorporates: SwiGLU activation function (a gated variant of GELU that provides better gradient flow), RMSNorm (a simplified layer normalization that normalizes by root mean square without recentering), LayerScale (learnable per-channel scaling parameters that stabilize training of deep ViTs), and Query-Key normalization (normalizing the query and key vectors before computing attention scores to improve training stability). These are standard ViT improvements that collectively yield better convergence and final performance.

Audio Components: Tokenizer, Encoder, and Decoder

The audio processing chain in LongCat-Flash-Omni involves three distinct components that serve different roles at different training stages.

Audio Tokenizer (LongCat-Audio-Codec). The tokenizer converts raw audio waveforms into four parallel streams of discrete tokens at a frame rate of 16.67 Hz — meaning for each ~60ms of audio, the tokenizer produces four integer-valued tokens (one per codebook). The four codebooks are specialized: one codebook represents semantic information (the linguistic content — what words are being said), while the other three codebooks capture acoustic details (speaker timbre, prosody, emotion, background noise characteristics). At 16.67 Hz, one second of speech produces approximately 16.67 × 4 = 66.68 tokens. This is a relatively low frame rate compared to some neural audio codecs (which may operate at 50–75 Hz), intentionally chosen to balance "robust semantic modeling, flexible acoustic feature extraction, and low-latency streaming synthesis capabilities." The discrete representation enables the LLM to process speech using the same next-token prediction paradigm as text — during stages 1–4 of pretraining, the LLM predicts speech tokens autoregressively, one timestep at a time across all four codebooks, with a one-step temporal offset between semantic and acoustic token predictions (the acoustic tokens at timestep t are predicted based on the semantic token at timestep t and previous acoustic tokens, enforcing a causal hierarchy where semantic content is determined before acoustic realization).

Audio Decoder (LongCat-Audio-Codec). The decoder performs the inverse operation: reconstructing a waveform from the four-codebook discrete tokens. Its architecture, shown in Figure 3, consists of LSTM layers, convolutional blocks (causal Conv1d, causal dilated Conv1d with dilation rates 1, 3, and 9), and causal transposed convolution layers for upsampling. The entire decoder is causal — each output sample depends only on past and current input tokens, never future ones — and supports streaming decoding with a look-ahead of only three frames (approximately 180ms). This low look-ahead is critical for real-time interaction, as it means the decoder can begin outputting audio almost immediately after receiving tokens from the LLM, rather than waiting for the entire response to be generated. The decoder is trained under a generative adversarial network (GAN) framework: a discriminator is trained to distinguish real audio from decoder-generated audio, and the decoder is trained to fool the discriminator while minimizing reconstruction error — a standard approach for high-quality neural vocoding.

The paper explicitly notes a key design choice: "unlike conventional waveform reconstruction methods that rely on diffusion- or flow-matching-based code2mel models followed by vocoders, we directly employ the decoder from LongCat-Audio-Codec as the audio decoder." Diffusion and flow-matching models produce high-quality audio but are computationally expensive and require multiple iterative refinement steps, making them unsuitable for streaming, low-latency scenarios. The direct GAN-based decoder achieves sufficient quality for conversational speech while being fast enough for real-time use.

Audio Encoder (Streaming Architecture). The audio encoder introduced in Stage-5 replaces discrete tokens with continuous features. As shown in Figure 4, it takes 80-dimensional Fbank filterbank features as input — these are spectrogram-like representations that capture the energy in different frequency bands over time, a standard front-end for speech processing. The architecture includes: (1) A Pre-FFN module that applies frame splicing downsampling, reducing the audio sequence length by a factor of 8 by concatenating features from adjacent frames — each output frame represents an 80ms time window rather than the typical 10ms. This aggressive downsampling reduces the sequence length processed by subsequent layers, cutting computational cost 8×. (2) A Pre-Norm configuration for training stability — applying layer normalization before each sub-layer rather than after, which is standard practice in modern transformer architectures. (3) FSMN (Feedforward Sequential Memory Network) layers replacing standard self-attention modules. FSMN is a lightweight alternative to self-attention that uses learnable memory blocks — fixed-size feedforward networks that aggregate information from neighboring time steps within a constrained context window. The key advantage is that FSMN has linear complexity in sequence length rather than the quadratic complexity of self-attention, making it suitable for processing long audio streams. (4) A hybrid look-ahead strategy: only the final six layers incorporate a one-frame look-ahead mechanism (looking 80ms into the future), while earlier layers are strictly causal. This balances latency (most layers don't wait for future frames) with performance (the final layers can use a small amount of future context for better feature extraction). (5) A Post-FFN module for final feature refinement. The audio encoder is trained under supervised learning using speech recognition data with CTC (Connectionist Temporal Classification) loss — a sequence-level objective that aligns variable-length audio with text transcriptions without requiring explicit alignment.

LLM Backbone: Shortcut-Connected Mixture-of-Experts with Zero-Computation Experts

The LLM backbone is inherited from LongCat-Flash (Meituan, 2025a), a 560B-parameter Mixture-of-Experts language model. The key architectural features are:

Multi-Head Latent Attention (MLA). Standard multi-head attention computes query, key, and value projections for each attention head independently, resulting in memory and computation costs that scale with the number of heads. MLA instead uses a low-rank factorization: the keys and values for all heads are computed from a shared, compressed latent representation, which is then projected into per-head key/value spaces. This significantly reduces the KV-cache size during inference — critical for long-context streaming scenarios where the model must store key-value pairs for the entire conversation history across 128K tokens, for all layers, for all modalities.

Shortcut-Connected Mixture-of-Experts (ScMoE) with Zero-Computation Experts. A standard MoE layer replaces a single feedforward network with multiple parallel "expert" networks, routing each token to a subset of experts (typically top-1 or top-2) via a learned routing function. This increases model capacity without proportionally increasing computation — only the activated experts' parameters are used for each token. LongCat-Flash extends this with two innovations: (1) Shortcut connections: inspired by residual connections but applied at the expert level, allowing tokens to bypass expert computation entirely and pass unmodified when the routing function determines no expert is a good match. This reduces unnecessary computation. (2) Zero-computation experts: some expert slots are designated as "zero-computation" — when a token is routed to these slots, no computation is performed, and the token's representation is unchanged. This enables the model to spend budget selectively: some tokens (e.g., punctuation, common function words, easy-to-predict audio frames) need minimal processing, while others (e.g., content words, reasoning steps, surprising audio events) need heavy processing. The result is variable computation per token: "activating 18.6B–31.3B parameters (27B on average)" despite the total model having 560B parameters.

This variable computation is what makes real-time interaction at 560B scale possible — the model automatically allocates more compute to difficult tokens and less to easy ones, achieving a balance between capacity and latency that a dense model could not match. For streaming audio-visual interaction, the average 27B activated parameters means the per-token computational cost is comparable to a much smaller dense model, while the full 560B parameter capacity is available for tokens that need deep reasoning.

Video Processing Strategy: Dynamic Sampling, Textual Timestamps, and Hierarchical Compression

Efficient video processing is a major challenge because videos vary enormously in duration (seconds to hours) and resolution, and naively encoding every frame at full resolution would be computationally prohibitive. LongCat-Flash-Omni uses a three-stage strategy:

Dynamic Video Frame Sampling. The default sampling rate is 2 frames per second (2 FPS), but this is dynamically adjusted based on video duration. Short videos are sampled at higher frame rates (above 2 FPS) to ensure at least 16 frames are captured — below this threshold, temporal information would be too sparse for effective understanding. For excessively long videos, frames are sampled uniformly up to a maximum frame number constraint to regulate memory consumption. This dynamic adjustment means the model adapts its temporal resolution to the video length — dense sampling for clips where every frame matters, sparse sampling for long videos where coarse temporal coverage suffices.

Textual Timestamps. Before each video frame's visual tokens are input to the LLM, the text "Second{i}" is prepended as a pure text token, where i is the timestamp in seconds. The resulting sequence is interleaved: "Second{i}||Vi||Second{j}||Vj|| ...", where Vi represents the visual tokens at second i and || denotes concatenation. These timestamps "strengthen the model's temporal awareness and improve its ability to recognize specific time points" — they give the LLM an explicit temporal coordinate system to reason about when events occur, rather than forcing it to infer timing from token position alone. This is similar to the approach in TimeMarker (Chen et al., 2024a).

Hierarchical Token Compression. Video inputs undergo three successive compression steps: (1) Frame rescaling: each frame is rescaled to fit within a predefined upper limit on the number of patches (described in Section 2.1's native resolution encoding). (2) Temporal 3D convolution: before feeding video frames into the vision encoder, a 3D convolution with temporal stride 2 is applied, compressing N input frames to N/2 in temporal size. This operates in the pixel space before the ViT, reducing the number of frames that need to be processed by the expensive transformer by a factor of 2. (3) Token interpolative downsampling: after the ViT and projector produce visual tokens, if the total number of visual tokens exceeds a predefined limit, they are further downsampled via interpolation. This three-stage compression — spatial (rescaling), temporal (3D conv), and token-level (interpolation) — progressively reduces the computational cost while attempting to preserve the most important visual information at each stage.

Streaming Audio-Visual Interaction: Chunk-Wise Interleaving and Sparse-Dense Sampling

The streaming interaction mechanism, described in Section 2.4.2, is the architectural component that enables real-time audio-visual conversation. It rests on two key design choices:

Streaming Audio-Visual Feature Interleaving. Unlike offline multimodal understanding, where all audio and visual features can be concatenated at the sequence level before processing, real-time interaction requires that features be "prefilled into the LLM backbone as early as possible to minimize response latency." The solution is a temporally-synchronized, chunk-wise interleaving mechanism. Features are chunked into 1-second segments (during user speaking) or 2-second segments (during model response). Within each chunk, the format is structured as:

<|timestamp|>:<|video-tokens|><|audio-start-token|><|audio-tokens|>

where the timestamp is the textual representation, the video tokens are the chunk's frame encodings, and the audio tokens are the chunk's audio encoder features (or discrete tokens, depending on training stage). The <|audio-start-token|> and <|audio-end-token|> markers delineate audio segments. This structured interleaving ensures the LLM can associate visual and auditory information from the same time window — it sees "at second 3, this is what the camera saw, and this is what the microphone heard" — enabling cross-modal reasoning about time-aligned events (e.g., "the sound of the door slamming coincided with the person entering the frame").

Sparse-Dense Sampling Strategy. During the user's speaking turn, the system uses dense sampling with 1-second chunk size and 2 FPS to "preserve as much audio-visual information as possible." During the model's response period, video frames are buffered with a sparser sampling rate — 2-second chunk size at 0.5 FPS — and prepended to the next user turn. This "effectively balances visual information retention during the model response period and computational overhead." Why does this work? When the model is speaking, the user is typically listening and not providing new information — the visual scene may change, but those changes are rarely critical to understand in real time. By sparsely sampling during the response period, the system saves significant computation (4× fewer frames processed) while still capturing enough visual context to maintain continuity when the user speaks again. The dense sampling during user speech ensures that when the user says something like "what's that?" while pointing, the visual information from the precise moment of the gesture is captured at full resolution.

Post-Training Pipeline: SFT and Joint Text-Speech DPO

Supervised Fine-Tuning (Section 4.1). The SFT stage uses approximately 3 million curated image-text SFT samples and ~700K video-text SFT samples, plus audio understanding data, vision-speech QA data, and audio-visual interaction data. The training freezes the audio encoder (preserving its robust acoustic representations from Stage-5) while updating all other modules (vision encoder, projector, LLM, audio heads). This selective freezing "stabilizes convergence and avoids catastrophic forgetting of low-level auditory features." Training uses AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , weight decay 0.1, learning rate $1 \times 10^{-5}$ with linear warm-up over the first 4% of steps followed by cosine decay to zero, batch size 1024, one epoch.

A key SFT innovation is the vision-speech QA data construction: text-based QA pairs from existing SFT datasets are rewritten by an LLM for spoken fluency and naturalness, then converted to speech using a TTS engine. This bridges offline vision-language capabilities to spoken interaction — the model learns to answer questions about images and videos in natural speech rather than text, without requiring separate speech-specific training data.

The audio-visual interaction data pipeline in Section 4.1.2 is particularly sophisticated. It uses a semi-automated, human-in-the-loop approach: (1) Model-driven automation: videos are segmented into scenes, a multimodal LLM generates progressively deeper, context-aware QA pairs with logical progression and referential dependency across turns, an LLM-as-a-judge framework evaluates and discards low-quality pairs, and remaining pairs are composed into multi-turn dialogues. (2) Human-in-the-loop verification: human annotators correct five specific error types — factual inconsistency, response insufficiency, referential ambiguity, linguistic infelicity, and semantic irrelevance — that are difficult for automated systems to fix. (3) TTS synthesis converts the verified textual dialogues to speech and integrates them into the videos, producing the final audio-visual speech interaction dataset. To enhance long-term memory, a long-context multi-turn variant reorders dialogue sequences so that queries appear at temporal positions distant from their corresponding visual segments, forcing the model to retain information over extended spans.

Direct Preference Optimization with Joint Text-Speech Objective (Section 4.2). The DPO stage extends standard DPO (which optimizes a model to prefer "chosen" over "rejected" responses) to handle the parallel text and multi-codebook speech outputs. The loss is:

$\mathcal{L}_{\text{DPO}} = \alpha \mathcal{L}_{\text{DPO}}(\text{text}_{\text{chosen}}, \text{text}_{\text{rejected}}) + \beta \sum_{i=1}^{N} \mathcal{L}_{\text{DPO}}(\text{audio}^i_{\text{chosen}}, \text{audio}^i_{\text{rejected}})$

where N is the number of audio heads (4, corresponding to the 4 codebooks), $\alpha : \beta = 1 : 1$ is the ratio balancing text and speech optimization, $\mathcal{L}_{\text{DPO}}(\text{text}_{\text{chosen}}, \text{text}_{\text{rejected}})$ is the standard DPO loss on text outputs, and each $\mathcal{L}_{\text{DPO}}(\text{audio}^i_{\text{chosen}}, \text{audio}^i_{\text{rejected}})$ is the DPO loss on the i-th audio codebook stream.

What this loss computes: For each response, the model generates 6 rollouts, a robust multimodal LLM scores them and identifies chosen/rejected pairs, and the DPO loss increases the log-probability of chosen responses relative to rejected ones. The $\alpha$ term handles the semantic quality of text output. Each $\beta$ term handles both linguistic content (via the semantic codebook) and pronunciation stability (via the acoustic codebooks) of the corresponding speech output. The sum means all four codebooks are jointly optimized to produce coherent speech — the semantic codebook must capture the right words, and the acoustic codebooks must produce a natural-sounding realization of those words.

Why this joint optimization: Most existing DPO variants for speech models "optimize text and speech separately," which the paper argues "is suboptimal for maintaining coherence between textual and speech responses." Separate optimization could result in semantically correct text that is spoken with inappropriate prosody, or natural-sounding speech that says the wrong thing. Joint optimization ensures the text semantics and speech acoustics are aligned — the chosen speech not only sounds good but also captures the same meaning as the chosen text. Training uses one epoch, batch size 256 with balanced modality mixing, learning rate $1 \times 10^{-6}$ with cosine decay and warm-up fraction 0.03, and a KL divergence regularizer with weighting factor 0.1 to prevent drift from the SFT model.

Training Infrastructure: Modality-Decoupled Parallelism and Memory Optimization

The infrastructure design (Section 5) addresses the core challenge that the three model components have drastically different computational requirements (Table 2: audio encoder 0.01–110 TFLOPs, vision encoder 0.08–400 TFLOPs, LLM decoder 1920–4668 TFLOPs per micro-batch). Applying a uniform parallelism strategy to all components would waste GPU resources during encoder execution or create pipeline bubbles.

Modality-Decoupled Parallelism (MDP). The core idea is to "completely decouple the modality encoders and the LLM backbone at the distributed level, enabling independent scheduling." The implementation co-locates modality encoders and LLM on the same GPUs but uses different parallelism strategies: modality encoders use Hybrid Sharding Data Parallelism (HSDP) with full activation recomputation, while the LLM uses a combined strategy of pipeline parallelism (PP), ZeRO-1 data parallelism (DP), context parallelism (CP), and expert parallelism (EP). An InnerDP dimension is introduced where d_{\text{inner\_dp}} = d_{\text{lm\_cp}} \times d_{\text{lm\_pp}} — the product of the LLM's context parallelism and pipeline parallelism degrees. The DP ranks of modality encoders map one-to-one with the LLM's DP ranks, and the InnerDP dimension partitions modality data across microbatches.

The execution timeline in Figure 8 shows four phases per training iteration: (1) Data Loading: the inner_dp = 0 rank fetches all micro-batches and broadcasts metadata to other DP ranks, with samples sorted by text sequence length to balance workload. (2) Modality Encoder Forward: the BalanceData module distributes modality data across inner_dp ranks, encoders compute vision and audio embeddings, and the ModalityBridge aggregates embeddings at inner_dp = 0. (3) LLM Decoder Forward and Backward: modality embeddings are partitioned on CP ranks and fed through the LLM, with gradients returned for the encoder backward pass. (4) Modality Encoder Backward: ModalityBridge redistributes embedding gradients from the LLM to inner_dp ranks, and encoder backward passes execute. This design ensures "isolated optimization for modality encoders" while maintaining the LLM's training efficiency.

ModalityBridge with Chunk-Based Processing. The ModalityBridge is the communication layer that transforms data organization between the modality encoders and the LLM decoder, handling both forward embeddings and backward gradients. The key challenge is memory pressure at long context lengths — the inner_dp = 0 rank must read all micro-batches and perform gather/scatter on modality embeddings. The solution is chunk-based processing with three components: (1) Embedding Redistribution: a two-stage chunking approach decomposing gather and scatter into num_chunk iterations, where each iteration aggregates inner_dp embeddings at inner_dp = 0 and partitions along the hidden_size dimension for CP ranks. (2) Modality Embedding Storage: stores chunked data with global offset information for indexing. (3) Embedding Indexing: provides micro-batch-level embedding retrieval during LLM forward and gradient backpropagation during backward. This chunking reduces peak memory to 1/num_chunk of the original while maintaining bitwise numerical consistency.

Memory Optimization Strategies (Table 4). Starting from a naive memory footprint of 137 GB per device (exceeding the 80 GB GPU limit), the paper applies a sequence of optimizations: (1) HSDP for modality encoders reduces static memory from 20.4 to 17.5 GB. (2) V-half (half-precision for activations) reduces dynamic memory from 103.2 to 58.6 GB. (3) Selective recomputation for low-FLOP, high-activation operators (SwiGLU, LayerNorm) further reduces dynamic memory to 49.2 GB. (4) Memory-efficient permute moves aggregation of routing probabilities and hidden states into the SwiGLU operation to reduce MoE unpermute memory, bringing dynamic memory to 42.8 GB. (5) NCCL memory optimization through fine-grained SM budgeting for communication kernels reduces NCCL overhead from 13.4 to 8.9 GB. The final footprint is 69.1 GB, safely within the ~72 GB practical budget (accounting for EP imbalance peaks). For additional safety, "dynamic expert recomputation" is triggered when any rank receives too many tokens — it recomputes the down-projection, freeing all MoE memory except down-projection inputs, preventing crashes from expert imbalance.

The paper claims this infrastructure sustains "over 90% of the throughput achieved by text-only training" — an important practical result implying multimodal training is only ~10% more expensive per token than text-only training at equivalent scale.

Inference and Deployment: Asynchronous Streaming Pipeline

Section 6 describes the deployment architecture that achieves millisecond-level response latency. The system is organized as a decoupled multimodal inference framework where modality-specific encoders/decoders and the LLM run on separate hardware accelerators tailored to their computational characteristics, with dedicated optimizations per component. The asynchronous streaming pipeline (Figure 11) has four sequentially linked but concurrently executed stages:

VAD & Frame Sampling: A voice activity detection module monitors whether the user is speaking in real time. When speaking, dense frames (2 FPS) and full audio are captured. When silent, sparse frames (0.5 FPS) track the high-level scene overview.
Audio-Visual Encoding: Audio and vision encoders process their respective streams incrementally as data arrives, producing features chunk-by-chunk for the LLM.
LLM Prefilling & Decoding: Incoming chunks are immediately prefilled (using the streaming prefill strategy) rather than waiting for the turn to end. A speculative prefill-decode switching strategy initiates decoding at an early speculative endpoint (t3) before the VAD confirms the turn has ended (t4). If the user resumes speaking, a rollback discards generated content and reverts to prefill mode. If the turn does end, the first packet latency is overlapped with the VAD confirmation window.
Audio Decoding: Audio tokens are streamed to the audio decoder for waveform reconstruction. Audio delivery to the user begins only when the VAD explicitly detects turn end (t4), even if packets are generated earlier. User interruption (t5) immediately terminates generation, with the LLM token sequence truncated at the nearest natural breakpoint (punctuation mark).

The paper reports that "by overlapping the VAD endpoint detection (600–700ms) and the prefill process, users can receive the model response within 100ms after the endpoint detection" — meaning the perceived latency from the moment the user finishes speaking to the first audio response is approximately 100ms, in the range of natural human conversation turn-taking.

4. Key Insights and Innovations

The dominant assumption in multimodal model development — whether explicit or implicit — has been that modalities can be added to a pretrained LLM in parallel or in arbitrary order, and that the primary challenge is architectural (how to connect encoders to the LLM) rather than sequential (in what order to introduce modalities). This assumption leads to training recipes where vision, audio, and text data are mixed together from the start, or where modalities are added opportunistically based on data availability rather than a principled curriculum. The result, frequently observed but rarely addressed head-on, is that multimodal models often sacrifice unimodal performance — particularly text reasoning — in exchange for broad modality coverage.

LongCat-Flash-Omni's central methodological contribution is the demonstration that the sequence of modality introduction is not a minor hyperparameter but a first-order determinant of final performance, and that a deliberate curriculum — text → speech → images → video → continuous audio — preserves unimodal strength while enabling deep cross-modal integration. This is not the common "curriculum learning" of easy-to-hard examples within a single modality, but a cross-modal curriculum where the nature of the modeling challenge changes qualitatively at each stage: from abstract symbolic sequences (text) to acoustic sequences with paralinguistic richness (speech) to spatial reasoning (images) to spatiotemporal dynamics (video) to continuous acoustic features (audio encoder alignment).

What makes this distinctive as an intellectual contribution — beyond a training recipe — is its diagnostic logic about why this ordering works. The paper's reasoning (Section 3.2) is grounded in the structural properties of each modality: speech is introduced first among non-text modalities because, despite being acoustic, it is "structurally closer to text" — both are sequential signals where temporal order matters. This structural similarity means the LLM's sequence modeling machinery, developed extensively during 16 trillion tokens of text pretraining, can transfer more directly to speech than to vision. Images are introduced only after speech-text alignment is established, because spatial reasoning is fundamentally different from sequential reasoning — the model must learn to attend over 2D grids rather than 1D sequences. Video is deferred to the annealing stage (Stage-3) because it compounds spatial and temporal complexity; introducing it before the model has mastered static image understanding would dilute the learning signal.

This framing reframes catastrophic forgetting from a problem to be mitigated (through replay buffers, elastic weight consolidation, or careful data mixing) into a problem to be designed around through ordering. The evidence that this works is not a single ablation but the consistent pattern across Tables 12 and 13: LongCat-Flash-Omni matches or exceeds LongCat-Flash (its text-only foundation) on text benchmarks — MMLU improves from 87.05 to 90.30, MATH500 from 96.40 to 97.60, HumanEval+ from 65.85 to 90.85 (Table 13 vs. Table 12) — while simultaneously achieving SOTA omni-modal performance. This simultaneous preservation-and-acquisition is the hallmark of a successful curriculum, and it distinguishes the approach from prior work where multimodal training degraded text capabilities.

The paper's innovation is not the idea of curriculum learning — that dates back to Bengio et al. (2009) — but the application of curriculum principles across modalities with a specific theoretical justification grounded in modality structure, and the empirical demonstration that this ordering is what enables a 560B-parameter model to successfully integrate four modalities without sacrificing any. This is a conceptual advance in how to think about multimodal training rather than a novel algorithm: the order matters because modalities differ not just in data format but in the fundamental type of reasoning they require.

Innovation 2: The Difficulty-Dependent Asymmetry Between Text and Speech as the Organizing Principle for Loss Weighting

A less prominent but intellectually significant contribution is the paper's handling of the fundamental asymmetry between text and speech as sequence modeling problems. The paper observes (Section 3.2) that speech, despite being the acoustic realization of language, is substantially harder to model than text for a specific structural reason: "a typical speech tokenizer operating at 12.5 Hz must generate roughly 12.5 tokens per second, [while] humans only speak about 3–4 text tokens per second." This means the semantic density of speech tokens is roughly 3–4× lower than text tokens — most speech tokens carry acoustic detail rather than linguistic content. This asymmetry creates a risk in joint training: if speech and text losses are weighted equally, the speech signal (with its higher token rate) will dominate the optimization, potentially degrading text capabilities.

The paper's response to this is the carefully tuned loss weighting in Stage-1's equation (with a = 1.75, b = 0.25, c = 1.5, d = 0.1), where the pure-text loss receives 7× the weight of the pure-audio loss. This is not an arbitrary hyperparameter choice — it reflects a principled understanding that text tokens carry more semantic information per token than speech tokens, and that preserving text capabilities requires the optimization to treat text signal as more valuable per token. The interleaved audio-text loss (c = 1.5) receives nearly as much weight as pure text, reflecting that cross-modal alignment (learning the mapping between speech acoustics and text semantics) is the primary goal of this stage, while pure speech modeling (b = 0.25) is secondary.

What distinguishes this from standard multi-task loss weighting is the explicit reasoning about per-token information density across modalities. Most multi-modal training recipes treat the loss weight as a hyperparameter to be tuned, not as a quantity with a principled relationship to the structural properties of each modality. The paper's approach — identifying that the token rate ratio (speech tokens per second ≈ 4× text tokens per second) implies a corresponding loss weight ratio — provides a transferable principle: when modalities operate at different temporal resolutions and semantic densities, loss weights should be inversely proportional to token rate to equalize the per-unit-time contribution of each modality to the gradient.

The evidence for this working is not isolated to a single ablation, but the paper reports that "after carefully hyperparameter tuning and extensive preliminary experiments, we set a = 1.75, b = 0.25, c = 1.5, and d = 0.1, which achieves the best balance between preserving text capabilities and enhancing audio performance." The fact that the text-heavy weighting (a = 1.75 vs. b = 0.25) is optimal validates the underlying principle.

Innovation 3: Modality-Decoupled Parallelism as a General Design Pattern for Heterogeneous Multimodal Training

The infrastructure contribution — modality-decoupled parallelism (MDP) — is significant less for the specific implementation than for the design pattern it establishes for handling model heterogeneity in large-scale training. The core insight is deceptively simple: instead of forcing all model components into the same parallelism strategy (the default in most training frameworks), completely separate the parallelism strategies for encoders and the decoder, and handle the cross-component communication through a dedicated bridge module (ModalityBridge) with chunked processing.

Prior to this work, the dominant approaches to multimodal training infrastructure fell into two categories, both of which the paper identifies as inadequate at 560B scale. FSDP-based approaches (OrchMLLM, veOmni) shard all parameters across all devices and rely on data-parallel group balancing to handle heterogeneity — but for a 560B-parameter model, "the total number of model parameters is too large for this approach to be practical" (Section 5.1). Decoupling approaches (DistTrain, PipeWeaver, Optimus) separate parallelization strategies but handle the scheduling differently — PipeWeaver uses fine-grained data partitioning and dynamic pipeline scheduling, while Optimus schedules encoder computations into LLM idle periods to eliminate bubbles. The paper's MDP scheme builds on Optimus but makes the decoupling more complete, introducing the InnerDP dimension that maps encoder DP ranks one-to-one with LLM DP ranks while independently optimizing each component's parallelism.

What makes this a conceptual contribution rather than just an engineering optimization is the explicit recognition that modality heterogeneity is not a bug to be worked around but a structural property to be designed for, and that the right response is not to homogenize (make all components use the same strategy) but to embrace heterogeneity and build communication infrastructure (ModalityBridge) that handles the resulting data format transformations. The claim of "over 90% of the throughput achieved by text-only training" (Section 5) is the empirical validation that this approach works — it demonstrates that the overhead of handling heterogeneous components can be kept to ~10%, making large-scale omni-modal training practically feasible rather than prohibitively expensive.

This innovation is incremental in its technical details (the individual parallelism strategies — HSDP, PP, CP, EP — are all established) but fundamental in its architectural philosophy: it establishes that the right abstraction for multimodal training infrastructure is a decoupled design where each component can be optimized independently, with a clean bridge interface handling cross-component communication. This is analogous to the microservices pattern in distributed systems — it sacrifices some efficiency from tight coupling in exchange for much greater flexibility and robustness to component heterogeneity.

Innovation 4: The Transition from Discrete Speech Tokens to Continuous Audio Features as a Two-Phase Strategy

The paper's handling of speech representation — using discrete four-codebook tokens during large-scale pretraining (stages 1–4) and then transitioning to continuous audio encoder features (stage 5) — represents a pragmatic resolution of a genuine tension in speech-language model design. The tension is: discrete tokens enable the LLM to process speech using the same autoregressive next-token prediction paradigm as text, making large-scale training efficient and architecturally simple; but discretization is inherently lossy, "hindering the model's ability to capture fine-grained acoustic details" (Section 3.2). Continuous audio features preserve acoustic richness but break the discrete-token paradigm, requiring a separate encoder and projector that must be aligned with the LLM's semantic space.

Prior work has largely chosen one side of this tradeoff: models like AudioLM and VALL-E commit fully to discrete tokens, accepting the acoustic quality loss as the price of architectural simplicity. Models like Qwen-Audio and SALMONN use continuous audio features throughout, accepting the architectural complexity of a separate encoder. The paper's innovation is the two-phase strategy that uses each representation where it is most valuable: discrete tokens during the compute-intensive early training stages (where the simplicity of next-token prediction matters most), and continuous features only at the final alignment stage (where acoustic fidelity matters and the LLM is frozen, so training cost is dominated by the lightweight audio encoder).

This is not just a training efficiency hack — it reflects a deeper insight about the different roles that speech plays in understanding versus generation. For understanding (ASR, audio QA, paralinguistic analysis), fine-grained acoustic details — pitch contours, breathiness, emphasis patterns — carry important meaning that discrete tokens may discard. For generation (TTS), the four-codebook discrete representation already captures sufficient acoustic detail for natural-sounding speech (as the TTS results in Table 7 demonstrate). The two-phase strategy allows the model to develop strong speech understanding (through the continuous audio encoder) without sacrificing the generation quality enabled by discrete token training.

The evidence for this working is the ASR and audio understanding results in Tables 7, 9, and 10. The base model after Stage-1 (discrete tokens only) achieves 1.98% WER on LibriSpeech test-clean — competitive but not SOTA. After Stage-5 (continuous audio encoder), the instruct model achieves 1.57% WER, outperforming Gemini-2.5-Pro (1.74%), GPT-4o-Audio (30.00%), and Qwen3-Omni (1.22%). More importantly, the transition does not degrade speech generation: the TTS evaluations in Table 7 show generation quality actually improves from Stage-1 (4.72% WER on LibriSpeech) to Stage-3 (3.68%), suggesting that later-stage training with continuous features does not harm the discrete-token generation capability learned earlier.

This contribution is incremental in its technical components (discrete audio tokenizers and continuous audio encoders both predate this work) but conceptually significant as a design pattern: it demonstrates that for modalities where lossy discretization is feasible, a two-phase strategy — discrete for efficient scale-up, continuous for fidelity — can capture the benefits of both approaches while mitigating their respective weaknesses. This pattern could generalize to other modalities with similar efficiency-fidelity tradeoffs, such as high-resolution images (discrete VQ tokens for efficient pretraining, continuous features for fine-grained understanding).

5. Experimental Analysis

Evaluation Methodology

Datasets. The evaluation spans four modality axes, each with multiple benchmarks. For vision, the paper uses MMBench (EN and ZH variants for general understanding), RealWorldQA (real-world visual QA), MMStar (general multimodal), MathVista (math reasoning with visuals), MMMU (multi-discipline expert-level understanding), MMVet (integrated vision-language capabilities), BLINK and MuirBench and Mantis (multi-image understanding), ChartQA, DocVQA, OCRBench, and OmniDocBench (text recognition and document understanding), RefCOCO and CountBench (grounding and counting), VisualWebBench, ScreenSpot-v2, and AndroidControl (GUI). For video, the paper uses MVBench, NextQA, and TempCompass (short video), VideoMME and LongVideoBench (long video), MMVU and Video-MMMU (STEM and reasoning). For audio, the paper uses LibriSpeech, AISHELL-1, AISHELL-2, FLEURS, CommonVoice15, and WenetSpeech (ASR), CoVost2 (speech-to-text translation), MMAU, VocalSound, TUT2017, ClothoAQA, Nonspeech7k, CochlScene, and MELD (audio understanding), and OpenAudioBench and VoiceBench (audio-to-text chat). For text, the paper uses MMLU, MMLU-Pro, CEval, and CMMLU (general domains), IFEval, COLLIE, and Meeseeks (instruction following), MATH500, AIME24, and BeyondAIME (math), GPQA-diamond, DROP, ZebraLogic, and GraphWalks (reasoning), and HumanEval+, MBPP+, and LiveCodeBench (coding). For cross-modality, the paper uses OmniBench, WorldSense, DailyOmni, and a newly introduced UNO-Bench with 1,880 human-crafted questions. For real-time audio-visual interaction, the paper uses a proprietary evaluation framework with 200 multi-turn dialogue sessions per model rated by 250 independent annotators on a 0–3 naturalness scale. All benchmarks are evaluated under their official configurations, with Gemini-2.5-Pro constrained to a 128-token thinking budget for fair comparison with instruct models (Section 7).
Base models. The primary model is LongCat-Flash-Omni (560B total parameters, 27B activated on average via ScMoE). Comparisons are made against proprietary omni-modal models (Gemini-2.5-Pro, Gemini-2.5-Flash, GPT-4o, GPT-4o-Audio), open-source omni-modal models (Qwen3-Omni-Instruct, Qwen2.5-Omni-Instruct), vision-language models (Qwen3-VL-235B-A22B-Instruct, Qwen2.5-VL-72B-Instruct, Seed-1.6), audio models (Kimi-Audio, Step-Audio-2-mini), text models (DeepSeek-V3.1, Qwen3-235B-A22B, Kimi-K2, GPT-4.1, Claude Sonnet-4, Gemini-2.5-Flash, LongCat-Flash), and real-time interaction products (Doubao, GPT-4o, iFlytek Spark, StepFun, ChatGLM, Qwen2.5-Omni, Qwen3-Omni). The variety of baselines reflects the paper's claim of being a unified omni-modal model — it must be evaluated against specialized models in each modality and against direct omni-modal competitors (Sections 7.1–7.4).
Metrics. Accuracy (%) is the primary metric across most vision, audio understanding, and text benchmarks — it measures the fraction of questions for which the model's selected answer matches the ground truth. For ASR and TTS, the paper uses Word Error Rate (WER) for English and Character Error Rate (CER) for Chinese — the edit distance between the predicted transcription and the reference, divided by the reference length, expressed as a percentage (lower is better). For speech-to-text translation, BLEU score is used (higher is better). For real-time interaction, human annotators rate naturalness and fluency on a 0–3 scale (0 = completely unnatural, 3 = completely natural and fluent), with results reported as mean scores with 95% confidence intervals. For coding benchmarks, pass@1 is reported — the fraction of problems for which the model's first generated solution passes all test cases. For specific math benchmarks (AIME24, BeyondAIME), avg@10 is reported — the average accuracy over 10 sampled solutions per problem (Section 7).
Baselines. The paper compares against an extensive set: (1) Proprietary omni-models: Gemini-2.5-Pro (thinking budget constrained to 128 tokens for fairness), Gemini-2.5-Flash (non-thinking mode), GPT-4o (vision and audio variants), GPT-4.1 (text). (2) Open-source omni-models: Qwen3-Omni-Instruct (from Alibaba, the most direct open-source competitor), Qwen2.5-Omni-Instruct (previous generation). (3) Vision-language models: Qwen3-VL-235B-A22B-Instruct, Qwen2.5-VL-72B-Instruct, Seed-1.6 (ByteDance's vision model). (4) Audio models: Kimi-Audio (from Moonshot), Step-Audio-2-mini (from StepFun). (5) Text models: DeepSeek-V3.1 (671B MoE, 37B activated), Qwen3-235B-A22B (2507 version), Kimi-K2 (1043B MoE, 32B activated), Claude Sonnet-4, LongCat-Flash (the text-only predecessor). (6) Interaction products: Doubao (ByteDance), iFlytek Spark, StepFun, ChatGLM. For closed-source models, evaluations are conducted through official APIs. For models supporting thinking/non-thinking modes, non-thinking mode is explicitly configured for fair comparison (Section 7.3.2).
Generation budget / compute accounting. The paper does not report a standardized compute budget for inference-time evaluation, as the focus is on model capability comparisons using official evaluation protocols for each benchmark. For the real-time interaction evaluation, latency is measured experimentally: the asynchronous streaming pipeline achieves ~100ms first-packet latency after VAD endpoint detection (Section 6.2), but no direct latency comparisons against other models are reported — qualitative real-timeness is instead assessed through the human annotation framework (Table 16, "Real-timeness" dimension). The paper does not conduct FLOPs-matched comparisons between models of different scales for inference.
Cross-validation / statistical protocol. For the real-time audio-visual interaction evaluation, 250 independent users triple-annotated the complete interaction videos, yielding mean naturalness scores with 95% confidence intervals reported in Table 15. For all other benchmarks, standard evaluation splits and official scoring protocols are used without cross-validation or statistical significance testing. The paper notes that OmniBench uses an "internally corrected version" because "the publicly released version contains scoring deficiencies" (Section 7.4.1) — a detail that limits direct comparability with published results on the original benchmark version. For the proprietary interaction evaluation, 10 trained professional conversationalists conducted approximately 3-minute multi-turn dialogue sessions with each model, collecting 200 sessions per model across four scenario categories (problem solving, entertainment, self-improvement, emotional support, 50 samples each). Qualitative analysis was performed by expert annotators across six dimensions (real-timeness, human-likeness, paralinguistic understanding, relevance, accuracy, memory capability), with the percentage of "good cases" reported per dimension (Table 16).

Main Quantitative Results

Vision Capability: Image Understanding

Table 5 presents results across 21 image understanding benchmarks. LongCat-Flash-Omni achieves performance comparable to Gemini-2.5-Flash and consistently outperforms Qwen3-Omni-Instruct, with the advantage particularly pronounced on multi-image tasks.

On general-domain benchmarks, LongCat-Flash-Omni achieves 87.5 on MMBench-EN (vs. 89.8 for Gemini-2.5-Pro ThinkingBudget128, 89.3 for Gemini-2.5-Flash, 86.8 for Qwen3-Omni), 88.7 on MMBench-ZH (vs. 89.2, 88.5, 86.4), 74.8 on RealWorldQA (vs. 76.0, 73.9, 72.9), and 70.9 on MMStar (vs. 78.5, 75.5, 68.5). The model performs at the level of Gemini-2.5-Flash on general understanding while modestly trailing Gemini-2.5-Pro.

On STEM and reasoning benchmarks, LongCat-Flash-Omni achieves 77.9 on MathVista (vs. 77.7 for Gemini-2.5-Pro, 77.1 for Gemini-2.5-Flash, 75.9 for Qwen3-Omni, and notably below Qwen3-VL at 84.9), 70.7 on MMMU (vs. 80.9, 76.3, 69.1), and 69.0 on MMVet (vs. 80.7, 79.5, 68.9). The MMMU score places LongCat-Flash-Omni above Qwen3-Omni (69.1) but meaningfully below Gemini-2.5-Pro (80.9), indicating remaining gaps in expert-level multi-discipline reasoning.

On multi-image benchmarks — which the paper identifies as a particular strength — LongCat-Flash-Omni achieves 63.1 on BLINK (vs. 56.1 for Qwen3-Omni), 77.1 on MuirBench (vs. 74.0 for Gemini-2.5-Pro, 73.7 for Gemini-2.5-Flash, 62.1 for Qwen3-Omni), and 84.8 on Mantis (vs. 83.9, 83.4, 80.7). The MuirBench result is particularly notable: a 15-point advantage over Qwen3-Omni and a 3-point advantage over Gemini-2.5-Pro. The paper attributes this to "high-quality interleaved image-text, multi-image data and video datasets" (Section 7.1.1).

On text recognition and document understanding, LongCat-Flash-Omni achieves 87.6 on ChartQA (vs. 86.8 for Qwen3-Omni, but below Qwen3-VL at 89.2), 91.8 on DocVQA (vs. 95.7 for Qwen3-Omni — the one clear regression relative to the open-source competitor), and 84.9 on OCRBench (vs. 85.5 for Qwen3-Omni, 91.2 for Qwen3-VL). The OmniDocBench results show 22.8/29.0 for EN/ZH (vs. 28.4/40.5 for Qwen3-Omni — substantially worse on Chinese document parsing).

On grounding, counting, and GUI, LongCat-Flash-Omni achieves 93.9 on RefCOCO-avg (vs. 91.6 for Qwen3-Omni), 92.4 on CountBench (vs. 90.0), and 91.2 on ScreenSpot-v2 (vs. 94.7 for Qwen3-Omni — one of the few GUI benchmarks where the open-source competitor leads). On AndroidControl, LongCat-Flash-Omni achieves 91.2 vs. 90.5 for Qwen3-Omni.

Vision Capability: Video Understanding

Table 6 presents results across 8 video understanding benchmarks. LongCat-Flash-Omni achieves state-of-the-art performance on short video understanding, significantly surpassing all compared models, while performing on par with leading models on long video tasks.

On short video benchmarks, LongCat-Flash-Omni achieves 75.2 on MVBench (vs. 66.4 for Gemini-2.5-Pro, 63.0 for Gemini-2.5-Flash, 69.3 for Qwen3-Omni), 86.2 on NextQA (vs. 84.2, 81.4, 82.4), and 82.2 on TempCompass (vs. 80.8, 80.2, 73.5). The MVBench lead is 8.8 points over Gemini-2.5-Pro and 5.9 points over Qwen3-Omni — a substantial margin on short-form temporal reasoning. The paper attributes this to "an advanced video processing strategy — using dynamic frame sampling and hierarchical token aggregation — and the strong long-context modeling capacity afforded by its efficient backbone" (Section 7.1.2).

On long video benchmarks, LongCat-Flash-Omni achieves 78.2 on VideoMME with audio (vs. 80.6 for Gemini-2.5-Pro, 78.5 for Gemini-2.5-Flash, 73.0 for Qwen3-Omni) — the best omni-modal result, though trailing Qwen3-VL at 79.2. On VideoMME without audio, the model achieves 76.2 (vs. 70.5 for Qwen3-Omni, 75.2 for Seed-1.6, and 79.2 for Qwen3-VL — the vision-language model that cannot process audio). On LongVideoBench, LongCat-Flash-Omni achieves 69.3 (vs. 69.4 for Gemini-2.5-Pro, 66.4 for Gemini-2.5-Flash, 65.4 for Qwen3-Omni).

On STEM and reasoning video benchmarks, LongCat-Flash-Omni achieves 67.1 on MMVU (vs. 75.6 for Gemini-2.5-Pro, 72.4 for Gemini-2.5-Flash, 62.4 for Qwen3-Omni) and 67.5 on Video-MMMU (vs. 79.4, 76.6, 60.3). The MMVU and Video-MMMU gaps to Gemini-2.5-Pro (8.5 and 11.9 points respectively) are substantial, indicating that video-based STEM reasoning remains an area where the proprietary model maintains a clear advantage.

Audio Capability: Base Model Evaluation

Tables 7 and 8 present base model evaluations across pretraining stages. For ASR, the Stage-3 base model achieves the best results: 3.01% CER on SpeechIO02, 1.93% WER on LibriSpeech test-clean, 3.74% WER on LibriSpeech test-other. The performance is remarkably stable across stages — Stage-2 shows slight regression (2.11% test-clean, 4.59% test-other) likely due to the introduction of visual data temporarily disrupting speech representations, with recovery in Stage-3. The context extension in Stage-4 (32K and 128K) causes minor ASR degradation (2.30% and 2.12% test-clean respectively), suggesting that long-context training modestly impacts short-form speech recognition — expected since the model's attention must span longer ranges, potentially diluting focus on local acoustic patterns.

For TTS, the best performance is at Stage-4 (32K) with 1.73% CER on SpeechIO02, though Stage-3 achieves better LibriSpeech TTS (3.68% WER vs. 5.99%). The TTS quality varies non-monotonically across stages — Stage-2 shows substantial regression on LibriSpeech TTS (8.64% WER, nearly double Stage-1's 4.72%), consistent with disruptive effects from visual data introduction, but Stage-3 recovers strongly. The conclusion is that the progressive curriculum successfully maintains both ASR and TTS capabilities through multimodal integration, with only temporary degradations that subsequent stages correct.

For speech continuation (Table 8), the base model's ability to respond to questions in either text or speech format improves markedly from Stage-1 (88.80 text, 84.80 speech) to Stage-3 (92.80 text, 92.00 speech), with the text-speech gap closing from 4 points to 0.8 points. Stage-4 maintains performance at 128K context (90.40 for both output modes), demonstrating that long-context extension does not degrade the speech continuation capability.

Audio Capability: Instruct Model Evaluation — ASR and S2TT

Table 9 presents ASR and speech-to-text translation results for the instruct model. LongCat-Flash-Omni achieves competitive ASR performance, though not uniformly best-in-class. On LibriSpeech, the model achieves 1.57%/4.01% WER on test-clean/test-other, compared to Gemini-2.5-Pro (1.74/3.80), GPT-4o-Audio (30.00/41.83 — notably poor, suggesting GPT-4o-Audio's ASR pipeline is not optimized for this benchmark), Qwen3-Omni (1.22/2.48), Kimi-Audio (1.28/2.42), and Step-Audio-2-mini (1.33/2.86). LongCat-Flash-Omni trails Qwen3-Omni and Kimi-Audio on English ASR by small margins (0.35–0.35 and 1.53–1.59 points on clean and other respectively).

On Chinese ASR benchmarks, LongCat-Flash-Omni achieves 0.63% CER on AISHELL-1 (vs. 3.11 for Gemini-2.5-Pro, 0.84 for Qwen3-Omni, 0.60 for Kimi-Audio — lowest CER among all models), 2.78% on AISHELL-2 (vs. 2.34 for Qwen3-Omni, 2.56 for Kimi-Audio), and 6.69%/6.09% on WenetSpeech test-meeting/test-net (vs. 5.89/4.69 for Qwen3-Omni, 6.28/5.37 for Kimi-Audio). The AISHELL-1 result is the strongest showing, while WenetSpeech shows modest but non-trivial gaps to Qwen3-Omni.

On multilingual ASR (FLEURS, CommonVoice15), LongCat-Flash-Omni achieves competitive results — 3.99/5.02 on FLEURS zh/en (vs. 2.20/2.72 for Qwen3-Omni), 4.98/13.59 on CommonVoice15 zh/en (vs. 4.31/6.05). The CommonVoice15 English result (13.59%) is notably worse than Qwen3-Omni (6.05%) and Step-Audio-2-mini (6.75%), suggesting the model's English ASR on diverse, crowd-sourced speech is a relative weakness. Gemini-2.5-Pro performs poorly on CommonVoice15 (47.30/49.86), likely reflecting the constrained thinking budget's impact on transcription tasks.

For speech-to-text translation (BLEU), LongCat-Flash-Omni achieves 47.23 on CoVost2 en→zh (vs. 41.94 for Gemini-2.5-Pro, 48.72 for Qwen3-Omni, 49.12 for Step-Audio-2-mini) and 27.32 on CoVost2 zh→en (vs. 25.38, 21.51, 29.47). The model is competitive but not dominant — Qwen3-Omni leads on en→zh and Step-Audio-2-mini leads on zh→en by small margins (1.49 and 2.15 BLEU points respectively).

Audio Capability: Instruct Model Evaluation — Audio Understanding

Table 10 presents audio understanding results. LongCat-Flash-Omni achieves strong performance across diverse audio tasks: 75.90 on MMAU (vs. 72.80 for Gemini-2.5-Pro, 77.50 for Qwen3-Omni, 68.40 for GPT-4o-Audio), 92.76 on VocalSound (vs. 89.45, 91.60, 82.37), 65.43 on TUT2017 (vs. 33.15, 40.74, 20.74 — a substantial 24.69-point lead over Qwen3-Omni), 72.83 on ClothoAQA (vs. 69.67, 75.16, 61.87), 93.79 on Nonspeech7k (vs. 87.59, 80.83, 72.28), 70.02 on CochlScene (vs. 45.34, 43.03, 34.94 — another large lead, though Kimi-Audio achieves 80.42), and 54.60 on MELD (vs. 46.74, 50.80, 39.00 — best among the compared models except Kimi-Audio at 59.13).

The pattern is that LongCat-Flash-Omni consistently outperforms GPT-4o-Audio across all audio understanding benchmarks, often by large margins (e.g., +52.28 on Nonspeech7k, +44.69 on TUT2017), and generally outperforms Gemini-2.5-Pro, though Qwen3-Omni and Kimi-Audio are competitive or superior on specific tasks. The TUT2017 (acoustic scene classification) and CochlScene results suggest particular strength in environmental sound understanding, while Kimi-Audio's lead on CochlScene (80.42 vs. 70.02) and MELD (59.13 vs. 54.60) indicates that specialized audio models retain advantages on certain fine-grained acoustic tasks.

Audio Capability: Audio-to-Text Chat

Table 11 presents results on audio-driven conversational benchmarks. LongCat-Flash-Omni achieves strong and balanced performance: 83.33 on OpenAudioBench LlamaQuestions (tied with Qwen3-Omni at 83.30, vs. 83.00 for Gemini-2.5-Pro), 79.71 on ReasoningQA (vs. 80.30, 84.16), 86.20 on TriviaQA (vs. 90.20, 75.90), 76.00 on Webquestions (vs. 80.90, 75.20), and 75.43 on AlpacaEval (vs. 76.58, 85.43 — notably below Qwen3-Omni on this instruction-following metric).

On VoiceBench, LongCat-Flash-Omni achieves 4.94 on AlpacaEval (vs. 4.70 for Gemini-2.5-Pro, 4.74 for Qwen3-Omni — the highest among compared models), 4.32 on CommonEval (vs. 4.11, 4.54), 93.41 on OpenBookQA (vs. 95.16, 89.70), 82.46 on SDQA (vs. 83.54, 76.90), 81.95 on MMSU (vs. 88.32, 69.00), 100 on AdvBench (vs. 97.69, 99.30), and 77.99 on IFEval (vs. 77.83, 77.80 — essentially tied with competitors). The AdvBench perfect score (100%) and the MMSU result (81.95 vs. 69.00 for Qwen3-Omni, a 12.95-point advantage) represent particular strengths in adversarial robustness and multimodal speech understanding respectively.

Text Capability: Base Model Evaluation

Table 12 compares LongCat-Flash-Omni Base (after Stage-5) with other text base models. The model maintains performance on par with its text-only predecessor LongCat-Flash Base across all dimensions, with some improvements: 86.81 on MMLU (vs. 87.05, essentially identical), 69.05 on MMLU-Pro (vs. 70.32, a small 1.27-point decrease), 87.95 on CEval (vs. 87.73), 87.14 on CMMLU (vs. 87.19 — unchanged). On reasoning: 51.76 on GPQA (vs. 51.09, a slight improvement), 90.42 on BBH (vs. 90.54), 80.75 on DROP (vs. 78.39, a 2.36-point improvement). On math: 93.10 on GSM8K (vs. 92.19, a 0.91-point improvement) and 66.80 on MATH (vs. 64.82, a 1.98-point improvement). On coding: 69.51 on HumanEval+ (vs. 65.85, a 3.66-point improvement), 76.46 on MBPP+ (vs. 77.25), 70.76 on MultiPL-E (vs. 69.25), 71.88 on CRUXEval-I (vs. 71.63), and 73.50 on CRUXEval-O (vs. 75.88, a 2.38-point decrease).

The consistent pattern is no systematic degradation — and in several cases improvement — in text capabilities despite the addition of speech, image, and video modalities across Stages 1–5. The improvements on HumanEval+ (+3.66), MATH (+1.98), and DROP (+2.36) are notable and suggest possible positive transfer from multimodal training to certain reasoning tasks, though the paper does not ablate this specifically. The model remains competitive with DeepSeek-V3.1 Base (671B, 37B activated) — slightly lower on MMLU (86.81 vs. 87.46) and GSM8K (93.10 vs. 92.22), but competitive or superior on most other benchmarks despite activating fewer parameters (27B vs. 37B).

Text Capability: Instruct Model Evaluation

Table 13 compares LongCat-Flash-Omni Instruct with frontier chat models. The model demonstrates superior text capability compared to its LongCat-Flash predecessor, with consistently leading performance across domains.

On general domains: 90.30 on MMLU (vs. 89.71 for LongCat-Flash, 90.96 for DeepSeek-V3.1, 90.23 for Qwen3-235B-A22B), 82.73 on MMLU-Pro (vs. 82.68, 84.45, 84.83), 91.68 on CEval (vs. 90.44, 89.21, 92.70), and 89.39 on CMMLU (vs. 84.34, 88.04, 88.14 — a 5.05-point improvement over LongCat-Flash).

On instruction following: 82.44 on IFEval (vs. 89.65 for LongCat-Flash — a surprising 7.21-point degradation), 45.69 on COLLIE (vs. 57.10 — notable regression), and 39.05 on Meeseeks-zh (vs. 43.03 — modest regression). The instruction-following results show consistent degradation relative to the text-only LongCat-Flash, which is the paper's clearest evidence that multimodal training does impact certain text capabilities negatively. The IFEval drop from 89.65 to 82.44 is substantial and suggests that the model's ability to follow precise formatting and constraint-based instructions may have been affected by the introduction of multimodal data and tasks.

On mathematical reasoning: 97.60 on MATH500 (vs. 96.40 for LongCat-Flash, 96.08 for DeepSeek-V3.1, 98.80 for Qwen3-235B-A22B), 72.92 on AIME24 avg@10 (vs. 70.42, 66.30, 81.67), and 47.40 on BeyondAIME avg@10 (vs. 43.00, 36.50, 57.60). The MATH500 improvement (+1.20 over LongCat-Flash) and AIME24 improvement (+2.50) continue the pattern observed in the base model, suggesting positive transfer to mathematical reasoning from multimodal training.

On general reasoning: 74.41 on GPQA-diamond (vs. 73.23 for LongCat-Flash, 77.43 for Qwen3-235B-A22B), 83.53 on DROP (vs. 79.06 — a 4.47-point improvement), 86.00 on ZebraLogic (vs. 89.30 — a 3.30-point regression), and 56.00 on GraphWalks-128K (vs. 51.05 — a 4.95-point improvement). The DROP and GraphWalks improvements suggest enhanced reasoning over structured information, possibly benefiting from multimodal training's emphasis on spatial and relational reasoning.

On coding: 52.64 on LiveCodeBench (vs. 48.02 for LongCat-Flash — a 4.62-point improvement), 90.85 on HumanEval+ (vs. 88.41), and 80.16 on MBPP+ (vs. 79.63). Consistent moderate improvements across all coding benchmarks suggest positive transfer from multimodal training to code generation.

Cross-Modality Understanding Evaluation

Table 14 presents cross-modal understanding results. LongCat-Flash-Omni outperforms Gemini-2.5-Flash non-thinking and achieves performance comparable to Gemini-2.5-Pro ThinkingBudget128, while substantially leading open-source competitors.

On OmniBench: 61.38 (vs. 66.80 for Gemini-2.5-Pro, 54.99 for Gemini-2.5-Flash, 58.41 for Qwen3-Omni, 48.16 for Qwen2.5-Omni). LongCat-Flash-Omni surpasses Qwen3-Omni by 2.97 points and Qwen2.5-Omni by 13.22 points, establishing it as the leading open-source omni-modal model on this benchmark, though trailing Gemini-2.5-Pro by 5.42 points.

On WorldSense: 60.89 (vs. 63.96 for Gemini-2.5-Pro, 58.72 for Gemini-2.5-Flash, 52.01 for Qwen3-Omni, 46.69 for Qwen2.5-Omni). The 8.88-point lead over Qwen3-Omni is the largest margin among open-source comparisons on this benchmark, which "emphasizes real-world audio-video understanding" (Section 7.4.1).

On DailyOmni: 82.38 (vs. 80.61 for Gemini-2.5-Pro, 80.78 for Gemini-2.5-Flash, 69.33 for Qwen3-Omni, 47.45 for Qwen2.5-Omni). LongCat-Flash-Omni achieves the highest score among all compared models — including proprietary ones — with a 13.05-point lead over Qwen3-Omni. The DailyOmni benchmark focuses on "audio-visual reasoning with temporal alignment across modalities" (Zhou et al., 2025), and this result is the paper's strongest single-benchmark evidence for superior cross-modal temporal reasoning.

On UNO-Bench (the paper's newly introduced benchmark): 49.90 (vs. 64.48 for Gemini-2.5-Pro, 54.30 for Gemini-2.5-Flash, 42.10 for Qwen3-Omni, 32.60 for Qwen2.5-Omni). The substantial gap to Gemini-2.5-Pro (14.58 points) on this benchmark — designed to have 98% of its 1,880 questions requiring cross-modal reasoning — indicates that proprietary models maintain a significant advantage on complex cross-modal reasoning tasks that the paper's manual annotation process was specifically designed to capture.

Real-Time Audio-Visual Interaction Evaluation

Table 15 presents the primary quantitative results for real-time interaction. LongCat-Flash-Omni achieves a mean naturalness score of 1.37 (95% CI [1.30, 1.44]) on the 0–3 scale, placing it third overall behind Doubao (1.92 [1.85, 1.98]) and GPT-4o (1.79 [1.72, 1.85]), but ahead of iFlytek Spark (1.25 [1.18, 1.32]), StepFun (1.22 [1.15, 1.28]), ChatGLM (0.99 [0.93, 1.05]), Qwen2.5-Omni (0.96 [0.89, 1.02]), and Qwen3-Omni (0.81 [0.75, 0.87]). The 0.56-point advantage over Qwen3-Omni (the current SOTA open-source model) is substantial on a 0–3 scale, translating to roughly one full category improvement (e.g., from "partially unnatural and affecting interaction" toward "partially unnatural but not affecting interaction"). However, the 0.55-point gap to GPT-4o and 0.42-point gap to Doubao (a ByteDance proprietary product evaluated during August 11–15, 2025) indicate that proprietary systems still deliver perceptibly more natural interaction experiences.

Table 16 provides qualitative dimensional analysis. LongCat-Flash-Omni achieves strong results on paralinguistic understanding (91.5% good cases) — the highest among all compared models, exceeding Doubao (88.0%), GPT-4o (87.5%), and Qwen3-Omni (89.0%). The paper highlights this qualitatively: "LongCat-Flash-Omni actively interprets user emotions from both facial expressions and vocal cues, demonstrating its superior paralinguistic understanding." This is a direct validation of the paper's architectural claim that processing raw audio continuously (rather than through an ASR pipeline) preserves paralinguistic information.

On memory capability (94.5% good cases), LongCat-Flash-Omni ranks second only to Doubao and GPT-4o (both 98.0%), and substantially ahead of Qwen3-Omni (36.0% — a 58.5-point gap). This demonstrates the effectiveness of the 128K context window and long-context multimodal training data (Section 3.1.7) for maintaining conversational coherence over extended interactions.

On relevance (54.5% good cases), LongCat-Flash-Omni outperforms all compared models except Doubao (66.5%) and GPT-4o (60.0%), with a 46.5-point advantage over Qwen3-Omni (8.0%). The paper attributes this to "strong comprehension capabilities by closely tracking dialogue topics and generating highly correlated responses."

However, the model shows clear weaknesses on real-timeness (49.5% good cases) — substantially below Doubao (65.5%), GPT-4o (71.5%), and iFlytek Spark (67.0%). The paper diagnoses this: "our model tends to be overly sensitive to user pauses, often initiating responses prematurely and interrupting users mid-conversation." This suggests the speculative prefill-decode switching strategy (Section 6.2) may trigger decoding too aggressively, before the VAD system has sufficient confidence that the user has finished speaking.

On human-likeness (62.5% good cases), LongCat-Flash-Omni trails Doubao (93.5%) and iFlytek Spark (92.5%) by large margins. The paper notes "occasional pronunciation errors, stuttering, and robotic or electronic audio artifacts" — suggesting the GAN-based audio decoder, while fast enough for streaming, does not yet match the naturalness of proprietary TTS systems that may use diffusion or flow-matching approaches (which the paper explicitly avoided for latency reasons, Section 2.2).

On accuracy (36.0% good cases), LongCat-Flash-Omni ranks near the bottom, only above ChatGLM (33.5%). The paper observes that "while our model shows strong capability in recognizing dynamic objects, its recognition performance declines when processing text and numerical information" and that it has "a tendency to over-agree with users' statements while overlooking relevant visual content." This accuracy weakness in real-time interaction contrasts with the model's strong offline benchmark performance — suggesting that the streaming setting (with its chunked, incomplete information) introduces challenges not captured by offline evaluation.

Ablation Studies and Robustness Checks

Audio Encoder Pre-Training Strategy (Section 3.2.6, text only): The paper reports investigating "the necessity of separate audio projector module pre-training for semantic space alignment, but empirical results showed negligible performance differences compared to direct end-to-end audio encoder training." This negative result — that a separate projector pre-training phase is unnecessary — is reported without a dedicated table but justifies the simpler direct training approach used in Stage-5.

SFT Freezing Strategy (Section 4.1.3): The paper reports that freezing the audio encoder while updating all other modules during SFT "stabilizes convergence and avoids catastrophic forgetting of low-level auditory features." No ablation comparing full fine-tuning vs. selective freezing is presented with quantitative results, so this claim is supported by the final model's performance but not by a controlled comparison.

Revision Model Verifier Choice (Appendix J, Figure 15): [Note: The paper references Figures 15a and 15b in Appendix J, but these figures are not included in the provided paper content. The following is based on the textual descriptions in Sections 6 and 7.] The paper states that "the PRM trained on base model outputs does not transfer well to the revision model's outputs due to distribution shift (Figure 15a)" and that "including revision history improves verifier performance slightly (Figure 15b)." Without the actual figures, the quantitative magnitude of these effects cannot be assessed. The paper reports that an ORM trained specifically on the revision model's outputs is used, and that the specific ORM training (with revision history in context) provides a small improvement over a no-history ablation.

DPO Loss Weight Ratio (Section 4.2.2): The joint text-speech DPO uses $\alpha:\beta = 1:1$ with a KL regularizer weight of 0.1. No ablation over different $\alpha:\beta$ ratios is reported, so the sensitivity of final performance to this choice is unknown.

Stage-1 Loss Weights (Section 3.2.2): The paper reports the loss weights a=1.75, b=0.25, c=1.5, d=0.1 as the result of "carefully hyperparameter tuning and extensive preliminary experiments," but no ablation table or learning curves for alternative weight configurations is provided. The reader cannot assess whether these specific values are critical (small perturbations cause degradation) or whether a broad range of weights would yield similar results.

Data Ratio Sensitivity (Section 3.2.3–3.2.5): The text:vision:speech ratio of 2:1:1 is maintained across Stages 2–4. No sensitivity analysis over different ratios (e.g., 1:1:1, 3:1:1) is reported. The paper does not establish whether this specific ratio is optimal or simply a reasonable choice that works.

PPL-Gap-Based Data Sampling in Stage-3 (Section 3.2.4): The paper describes this automatic data sampling strategy in detail but provides no direct ablation comparing fixed-mixture training vs. PPL-gap-guided training. The contribution of this mechanism to final performance is unquantified.

Modality-Decoupled Parallelism Throughput Claim (Section 5): The paper claims "over 90% of the throughput achieved by text-only training" in multimodal settings, but does not provide a detailed breakdown of throughput measurements under different configurations (e.g., with vs. without MDP, with different DP/CP/PP ratios). Figure 10 provides benchmark results for MoE GEMMs under different CP/EP configurations, demonstrating the efficiency rationale for reducing CP (fewer context-parallel ranks improves core operator efficiency), but no end-to-end throughput comparison is tabulated.

Memory Optimization Ablation (Table 4): The paper systematically walks through memory footprint reduction from 137 GB to 69.1 GB with each optimization enumerated. This is a clear and well-documented ablation, though the individual contribution of each optimization to throughput (not just memory) is not reported — some optimizations (like selective recomputation) may reduce throughput while saving memory, but this tradeoff is not quantified.

Audio Tokenizer Frame Rate Ablation: The paper uses 16.67 Hz (4 codebooks) but does not provide ablation over alternative frame rates or codebook counts. The relationship between compression rate and downstream task performance is unexplored.

Critical Assessment

Claim 1: "LongCat-Flash-Omni achieves state-of-the-art results on omni-modal benchmarks among open-source models."

This claim is well-supported for the specific benchmarks reported. On OmniBench, the model achieves 61.38 vs. Qwen3-Omni's 58.41 (+2.97 points). On WorldSense, 60.89 vs. 52.01 (+8.88). On DailyOmni, 82.38 vs. 69.33 (+13.05). On UNO-Bench, 49.90 vs. 42.10 (+7.80). However, three caveats apply. First, the set of open-source omni-modal competitors is small — essentially Qwen3-Omni and Qwen2.5-Omni are the only direct comparisons on cross-modality benchmarks (Table 14). The paper cites other efforts in Section 1 (M2-Omni, Baichuan-Audio, Ola, Vita-1.5) but does not evaluate against them, making the "state-of-the-art among open-source models" claim based on a limited comparison set. Second, OmniBench uses "an internally corrected version" because the public release "contains scoring deficiencies" — this means the 61.38 result is not directly comparable to numbers reported in other papers using the original OmniBench, which is a significant limitation for community benchmarking. Third, LongCat-Flash-Omni is substantially larger (560B total, 27B activated) than Qwen3-Omni (whose parameter count is not specified in the paper but is likely smaller based on the performance gap). The claim of SOTA performance among open-source models is thus supported, but the comparison set is narrow and the scale disparity is substantial.

Claim 2: "Real-time audio-visual interaction capability with low latency."

The interaction evaluation in Table 15 shows LongCat-Flash-Omni ranking third behind two proprietary products (Doubao and GPT-4o) but ahead of all other open-source and several proprietary systems. The 1.37 mean score (on a 0–3 scale) is above the midpoint, indicating interactions are "partially unnatural but not affecting interaction" on average. The latency claim of "~100ms first packet latency after VAD endpoint detection" is reported but not independently verified — no latency benchmarks against other models are provided, and the 100ms figure is from the paper's own deployment rather than a standardized latency measurement protocol. The real-timeness dimension in Table 16 (49.5% good cases) indicates that users perceive latency/ interruption issues in roughly half of interactions, which is substantially worse than Doubao (65.5%), GPT-4o (71.5%), and iFlytek Spark (67.0%). The claim of "low-latency" interaction is thus conditionally supported: the system achieves low server-side latency (~100ms), but user-perceived real-timeness lags behind leading proprietary systems, likely due to premature response triggering rather than raw computation speed.

Claim 3: "Training infrastructure sustains over 90% of text-only training throughput during multimodal training."

This claim is stated in the abstract and Section 5 but is supported primarily by assertion rather than detailed experimental evidence. Table 2 and Figure 7 document the computational heterogeneity that motivates the MDP design. Table 4 documents memory optimizations. But no end-to-end throughput measurement comparing text-only vs. multimodal training with and without MDP is presented. The 90% figure is a headline number without a supporting table showing tokens-per-second or FLOPs utilization under different configurations. This is a significant gap — the infrastructure contribution would be substantially stronger with a controlled throughput ablation demonstrating the contribution of MDP, ModalityBridge chunking, and communication optimizations individually.

Claim 4: "Progressive training preserves unimodal text capabilities."

Tables 12 and 13 provide the evidence: LongCat-Flash-Omni matches or exceeds LongCat-Flash (its text-only foundation) on most text benchmarks. However, the claim requires careful qualification. Table 13 shows regression on instruction-following benchmarks: IFEval drops from 89.65 to 82.44 (−7.21 points), COLLIE drops from 57.10 to 45.69 (−11.41 points), and Meeseeks-zh drops from 43.03 to 39.05 (−3.98 points). These are non-trivial degradations, and they affect precisely the capability most important for a conversational assistant — following user instructions precisely. The paper does not ablate whether this regression is caused by the multimodal training data, the SFT data mixture, or the DPO stage. The claim of "no degradation in any single modality" (stated in the abstract and Section 7.3) is therefore overstated: text reasoning and knowledge capabilities are preserved (and sometimes improved), but instruction-following capabilities measurably degrade. A more accurate characterization would be: "text knowledge and reasoning are preserved or improved, while structured instruction-following shows moderate regression."

Claim 5: "Cross-modal heterogeneity is addressed through early-fusion training."

The paper's approach of interleaving audio, visual, and text tokens within the same LLM input sequence (chunk-wise interleaving with timestamps) is a form of early fusion — all modalities are processed by the same transformer layers from the earliest stages of multimodal training. The evidence that this works is the cross-modality benchmark performance (Table 14) and the qualitative finding of strong paralinguistic understanding (91.5% in Table 16). However, the paper does not ablate early fusion vs. late fusion (e.g., processing modalities in separate encoders and combining only at the final layers). Without this comparison, it is impossible to attribute the cross-modal reasoning performance specifically to early fusion rather than to scale, data quality, or other factors. The claim is plausible and consistent with the results, but not experimentally isolated.

Additional Weaknesses:

The interaction evaluation (Section 7.4.2) is a major strength of the paper but has methodological limitations. The evaluation is conducted against models at specific dates (August–September 2025), and API-based models may have been updated since. The 10 conversationalists and 200 sessions per model represent a substantial effort, but the session count is modest for statistical power on the qualitative dimensions (Table 16, where per-dimension percentages represent expert judgments on a subset of the 200 sessions). The evaluation categories (problem solving, entertainment, self-improvement, emotional support) may not be representative of all real-world interaction scenarios.
No statistical significance testing is reported for any benchmark results. For benchmarks with small test sets (e.g., some audio benchmarks may have only hundreds of examples), differences of a few percentage points may not be statistically significant. Confidence intervals are only reported for the interaction evaluation (Table 15).
The paper does not evaluate LongCat-Flash-Omni against GPT-4o or Seed-1.6 on cross-modal or audio benchmarks because "they don't have open ones for audio" (Section 7.4). This means the cross-modality comparison is limited to Gemini models and Qwen models, missing the most prominent omni-modal proprietary system (GPT-4o) on the benchmarks where omni-modal capability is the primary evaluation target.
The PPL-gap-based data sampling strategy (Stage-3) is described as a significant innovation but receives no quantitative ablation. Without evidence that it improves over uniform sampling or static mixture weights, its contribution to final performance is unknown.
No training data contamination analysis is reported. Given that the model is trained on "over 2.5 trillion tokens" of web-sourced multimodal data, the possibility of benchmark leakage (training data containing evaluation examples) is non-trivial. The paper does not describe any decontamination procedures.
The audio decoder quality is a limiting factor for interaction naturalness. The paper explicitly notes "robotic or electronic audio artifacts" (Section 7.4.2) as a weakness. The choice of a GAN-based decoder over diffusion/flow-matching approaches trades quality for latency, but the paper does not quantify this tradeoff — what audio quality metric (e.g., MOS) would a diffusion-based decoder achieve, and what would the latency cost be?

In summary, the experimental evaluation is comprehensive in breadth — covering the full spectrum from text to vision to audio to cross-modal to real-time interaction — and establishes LongCat-Flash-Omni as the strongest open-source omni-modal model at the time of reporting. However, the evaluation has meaningful gaps: missing ablations for key design choices (data ratios, loss weights, early vs. late fusion), missing throughput benchmarks to support the infrastructure efficiency claim, limited open-source competitor comparisons on cross-modal benchmarks, and no statistical significance testing. The real-time interaction evaluation, while methodologically ambitious, reveals specific weaknesses (real-timeness, human-likeness, accuracy) that temper the "low-latency, high-quality" characterization in the paper's framing. The preservation of text capabilities is demonstrated for knowledge and reasoning but not for instruction-following, which the paper's own results show measurably degrades.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For in the Headline 4× Efficiency Claim

The paper's compute-optimal test-time scaling strategy — which routes easy questions to best-of-N and medium-hard questions to beam search, achieving up to 4× efficiency gains over a uniform best-of-N baseline — depends on knowing each prompt's difficulty before allocating the inference budget. The method for estimating difficulty, however, is extraordinarily expensive: it requires generating 2,048 complete solutions per question and scoring them with the process reward model (or checking against ground truth, in the oracle case). The authors are transparent about this:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence is that the reported 4× efficiency improvement (e.g., 16 generations matching the accuracy of 64 uniform generations in Figure 4) is computed after difficulty is already known, excluding the cost of acquiring that knowledge. In a realistic deployment, the total compute budget would be: (difficulty estimation cost) + (strategy execution cost). Since difficulty estimation consumes 2,048 samples per question — 8× to 128× more than the test-time budgets studied (16–256 generations) — the amortized cost would dominate the total, potentially erasing or even reversing the reported gains. For a batch of N questions, paying 2,048N samples up front to save a few hundred samples per question on strategy execution is net-negative unless N is very large and the difficulty estimates can be reused across many similar questions — a scenario the paper does not explore.

The evidence for this limitation is not in an ablation but in the method itself: Section 3.2 describes the 2,048-sample estimation procedure. The paper attempts a partial mitigation by using the PRM's predicted scores rather than ground-truth labels (the "predicted difficulty" bins in Figures 4 and 8), which removes the need for oracle answers but does not reduce the sample cost. The authors acknowledge this as "a key avenue for future work" (Section 3.2), specifically mentioning "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), but no such model is developed or evaluated. The mitigation is aspirational, not implemented.

A practitioner considering this method must therefore decide whether the compute-optimal policy can be estimated offline for their problem distribution (amortizing the 2,048-sample cost across many future queries), or whether an online difficulty estimation scheme with far fewer samples (e.g., 4–8 initial samples followed by adaptive reallocation) can approximate the oracle policy. The paper provides no guidance on either.

The Hardest Problems Are Fundamentally Outside the Method's Reach

The paper demonstrates that test-time compute scaling produces large gains on easy and medium problems but essentially zero gain on the hardest problems (difficulty bin 5, the bottom quintile by the base model's pass@1 rate). This is not a minor degradation — it is a hard capability ceiling. The evidence is consistent across both search-based and revision-based approaches:

In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods (best-of-N, beam search, lookahead search) at all budgets from 4 to 256 generations.
In Figure 7 (right), bin 5 accuracy is roughly 2–3% irrespective of the sequential-to-parallel ratio at 128 generations.
In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is flat near 0–5%, and the 14× larger pretrained model consistently outperforms test-time compute on these problems — by margins up to 52.9% relative disadvantage for PRM search at R ≫ 1 (Section 7).

The cause is structural, not incidental: test-time compute can only select or refine solutions that exist in the model's proposal distribution. If the base model's pass@1 is near zero — it almost never produces a correct solution, even by chance — then no amount of search, revision, or verifier optimization can find a correct solution. The paper is explicit about this: the FLOPs-matched comparison in Section 7 states in its takeaway box that "for problems fundamentally outside the base model's capabilities, additional test-time compute is not a substitute for larger-scale pretraining."

What this means for a practitioner is that compute-optimal test-time scaling cannot expand the set of solvable problems — it can only improve efficiency on problems the base model already sometimes solves. For any deployment where the problem distribution includes a non-trivial fraction of genuinely novel or out-of-distribution reasoning tasks (difficulty bin 5), the method provides no benefit, and the only path forward is to train a larger or more capable base model. The paper provides no mitigation for this limitation — it is presented as a boundary condition on the method's applicability, not a solvable issue.

The FLOPs-Matched Comparison Uses a Suboptimal Pretraining Baseline

The paper's central claim about the pretraining-vs-inference tradeoff — that a smaller model with compute-optimal test-time strategies can outperform a ~14× larger model — depends on a specific experimental choice: the larger model is trained by scaling parameters only, holding data fixed. The authors acknowledge this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence is that the pretraining baseline is weaker than it could be. Under Chinchilla-optimal scaling (Hoffmann et al., 2022), a model trained with 14× more total FLOPs would scale both parameters and data, yielding better performance than a parameters-only-scaled model at the same FLOP budget. The paper's comparison therefore overstates the advantage of test-time compute relative to a properly compute-optimal pretraining investment. A practitioner deciding how to allocate a total FLOPs budget between pretraining and inference cannot use these results directly without knowing how much the advantage would shrink against a Chinchilla-optimal larger model.

Additionally, the 14× larger model uses only greedy decoding — no best-of-N, no majority voting, no search. The paper does not evaluate the larger model with even a modest test-time compute budget (e.g., best-of-8), which would be the fair comparison: both models get the same total FLOPs, but the larger model might spend some of its inference budget on parallel sampling. The current comparison gives the larger model zero test-time compute, making it a straw-man baseline for the inference component of the FLOPs-matched tradeoff.

The evidence for this limitation is in the paper's own description of the experimental setup (Section 7). The mitigation status is that the paper explicitly acknowledges the departure from compute-optimal pretraining and frames future work on jointly optimizing pretraining and inference compute. The acknowledgment is transparent, but it does not change the fact that the headline result — "smaller model + test-time compute beats 14× larger model" — is evaluated against a suboptimal larger model.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with Only Partial Mitigation

The paper reports a significant practical issue with the sequential revision approach: approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent revision step (Section 6.1). This is a direct consequence of the training data construction: the model was trained only on sequences where all in-context answers are incorrect, followed by a correct target — "since the model was trained only on sequences where all in-context answers are incorrect (followed by a correct target), at test time the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers."

The consequence is that longer revision chains are not monotonically beneficial — after some number of steps, the model may degrade previously correct answers. This forces the use of a selection mechanism (majority voting or verifier-based selection) that scans the entire revision chain and picks the best answer, rather than simply taking the final revision. While this mitigates the problem, it is an imperfect patch: the selection mechanism must be reliable enough to identify the correct answer among a chain that may contain both correct and incorrect revisions, and it introduces additional computation (scoring every step in the chain).

The paper demonstrates the mitigation's effectiveness — sequential revisions with verifier-based selection outperform parallel sampling in Figure 6 (right) — but the underlying issue is not solved. A more principled approach would train the model to recognize when no revision is needed (i.e., include trajectories where the correct answer remains unchanged), but this is not investigated. The ReST^(EM) experiment (Appendix K, Figure 16) shows that attempting to further optimize the revision model with RL-style training actually worsens the reversion problem — "fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio" — indicating that the revision training is sensitive to data distribution in ways that are not fully understood.

For a practitioner, this means the revision model cannot be used as a simple "generate, then refine" pipeline where the last output is trusted. It requires a separate verifier and a chain-scanning selection procedure, adding complexity and latency. And the fragility revealed by the ReST^(EM) experiment suggests that further improving the revision model (e.g., with more data, better data construction, or RL fine-tuning) may not straightforwardly improve performance.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The paper's entire experimental analysis — every difficulty-dependent scaling curve, every FLOPs-matched comparison, every revision model result — is conducted on the MATH benchmark using PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this assumption is untested.

The consequence is that the paper's specific findings may not generalize to other domains, model families, or task types. Several aspects of the results could be model-dependent:

The PRM's quality and over-optimization threshold depend on PaLM 2-S*'s output distribution and error patterns. A model with different calibration (e.g., better-calibrated probabilities, different reasoning patterns) might exhibit different difficulty-dependent scaling curves — perhaps beam search does not over-optimize on easy problems for a better-calibrated model, or perhaps the over-optimization threshold shifts to higher budgets.
The revision model's ability to learn from in-context incorrect examples depends on the base model's in-context learning and self-correction capabilities. Different model families (e.g., models with stronger or weaker in-context learning) might show different optimal sequential-to-parallel ratios.
The MATH benchmark consists of competition-level math problems requiring symbolic reasoning and algebraic manipulation. The finding that revisions help on easy problems and search helps on medium problems may not transfer to domains requiring factual recall (where "reasoning" is retrieval rather than derivation), code generation (where unit tests provide a different verifier signal), or open-ended generation (where correctness is ambiguous).

The paper does not evaluate on any other benchmark (e.g., GSM8K, MMLU, HumanEval), nor does it use any other model family as the base. The 500-question MATH test set, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin — a small sample on which strategy selection could be noisy.

The mitigation status is that the paper does not claim broader generalization — the abstract and introduction are specific about the MATH/PaLM 2-S* setting. But a practitioner considering adopting compute-optimal test-time scaling for a different domain or model family has no direct evidence that the difficulty-dependent patterns (beam search hurts easy problems, revisions help easy problems, search helps medium problems) will replicate. The paper's framework for analyzing these patterns is generalizable, but the specific strategy recommendations are not validated beyond the studied setting.

Latency and Wall-Clock Constraints Are Not Considered in the Compute-Optimal Framework

The paper measures test-time compute in "generations" (number of complete solutions sampled) and treats all generations as fungible units of cost. This is a reasonable proxy for total FLOPs but ignores a dimension that matters enormously in practice: latency. Sequential revisions are inherently serial — each revision step depends on the previous one — while parallel best-of-N sampling can be executed simultaneously given sufficient hardware.

The consequence is that the compute-optimal policy recommended by the paper may be latency-suboptimal in deployment. For example, on easy questions, the paper finds that purely sequential revisions (one long chain) are optimal in terms of generation efficiency (Figure 7, right). But a strategy that allocates 64 generations as a chain of 64 sequential revisions takes roughly 64× longer wall-clock time than 64 parallel samples — a 6.4-second delay vs. a 100ms delay, assuming ~100ms per generation. For interactive applications where sub-second response times are required, the sequential-heavy strategies favored by the compute-optimal policy may be unusable regardless of their accuracy advantages.

The paper does not discuss this tradeoff anywhere. The compute-optimal objective (Equation 1 in Section 3.1) optimizes only for accuracy given a generation budget N, with no constraint on wall-clock time or latency. This is not an oversight — it is a deliberate scoping choice to study the FLOPs-accuracy tradeoff — but it means a practitioner deploying this method in a latency-sensitive setting (chatbots, real-time assistants, interactive tutoring) cannot directly apply the recommended strategies. They would need to add a latency constraint, which might shift the optimal policy toward more parallel sampling, especially on easy problems where the paper's sequential-heavy recommendations are most aggressive.

The paper provides no latency measurements, no generation-time estimates for different strategies, and no analysis of how latency constraints would modify the compute-optimal policy. This is an unaddressed dimension of the method's practical applicability.

7. Implications and Future Directions

How This Work Changes the Landscape

LongCat-Flash-Omni represents a capabilities demonstration rather than a paradigm shift — it does not introduce fundamentally new architectures or training algorithms, but it establishes a new performance frontier for open-source omni-modal models and, more importantly, provides an engineering blueprint for how to integrate four modalities at 560B scale without sacrificing any single one. The paper's impact on the field operates at three levels: empirical (it proves what is possible), methodological (it documents how to do it), and infrastructural (it demonstrates that the computational cost of multimodal training need not be dramatically higher than text-only training).

The primary empirical shift is proving that open-source models can compete with proprietary omni-modal systems on real-time interaction, not just offline benchmarks. Prior to this report, the narrative in both research and product communities was that end-to-end speech-and-vision interaction — the kind demonstrated by GPT-4o and Gemini 2.5 — required proprietary infrastructure, massive undisclosed training budgets, and closed-source audio components. LongCat-Flash-Omni challenges this by achieving a 1.37 mean naturalness score on real-time audio-visual interaction (Table 15, Section 7.4.2), placing it ahead of several proprietary products (iFlytek Spark at 1.25, StepFun at 1.22) and dramatically ahead of the previous best open-source model (Qwen3-Omni at 0.81). The 0.56-point gap to Qwen3-Omni on a 0–3 scale represents a qualitative difference — roughly one full category improvement in perceived naturalness. This does not mean open-source has caught proprietary systems (Doubao at 1.92 and GPT-4o at 1.79 remain clearly ahead), but it demonstrates that the gap is bridgeable with publicly documented methods.

The methodological shift is the reframing of multimodal training as a curriculum design problem rather than an architecture design problem. The paper's central insight — that the order of modality introduction (text → speech → images → video → continuous audio) is a first-order determinant of final performance — challenges the prevailing assumption that modalities can be mixed arbitrarily and that the primary challenge is encoder design. The evidence is not a single ablation but the consistent pattern across Tables 12 and 13: text capabilities are preserved or improved (MMLU increases from 87.05 to 90.30, HumanEval+ from 65.85 to 90.85) while simultaneously achieving SOTA omni-modal performance. This is difficult to achieve through architectural innovation alone — it requires a training strategy that respects the structural differences between modalities. The paper's explicit reasoning about why speech comes before vision (sequential structure matches text) and why video is deferred to the annealing stage (spatial + temporal complexity compounds) provides a transferable principle: modalities should be ordered by their structural similarity to the foundation modality, with more complex combinations introduced only after simpler ones are mastered.

This reframing reconciles a tension in the multimodal literature. Prior work showed both that multimodal training could work (GPT-4o, Gemini) and that it often degraded text capabilities in open-source efforts (many vision-language models underperform their text-only backbones on reasoning benchmarks). The contradictory findings reflected differences not primarily in architecture but in training recipe — specifically, whether modalities were introduced gradually with careful data balancing or mixed together from the start. LongCat-Flash-Omni provides a documented, reproducible instance of the former approach succeeding, shifting the research question from "can we prevent multimodal training from degrading text?" to "what curriculum and data mixture optimally preserves each modality while enabling cross-modal synergy?"

The infrastructural shift is the demonstration that modality-decoupled parallelism can sustain near-text-only throughput during multimodal training. The paper's claim of "over 90% of the throughput achieved by text-only training" (Section 5) — if taken at face value — substantially lowers the barrier to entry for large-scale omni-modal training. The heterogeneity documented in Table 2 (a 4,000× range in per-microbatch FLOPs across components) is a genuine obstacle that has made 500B+ parameter multimodal training prohibitively expensive for most research groups. By showing that a decoupled parallelism design with a dedicated bridge module can handle this heterogeneity with only ~10% overhead, the paper makes large-scale omni-modal training a realistic goal for well-resourced open-source teams rather than an exclusive capability of a few proprietary labs. The detailed memory optimization walkthrough in Table 4 (137 GB → 69.1 GB) and the chunk-based ModalityBridge design provide an implementation manual that other teams can adapt.

A less prominent but significant shift is the elevation of real-time interaction evaluation to a first-class research contribution. The paper does not treat streaming interaction as an engineering afterthought but devotes substantial space to its design (Section 2.4.2, Section 6.2) and evaluates it rigorously with 250 human annotators across 200 dialogue sessions per model (Section 7.4.2). This establishes a template for what comprehensive omni-modal evaluation should include — not just offline benchmarks (OmniBench, WorldSense) but also subjective interaction quality, dimensional analysis (Table 16's six factors), and direct comparison against deployed products. The finding that interaction quality does not perfectly correlate with benchmark performance — LongCat-Flash-Omni scores well on offline cross-modal benchmarks but shows clear weaknesses in real-timeness (49.5% good cases) and human-likeness (62.5%) during live interaction — is an important caution against over-relying on static benchmarks for omni-modal evaluation.

Research directions that become more attractive include: curriculum design for multimodal training (the paper establishes that ordering matters but does not exhaustively explore alternative orderings), cheap difficulty estimation for test-time adaptive strategies (the paper's omitted cost is the bottleneck to practical deployment), robust streaming interaction (the real-timeness and human-likeness gaps in Table 16 are clear targets), and open-source audio interaction quality (the GAN-based decoder's artifacts are a measurable weakness). Directions that become less central include: trying to beat proprietary models purely through architectural innovation (the paper's results suggest training recipe and scale matter more than novel architectures per se), and treating modalities as independent problems to be solved separately and stitched together (the paper's early-fusion approach with joint optimization outperforms pipeline approaches on cross-modal benchmarks).

Follow-Up Research This Work Enables

1. Ablation of modality introduction order to establish when the curriculum matters vs. when it is incidental. The paper argues that the text → speech → images → video ordering is principled, but provides no ablation comparing alternative orderings (e.g., text → images → speech → video, or all modalities introduced simultaneously in Stage-2). A strong follow-up would train multiple 10B-parameter-scale variants of the same architecture with different modality introduction orders, measuring: (a) final omni-modal benchmark performance (OmniBench, WorldSense, DailyOmni), (b) unimodal text preservation (MMLU, MATH, HumanEval+), and (c) training dynamics (does convergence speed or stability depend on ordering?). The paper's own results show that text capabilities improve in some cases after multimodal training — testing whether this positive transfer depends on introducing speech before vision (since speech shares sequential structure with text, potentially reinforcing sequence modeling) vs. vision before speech (where spatial reasoning might interfere with the sequential structure) would clarify whether the curriculum principle is causal or correlational.

2. Training a cheap difficulty predictor to close the compute-optimal test-time scaling deployment gap. The paper's compute-optimal framework from the companion work depends on estimating question difficulty by generating 2,048 samples per prompt — a cost that dominates the actual problem-solving budget. LongCat-Flash-Omni's multimodal training provides a natural foundation for a difficulty predictor: the model has learned rich representations of question complexity across modalities, and its internal activations (or a lightweight probe trained on top) might encode difficulty signals that could be extracted from a single forward pass. A concrete experiment: take the 500-question MATH test set used in the companion paper, record the LLM's intermediate representations at a middle layer for each question, train a linear classifier (or small MLP) to predict the oracle difficulty quintile from these representations, and measure whether a difficulty predictor trained on 10% of the questions can achieve bin-classification accuracy sufficient to replicate the compute-optimal policy's 4× efficiency gain. This would transform the compute-optimal framework from an analytical contribution to a practically deployable system.

3. Streaming interaction quality improvement through targeted architectural changes informed by the dimensional analysis in Table 16. The paper identifies three specific weaknesses in real-time interaction: real-timeness (49.5% good cases, caused by "overly sensitive" turn-taking that interrupts users), human-likeness (62.5%, caused by "pronunciation errors, stuttering, and robotic audio artifacts"), and accuracy (36.0%, caused by poor text/numerical recognition in streaming contexts). Each suggests a concrete follow-up: (a) For real-timeness: train the VAD system's endpoint detection threshold specifically on the model's interaction data, using the human annotators' judgments of premature interruption as labels to find the optimal silence duration before triggering response — this is a supervised learning problem on a binary label (interruption was premature vs. appropriate) that the paper's 200-session evaluation dataset could directly support. (b) For human-likeness: ablate the GAN-based audio decoder against a diffusion-based alternative, measuring both audio naturalness (via crowdsourced MOS) and end-to-end latency (via wall-clock measurements), to quantify the quality-latency tradeoff the paper alludes to but does not measure. The paper's streaming pipeline (Section 6.2) reports ~100ms first-packet latency — a diffusion decoder adding, say, 200ms of latency while improving MOS by 0.5 points would represent a clear design tradeoff. (c) For accuracy: evaluate whether the streaming setting's accuracy degradation is caused by the chunked processing (incomplete visual/audio information at decision time) or by the model's inherent difficulty with text recognition under time pressure, by comparing offline image recognition accuracy on the same visual content against streaming accuracy — if the gap is large, architectural changes to the chunk-wise interleaving (e.g., allowing cross-chunk attention for visual tokens) might recover accuracy without sacrificing latency.

4. Joint optimization of pretraining and inference compute allocation for omni-modal models. The companion paper's FLOPs-matched comparison studies the pretraining-vs-inference tradeoff for math reasoning with a text-only model. LongCat-Flash-Omni extends the modality space to four modalities, each with different computational characteristics at both training and inference time (Table 2 documents the 4,000× FLOPs range across components). A natural follow-up asks: given a fixed total FLOPs budget for an omni-modal model, what is the optimal allocation between (a) scaling the vision encoder vs. the audio encoder vs. the LLM during pretraining, and (b) allocating test-time compute across modalities during inference? A concrete experiment: train three variants of a scaled-down LongCat-Flash-Omni (e.g., 70B parameters) with different pretraining FLOPs allocations (vision-heavy, audio-heavy, balanced), then for each variant, sweep test-time strategies (e.g., allocating more frames to video processing vs. more beam search budget to audio understanding) on cross-modal benchmarks like WorldSense and DailyOmni. The result would be a 2D efficiency map — pretraining allocation × inference allocation — showing whether the optimal strategy is modality-symmetric (invest equally in all modalities at both training and inference) or modality-asymmetric (overinvest in one modality during training and compensate for the other at inference).

5. Stress-testing the progressive curriculum on out-of-distribution cross-modal tasks. The paper demonstrates strong cross-modal understanding on benchmarks where audio and visual streams are temporally aligned (WorldSense, DailyOmni). But real-world omni-modal interaction includes scenarios where modalities are misaligned, contradictory, or only partially available — e.g., a video call where the audio cuts out, or a cooking tutorial where the visual demonstration contradicts the spoken instructions, or a scene where the model must identify that a sound does not match the visual source. A stress-test would construct a "cross-modal conflict" benchmark with three categories: (a) temporal misalignment (audio events shifted relative to visual events), (b) semantic contradiction (visual and audio information convey conflicting signals — a barking sound paired with a cat video), and (c) modality dropout (one modality missing for portions of the interaction). Testing LongCat-Flash-Omni on this benchmark would reveal whether the progressive curriculum's step-by-step modality addition produces robust cross-modal representations or brittle ones that depend on perfect temporal alignment. A negative result (the model handles temporal misalignment well but fails on semantic contradiction) would suggest that the curriculum successfully teaches temporal binding but not semantic cross-modal verification — a finding that would motivate additional training stages specifically targeting cross-modal consistency checking.

6. Quantifying and mitigating the instruction-following regression observed in Table 13. The paper's own results show a notable degradation on instruction-following benchmarks: IFEval drops from 89.65 (LongCat-Flash, text-only) to 82.44 (LongCat-Flash-Omni), COLLIE from 57.10 to 45.69, and Meeseeks-zh from 43.03 to 39.05 (Table 13). These are non-trivial regressions (7.21, 11.41, and 3.98 points respectively) that affect the model's reliability as a conversational assistant. The paper does not ablate the cause — is it the multimodal pretraining data, the SFT data mixture (which includes audio interaction data that may have different response patterns), or the DPO stage (where the joint text-speech objective may have diluted the text preference signal)? A diagnostic follow-up would: (a) evaluate instruction-following benchmarks at each training stage (after Stage-1, Stage-2, etc.) to pinpoint when the regression occurs, (b) train an ablation model with multimodal pretraining but text-only SFT to isolate whether the regression comes from pretraining or post-training, and (c) experiment with a two-stage SFT recipe where the model first undergoes text-only instruction tuning to recover instruction-following, then undergoes multimodal SFT with a lower learning rate to add multimodal capabilities without disturbing the recovered text alignment. The paper's SFT recipe (Section 4.1.3) already freezes the audio encoder while updating all other modules — extending this to freeze the LLM's later layers during multimodal SFT (to protect text instruction-following circuits) would be a direct architectural intervention motivated by this diagnostic.

Practical Applications and Downstream Use Cases

1. Open-source real-time multimodal assistants for latency-sensitive domains. LongCat-Flash-Omni's end-to-end architecture — processing raw audio and video directly, reasoning jointly, and generating speech tokens natively — eliminates the cascade of ASR → LLM → TTS models that introduces cumulative latency and information loss at each stage boundary. The paper reports ~100ms first-packet latency after VAD endpoint detection (Section 6.2), which is in the range of natural human conversation turn-taking (typically 200–500ms). For applications where this latency matters — real-time language tutoring (the model hears the student's pronunciation, sees their facial expressions, and provides immediate spoken feedback), accessibility tools for visually impaired users (continuous audio description of visual scenes with natural interaction), or hands-free industrial inspection (a technician wearing a camera and microphone asking "what's this component?" while pointing) — the unified architecture provides a qualitative advantage over pipelined systems. The 1.37 interaction score on the 0–3 naturalness scale, while not matching proprietary leaders, is sufficient for task-oriented interaction where utility matters more than human-likeness. The open-source availability means these applications can be deployed on-premise or on private cloud infrastructure without depending on proprietary APIs.

2. Video understanding at scale for archival and educational content. The video understanding results in Table 6 — particularly 78.2 on VideoMME with audio (best among omni-modal models) and 86.2 on NextQA — position LongCat-Flash-Omni for large-scale video indexing and question-answering on institutional video archives. Universities with decades of recorded lectures, corporations with internal training video libraries, or media companies with broadcast archives could deploy the model for tasks like "find all moments in this semester's lectures where the professor discusses backpropagation and draws a diagram on the board" — a query requiring joint understanding of spoken content (audio), visual content (the board diagram), and temporal structure (when in the lecture). The model's 128K context window and the long-context multimodal training data (Section 3.1.7) are specifically designed for this type of extended-sequence reasoning. The hierarchical video token compression (Section 2.4.1) — reducing N frames to N/2 temporally before the ViT, then further downsampling visual tokens — makes processing hour-long videos computationally feasible despite the 560B parameter scale.

3. Multimodal data generation for model distillation and specialized task training. The paper's training pipeline produces a model that can generate high-quality multimodal outputs (text and speech) conditioned on multimodal inputs (audio, video, images, text). This is a powerful data generation engine for downstream tasks that lack large-scale training data. For example: (a) Generating audio-visual question-answering datasets for specialized domains (medical training videos, mechanical repair tutorials) by having LongCat-Flash-Omni watch domain videos and generate QA pairs, which can then be used to fine-tune smaller, task-specific models. The paper's own audio-visual interaction data pipeline (Section 4.1.2) uses a similar model-driven automation approach but with external multimodal LLMs — using LongCat-Flash-Omni itself for this data generation could produce higher-quality, more consistent training data since the generator and the target model share the same representational space. (b) Speech-to-speech translation data: the model's native multilingual speech capabilities (demonstrated by ASR results on FLEURS and CommonVoice15 in Table 9) could generate parallel speech corpora for low-resource language pairs. (c) Embodied AI simulation: the model's grounding capabilities (93.9 on RefCOCO-avg, 91.2 on ScreenSpot-v2 in Table 5) could generate instruction-following trajectories for robotics or GUI agents by watching demonstration videos and producing natural language action descriptions. The 27B average activated parameters means inference for data generation is relatively efficient despite the 560B total parameter count.

4. Real-time audio-visual monitoring with natural language querying. The model's streaming architecture — continuous chunk-wise processing of synchronized audio and video with textual timestamps (Section 2.4.2) — enables a deployment scenario where a model continuously monitors a live feed (security camera, baby monitor, factory floor camera) and responds to natural language queries about recent events. "Has anyone entered the back door in the last 5 minutes?" requires the model to have processed and retained the visual and auditory information from the recent past, including the temporal correspondence between audio (door sounds) and video (person appearing). The sparse-dense sampling strategy (dense 2 FPS during user input, sparse 0.5 FPS during monitoring) makes continuous operation feasible by reducing computation during idle periods. The model's memory capability (94.5% good cases in Table 16, second only to Doubao and GPT-4o) is directly relevant — it can recall events from earlier in the monitoring session in response to queries. The specific weakness identified in the paper — declining accuracy "when processing text and numerical information" in streaming contexts (Section 7.4.2) — would need to be addressed for applications requiring precise license plate reading or timestamp logging, but general event detection and description are well within the demonstrated capabilities.