LongCat-Next: Lexicalizing Modalities as Discrete Tokens

🎯 Pitch

For the first time, a purely discrete vision tokenizer can match or beat continuous visual encoders on understanding benchmarks—shattering the long-held belief that discrete tokens are inherently inferior for visual comprehension. The same model simultaneously rivals top-tier image generators and speech systems, reconciling the fundamental conflict between understanding and generation inside a single autoregressive framework.

1. Executive Summary

This paper introduces Discrete Native Autoregression (DiNA), a unified paradigm that extends next-token prediction across text, vision, and audio by representing all modalities within a shared discrete token space, and instantiates it in LongCat-Next — a native multimodal model built on the LongCat-Flash MoE backbone — to demonstrate that discrete visual modeling can break its historical performance ceiling on understanding tasks while simultaneously reconciling the traditionally competing objectives of understanding and generation. The core visual mechanism is the Discrete Native-Resolution Vision Transformer (dNaViT), which pairs a Semantic-and-Aligned Encoder with Residual Vector Quantization to produce hierarchical discrete tokens that preserve both high-level semantics and fine-grained details (supporting arbitrary-resolution tokenization at up to 28× compression), while the audio pipeline employs an RVQ-based tokenizer coupled with a Whisper encoder and flow-matching decoder operating at 12.5 Hz. LongCat-Next achieves competitive or superior performance against specialized models across understanding and generation benchmarks — matching Qwen3-VL-A3B on visual comprehension, outperforming Flux-dev on GenEval (84.44 vs. 66.00) and text rendering, and surpassing both omni and speech-specialized models like Gemini 2.5 Flash-Lite and MiMo-Audio on audio tasks — establishing that a natively discrete, modality-agnostic architecture can serve as an industrial-strength unified foundation, though the approach depends critically on semantic completeness in the tokenizer and does not yet extend beyond the canonical understanding-and-generation tasks to any-to-any interleaved multimodal reasoning.

2. Context and Motivation

The Core Problem: We Fragment Modalities Instead of Unifying Them

The fundamental problem this paper addresses is architectural fragmentation in multimodal AI. Contemporary large language models have converged on a remarkably clean paradigm: represent everything as discrete tokens and predict the next one. This Next-Token Prediction (NTP) formulation has scaled beautifully for text, producing models that unify comprehension and generation within a single autoregressive objective. But when the field extends these language models to handle vision, audio, or other modalities, the clean paradigm breaks down. Instead of treating these signals as first-class token sequences, most systems bolt on modality-specific encoders that project continuous representations into the language model's embedding space — creating what the authors call a "language-plus-auxiliary" architecture (Section 1).

This fragmentation matters because it creates a self-reinforcing cycle: because vision and audio are treated as external attachments, they cannot fully benefit from the training infrastructure, scaling insights, and optimization techniques developed for discrete token modeling. The paper frames this as the fundamental question: "how can non-linguistic modalities be effectively represented within a discrete token space?" (Section 2). The entire paper is an argument that this question is not just theoretically interesting but practically solvable at industrial scale.

The gap is precisely this: we lack a training recipe that treats multimodality as a native linguistic extension, where vision tokens and audio tokens are as fundamental to the model as subword tokens, sharing the same embedding space, the same autoregressive objective, and the same optimization pipeline. The paper positions itself as filling this gap not with a conceptual demonstration but with an "industrial-strength" recipe that satisfies three concrete criteria (Section 2): performance parity with specialized models (not just "also works"), modality synergy where adding vision/audio doesn't degrade text capabilities, and infrastructure friendliness where the transition from pure language to native multimodal requires minimal architectural surgery.

Why This Problem Is Important

The significance operates on multiple levels, spanning practical deployment economics, theoretical understanding of representation learning, and the long-term trajectory of AI systems.

Practical deployment economics. If vision and audio can be fully discretized and processed through the same decoder-only backbone as text, the entire ecosystem built around LLMs — training infrastructure, inference optimization, quantization techniques, serving frameworks — becomes immediately reusable for multimodal applications. This is not a small efficiency gain. The current approach of maintaining separate vision encoders with different architectures, different attention patterns (sometimes bidirectional while the LLM is causal), and different training objectives introduces substantial engineering complexity. A fully unified discrete approach collapses this complexity, enabling a single training pipeline, a single serving stack, and a single optimization surface. For industrial deployment at scale — which is the explicit framing of this paper, coming from Meituan's LongCat team — this simplification translates directly to reduced engineering cost, faster iteration, and more reliable production systems.

Theoretical significance for representation learning. The paper raises and provides evidence for a deeper question: can continuous perceptual signals be discretized without a fundamental performance penalty? The dominant narrative in vision-language modeling has been that continuous representations are superior for understanding because they preserve fine-grained information that quantization destroys. This is not an unreasonable position — compressing a high-resolution image through a discrete codebook inevitably loses information, and for tasks requiring precise spatial reasoning (OCR, document parsing, GUI interaction), that loss could be catastrophic. The paper challenges this narrative head-on, arguing that the performance gap attributed to discretization is instead a gap in representation capacity and training data scale. If this argument holds, it would revise our understanding of what makes multimodal models work — suggesting that semantic completeness of the pre-quantization representation matters more than the continuous-vs-discrete distinction.

Unification of understanding and generation. Perhaps the most practically important motivation is the paper's framing of understanding and generation as two instances of the same predictive process — just with different conditional priors. In the prevailing continuous-representation paradigm, understanding (encoding an image into features for the LLM) and generation (producing an image from the LLM's output) are architecturally distinct operations requiring different training objectives, different loss functions, and often entirely separate model components. The discrete paradigm collapses this distinction: understanding is "given image tokens, predict text tokens," and generation is "given text tokens, predict image tokens." Both are just autoregressive next-token prediction. This unification is not merely elegant — it directly addresses a documented conflict between understanding and generation objectives that has plagued unified multimodal models. When understanding and generation compete for model capacity under different optimization regimes, performance on both suffers. A single objective eliminates this competition by design.

The multimodality tax problem. The paper explicitly identifies and empirically addresses a phenomenon it calls the "multimodal tax" — the degradation of text capabilities when non-linguistic modalities are added to a language model (Section 3.1.4). This is a real and underappreciated problem: many multimodal models show strong visual performance but measurably worse text reasoning compared to their language-only counterparts. LongCat-Next's results on MMLU (83.95), SWE-Bench (43.0), and agentic benchmarks suggest that native discrete integration can mitigate or eliminate this tax. If replicable across model families, this finding would have direct implications for how organizations allocate their training compute — making the case that native multimodal training is not a capability tradeoff but a strict Pareto improvement.

Where Prior Approaches Fall Short

The paper organizes prior work into a taxonomy of representation strategies, each with documented limitations that motivate the DiNA approach.

Continuous projection approaches (the dominant paradigm). The most common strategy, exemplified by LLaVA-style models (Liu et al., 2023), Qwen-VL series, and InternVL, uses a vision encoder (typically a ViT) to produce continuous feature representations that are projected into the LLM's embedding space through a learned connector (an MLP or Q-Former). This approach has produced strong results on visual understanding benchmarks, but the paper identifies several structural limitations:

Architectural asymmetry. The vision encoder typically uses bidirectional attention (since images have no natural causal ordering), while the LLM backbone uses causal attention. This creates an impedance mismatch — the vision features encode global image context that the autoregressive LLM cannot fully exploit during generation since it only has access to past tokens. The paper's modality-agnostic MoE approach avoids this by processing everything causally.
Inability to generate. Continuous projection is fundamentally one-directional: image → features → text. There is no natural path back from text to images because the LLM operates in a discrete token space while the vision encoder's output space is continuous. Models that add generation capability on top of continuous understanding (like the EMU series, Janus, or Show-o) must bolt on separate generation heads, diffusion decoders, or additional training stages — undermining the architectural unity that the discrete paradigm promises.
Modality-specific routing. Many unified models introduce modality-aware components — modality-specific experts in MoE layers, 3D rotary position embeddings for video, or separate attention masks. The paper argues these are symptoms of incomplete unification: they reflect underlying representational incompatibility that the architecture must work around rather than resolve.

Discrete visual modeling (prior attempts with a performance ceiling). The paper acknowledges several prior efforts at discrete multimodal modeling, particularly Chameleon (Team, 2024), Emu3, and the EVE series. These works established the conceptual viability of tokenizing images into discrete codes and processing them autoregressively. However, the paper identifies a critical limitation that has prevented these approaches from achieving competitive understanding performance:

"There remains widespread doubt as to whether discrete visual modeling can achieve strong performance in both comprehension and autoregressive generation, as compressing rich visual information into a finite codebook inevitably hinders representation capacity." (Section 1)

This is not a strawman — it reflects genuine empirical results. Prior discrete models have consistently underperformed continuous-projection models on visual understanding benchmarks, particularly on tasks requiring fine-grained perception like OCR and document understanding. The paper's Table 3 shows that while prior unified discrete models like Janus-Pro, Show-o2, and OneCAT achieved decent generation quality, their understanding scores on benchmarks like MMMU (41.0, 48.9, 41.9 respectively) lagged far behind both specialized understanding models and LongCat-Next (70.6). The interpretation in the literature has been that discretization imposes an inherent performance ceiling.

The paper challenges this interpretation directly, arguing that the ceiling is not fundamental but rather reflects two specific bottlenecks that prior work failed to address: (1) insufficient representation capacity in the pre-quantization encoder, and (2) information loss from the discretization process itself. The paper's key insight is that if you solve both simultaneously — using a Semantic-and-Aligned Encoder for capacity and Residual Vector Quantization for loss mitigation — the ceiling disappears. Section 3.2.1 provides the empirical evidence: with sufficient training data, the discrete dNaViT achieves near-parity with a continuous baseline on understanding tasks, demonstrating that "discrete modeling does not have an inherent performance ceiling."

Encoder-free approaches. The paper also acknowledges encoder-free raw-pixel tokenization (like EVE, NEO) which bypasses the encoder entirely and tokenizes pixels directly. These approaches offer appealing simplicity and scalability but are acknowledged to "suffer from pixel redundancy" — the raw token sequences are extremely long, making autoregressive generation computationally expensive and making it difficult to capture the high-level semantic structure that an encoder would compress.

Audio modeling approaches. The audio landscape has been more favorable to discrete modeling, particularly through the RVQ-based tokenization popularized by models like Moshi. However, the paper identifies a limitation in how speech-language models handle text-speech alignment. Most models either generate speech tokens directly (pure audio) or use a two-stage pipeline (text-to-text then text-to-speech). The paper's approach of internal linguistic guidance — where the model simultaneously generates text and audio tokens with stochastic delay — aims to unify these strategies, enabling both parallel generation (suitable for low-latency full-duplex dialogue) and serial generation (suitable for high-quality streaming TTS) within a single training paradigm (Section 2.5.3).

How This Paper Positions Itself

The paper's positioning is carefully constructed around three framing moves that distinguish it from prior work while building on established foundations.

From "language plus auxiliary" to "language as native framework." The paper's most important conceptual move is reframing multimodality not as extending LLMs to handle other signals, but as recognizing that the NTP paradigm is already a universal sequence modeling framework — we just need to build the tokenizers that can represent non-linguistic data as discrete sequences. This is captured in the paper's central analogy:

"When multimodality is conceptualized analogously as a native linguistic extension of language, the problem simplifies considerably, where all modalities are represented as interoperable token sequences governed by a single shared autoregressive objective." (Section 1)

This framing has practical consequences. It means the research problem reduces to tokenizer design rather than architecture design. The decoder-only backbone remains unchanged — it's the tokenizer-detokenizer pairs that carry the burden of modality-specific processing. This decomposition allows the paper to leverage mature LLM training infrastructure (the LongCat-Flash MoE backbone, VHalf-based pipeline parallelism) while innovating primarily on the vision and audio tokenization components.

The concept of semantic completeness as a design principle. Rather than proposing a new architecture from scratch, the paper introduces a criterion — semantic completeness — that any successful discrete tokenizer must satisfy, and then constructs the tokenizer to meet this criterion. This is a different kind of contribution: it provides a framework for evaluating and improving tokenizers rather than just presenting a single implementation. The formal definition (Equation 1):

$P(A \mid z, Q) \approx P(A \mid I, Q)$

establishes that the discrete tokens $z$ must preserve the same downstream task-relevant information as the original image $I$ . This criterion decomposes into two sub-requirements — discriminative invariance (for understanding) and generative sufficiency (for generation) — giving the paper a principled way to diagnose and address tokenizer weaknesses.

Empirical refutation of the discrete performance ceiling hypothesis. The paper's most aggressive positioning move is its direct empirical challenge to the widespread belief that discrete visual modeling is inherently limited. The experiment in Section 3.2.1 is designed specifically to test this claim: by comparing discrete and continuous versions of the same architecture under identical training conditions, the paper shows that the initial performance gap shrinks as training data scales, and with sufficient data ( $\sim$ 300B tokens in the mid-training + SFT stages), the discrete model achieves near-parity. The paper then scales this to full size, achieving competitive performance with specialized continuous models like Qwen3-VL-A3B on visual understanding benchmarks (Table 1). This is the paper's central empirical claim: discrete tokens can fully substitute for continuous features if the tokenizer achieves semantic completeness and the model is trained at sufficient scale.

Reconciling understanding and generation within a single objective. The paper positions its approach as solving a well-documented conflict. Prior unified models (like Janus, Show-o, and others in Table 3) consistently show that generation quality and understanding quality trade off against each other — improving one often degrades the other. The paper's experiment in Section 3.2.3 provides direct evidence for why: under the DiNA framework, the unified model actually shows slightly lower loss on generation than a pure-generation model (0.02 lower, despite having half the task-specific data), while matching the pure-understanding model's performance (loss difference of only 0.006). This asymmetric benefit — understanding helps generation more than generation helps understanding — is presented as evidence that the conflict is an artifact of having separate objectives rather than an inherent tension in multimodal learning.

Industrial-strength rather than proof-of-concept. Throughout the paper, there is an emphasis on scale and production readiness that distinguishes it from many academic unified-model papers. The model uses a 68.5B-parameter MoE backbone (3B activated), was trained on over 2 trillion tokens, and is evaluated against the strongest available baselines at comparable scale (Qwen3-Omni, Gemini 2.5 Flash-Lite, specialized MLLMs). The infrastructure section (Section 5) describes custom pipeline parallelism optimizations (VHalf) designed specifically for the heterogeneous compute patterns of multimodal training. This is not a small-scale academic prototype — it's positioned as a deployment-ready foundation model, with open-source release of both the model and tokenizers to enable community replication and extension.

What the paper explicitly does NOT claim. It's important to note the boundaries the paper draws around its contributions. Section 6.2 is explicit about limitations: the current work does not extend beyond the canonical image-to-text and text-to-image directions to any-to-any generation or interleaved multimodal reasoning; the visual tokenizer is not fully optimized and focuses on semantic decoding consistency rather than pixel fidelity; and many components "remain to be validated and scaled." The paper positions itself as an initial step toward native multimodality — establishing feasibility and providing a training recipe — while acknowledging that true general-purpose multimodal intelligence requires future work on flexible cross-modal interaction, long-context multimodal reasoning, and co-design of pretraining objectives with discretization strategies.

3. Technical Approach

3.1 Reader Orientation

LongCat-Next is a decoder-only transformer model that processes text, images, and audio as sequences of discrete tokens — analogous to how language models process words — using a shared autoregressive next-token prediction objective to perform both understanding tasks (answering questions about images, transcribing speech) and generation tasks (creating images from text descriptions, synthesizing speech). The system solves the problem of multimodal architectural fragmentation by treating every modality as a "native language" that gets tokenized into discrete IDs, embedded into a shared vector space, and fed through a single modality-agnostic Mixture-of-Experts backbone, eliminating the need for separate vision encoders with continuous features, modality-specific routing, or distinct training objectives for understanding versus generation.

3.2 Big-Picture Architecture (Diagram in Words)

The system decomposes into five major components, illustrated in Figure 2 of the paper:

Modality-Specific Tokenizers — three separate encoders (dNaViT for vision, a Whisper-based RVQ encoder for audio, and a standard text tokenizer) that convert raw signals into sequences of discrete integer IDs. Each tokenizer is trained independently before being integrated into the full model.
Multimodal Embedding Layer — a learned embedding table (one per modality) that maps discrete token IDs to dense vectors in a shared embedding space. Visual embeddings use a codebook of size 8 × 16,384 (8 residual quantization levels, each with 16,384 entries); audio embeddings use decreasing codebook sizes across 8 levels (8K, 4K, 2K, 1K, 1K, 1K, 1K, 1K). These embeddings are randomly initialized and trained end-to-end with the full model.
Modality-Agnostic MoE Backbone (LongCat-Flash-Lite) — a 68.5B-parameter Mixture-of-Experts decoder-only transformer (3B activated parameters on average, ranging from 2.9B to 4.5B depending on the input context) that processes all token sequences identically, with no modality-specific attention patterns, position encodings, or routing logic. The architecture uses the Zero-Expert and Shortcut MoE design from prior work.
DepthTransformer Decoding Heads — modality-specific output heads that convert the backbone's hidden states back into predictions over the discrete token space. For vision and audio, a DepthTransformer (a specialized decoder architecture) generates multi-level tokens in parallel at each autoregressive step, enabling efficient decoding of the hierarchical residual quantization structure without sequential generation per level.
Modality-Specific De-Tokenizers — decoders that transform predicted discrete token sequences back into perceptual signals: a ViT-based pixel decoder plus a flow-matching refiner for images, and a flow-matching decoder plus vocoder for audio. These operate only during generation; understanding tasks do not invoke the de-tokenizer.

Information flow for understanding: Raw input (image/audio/text) → tokenizer → discrete IDs → embedding lookup → modality-agnostic backbone → text head → text tokens → output. The backbone produces hidden states; only the text head is used, generating answer tokens autoregressively.

Information flow for generation: Text input → text tokenizer → discrete IDs → embedding lookup → modality-agnostic backbone → DepthTransformer head → multi-level discrete tokens → de-tokenizer → perceptual output. The backbone conditions on text tokens and predicts image or audio tokens step by step, with the DepthTransformer decoding all residual quantization levels in parallel at each step.

3.3 Roadmap for the Deep Dive

First, the semantic completeness criterion (Equation 1), which establishes the theoretical requirement that discrete tokens must satisfy to avoid a performance ceiling — this is the foundation that motivates all subsequent design choices.
Second, the Semantic-and-Aligned Encoder (SAE) concept, which defines the class of pre-quantization feature extractors that provide sufficient representation capacity, and the specific choice of Qwen2.5-ViT as the SAE for dNaViT.
Third, the Residual Vector Quantization (RVQ) mechanism (Equations 3–5), including how multi-level tokens are produced, how codebooks are updated via EMA, and how the quantization loss preserves semantic information.
Fourth, the de-tokenizer design (Equation 6), including the pixel decoder, the flow-matching refiner, and the theoretical analysis of why residual connections enable information recovery even without reconstruction supervision.
Fifth, the additive encoding and DepthTransformer mechanism, which enables efficient autoregressive modeling over multi-level tokens by summing across levels for input and decoding all levels in parallel for output.
Sixth, the audio tokenizer (Equation 8), following the same RVQ principles but adapted for temporal signals with a Whisper encoder and flow-matching decoder.
Seventh, the language model backbone and modality-agnostic MoE design, including the training dynamics analysis showing how modality specialization emerges naturally without architectural enforcement.
Eighth, the internal linguistic guidance mechanism for speech generation, including the unified parallel/serial training paradigm with stochastic delays.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that discrete autoregressive modeling can be extended from language to vision and audio if the tokenizers achieve semantic completeness — preserving sufficient information in the discrete representation to support both discriminative understanding and high-fidelity generation — and that this unification can be realized within a single modality-agnostic decoder-only architecture at industrial scale.

Semantic Completeness: The Design Criterion for Discrete Multimodal Tokens

The paper establishes a formal criterion that any successful discrete tokenizer must satisfy, using this criterion to motivate the choice of encoder (SAE), the quantization strategy (RVQ), and the de-tokenizer design. The criterion is called semantic completeness and is defined through a probabilistic equivalence (Section 2.2.1, Equation 1):

$P(A \mid z, Q) \approx P(A \mid I, Q)$

where $I$ is an input image sampled from the continuous visual manifold $\mathcal{M}$ , $z = \{\text{idx}_1, \text{idx}_2, \ldots, \text{idx}_n\}$ is the sequence of discrete indices produced by a quantization mapping $Q(\cdot)$ , $Q$ denotes an image-centric inquiry associated with some downstream task $\mathcal{T}$ , and $A$ denotes the optimal response or latent output corresponding to that inquiry.

What this equation means in operational terms: given any question you might want to ask about an image (captioning it, reading text from it, answering a reasoning question about it, or reconstructing it), conditioning on the discrete token sequence $z$ should give you the same answer distribution as conditioning on the raw image $I$ . If this holds, then the discrete tokens are a lossless proxy — you can throw away the pixels once you have the tokens, and downstream performance on any task will be unaffected.

Why this form: the equation formalizes what "good enough" means for a tokenizer intended to serve as the sole input representation for a multimodal model. It is a necessary condition for the DiNA paradigm to work: if the equivalence fails, then the model operating on discrete tokens will have strictly less information than one operating on continuous features, creating an inherent performance ceiling. The paper's core empirical claim is that this condition can be approximately satisfied with the right encoder and quantization strategy, and the experimental evidence in Section 3.2.1 (comparing discrete and continuous versions trained on the same data) is designed to validate this claim.

The criterion decomposes into two sub-properties:

Discriminative Invariance: the discretization process $Q(\cdot)$ must preserve the core semantic attributes of the original image. For any understanding task ranging from fine-grained recognition to complex reasoning, the discrete representation $z$ must retain the critical information contained in the raw pixels $I$ .
Generative Sufficiency: given the high redundancy in pixel space, the discrete codes $z$ should capture the essential visual semantics required for faithful image reconstruction. The de-tokenizer $D(\cdot)$ should produce $I' \approx D(z)$ that preserves structural and textural content, and $z$ itself should serve as a semantically sufficient descriptor for the language model to generate images.

These two sub-properties are not independent: a representation that captures only high-level semantics (like a CLIP embedding) may be discriminatively invariant but not generatively sufficient (you cannot reconstruct the image from it). Conversely, a representation optimized purely for pixel-level reconstruction (like a VAE latent) may be generatively sufficient but not discriminatively invariant (it captures texture but not semantic structure). The paper's key design insight is that a Semantic-and-Aligned Encoder trained with large-scale vision-language supervision can satisfy both simultaneously because the training objective (Equation 2) forces the representation to capture information relevant to a diverse set of language-conditioned tasks, which includes both semantic understanding and the visual details needed to answer detailed questions.

The Semantic-and-Aligned Encoder (SAE): A Class of Pre-Quantization Feature Extractors

The paper defines SAE as a mapping $\mathbf{z}_p = E_{\text{sae}}(I)$ that projects an image $I$ into a pre-quantization representation $\mathbf{z}_p$ . The requirement is that this representation preserves the information necessary for answering diverse image-centric queries, formally:

$P(A \mid \mathbf{z}_p, Q) \approx P(A \mid I, Q)$

where $A$ denotes the ground-truth response to query $Q$ .

What makes an encoder an SAE: the encoder must have been trained with large-scale language-grounded supervision over diverse image-centric tasks. The paper identifies two key properties that emerge from such training:

Semantic richness: the representation captures both high-level concepts and fine-grained visual details (such as textual content in images), because the language-conditioned training objective — answering questions, generating captions, reading text — requires attending to all levels of visual information.
Affinity with language models: because the representation was optimized to interact with language through tasks like captioning and QA, it already lives in a space that is "close" to the language model's representational space, facilitating integration into a unified discrete token space.

Training objective for SAE from scratch (Equation 2):

$L_{\text{SAE}} = \mathbb{E}_{(I, Q, A)} \left[ -\log P(A \mid \mathbf{z}_p, Q) \right]$

where the expectation is over image-query-answer triples, and the probability is modeled by a predictor that takes the SAE representation and the query text as input.

What this computes: this is the negative log-likelihood of the correct answer given the SAE features and the query. Minimizing this objective forces the SAE features to retain whatever information is needed to correctly answer the diverse set of training queries — which implicitly enforces semantic completeness.

Why this form: unlike contrastive objectives (e.g., CLIP) that only require the representation to distinguish between matching and non-matching image-text pairs, this multi-aspect predictive objective provides a richer training signal. A query might require reading specific text, identifying object attributes, or performing spatial reasoning — the representation must preserve all this information to minimize the loss. This makes it a natural candidate for the pre-quantization space.

Practical instantiation: the paper does not train an SAE from scratch due to computational cost. Instead, it directly adopts Qwen2.5-ViT (with a 28× spatial compression ratio) as the SAE. This choice is pragmatic: Qwen2.5-ViT was trained through multiple progressive stages — first global alignment via CLIP-style contrastive learning, then detailed learning via multi-aspect language-conditioned supervision across diverse QA tasks — which matches the paper's definition of an SAE. The encoder operates at native resolution, producing latent representations under variable sequence lengths depending on the input image dimensions.

The paper explicitly notes that other SAE candidates (QwenViT, MoonViT, AIMv2) could serve the same role, and that a better-optimized SAE "could potentially yield further improvements." The SAE is frozen after training and prior to any quantization or autoregressive modeling — it serves purely as a feature extractor whose output is the input to the tokenization process.

Why an SAE over alternatives: the paper contrasts SAE with three other representation strategies (Section 2.2.1):

Low-level reconstructive models (VAEs, VQ-VAEs, used in EMU, Chameleon, LWM, VILA-U): these excel at pixel-level fidelity but struggle with high-level conceptual reasoning because their training objective (pixel reconstruction) does not incentivize semantic abstraction. They achieve generative sufficiency but not discriminative invariance.
Self-supervised semantic encoders (DINOv2, SigLIP, used in Janus series): these capture structural or contrastive features that work well for understanding but lack the explicit semantic grounding needed for generative reconstruction. They achieve discriminative invariance but not generative sufficiency.
Encoder-free raw-pixel tokenization (EVE series, NEO): these offer simplicity and scalability by bypassing the encoder entirely, but suffer from pixel redundancy — the sequences are extremely long, and the model must learn all visual semantics from scratch.

The SAE sits at a sweet spot: it leverages large-scale pretraining to already capture rich semantic structure, then the subsequent quantization and autoregressive modeling build on this foundation rather than learning visual semantics from scratch.

Residual Vector Quantization: Hierarchical Discretization with Information Preservation

Once the SAE produces its continuous feature representation $\mathbf{z}$ , it must be converted into discrete tokens. Single-stage vector quantization (VQ) maps each feature vector to the single nearest codebook entry, introducing substantial information loss because the high-dimensional, semantically rich SAE space gets collapsed to a finite set of codebook vectors. The paper adopts Residual Vector Quantization (RVQ) to mitigate this loss through hierarchical, iterative refinement.

The RVQ tokenization process (Equations 3):

$\mathbf{r}_0 = f_{\text{proj}}(\mathbf{z})$ $\hat{\mathbf{q}}_l = \text{VQ}(\mathbf{r}_{l-1})$ $\mathbf{r}_l = \mathbf{r}_{l-1} - \hat{\mathbf{q}}_l$ $\hat{\mathbf{z}} = \sum_{l=1}^{L} \hat{\mathbf{q}}_l$

where $\mathbf{z}$ is the SAE output, $f_{\text{proj}}$ is a learnable projection layer that maps the SAE features to the quantization space, $\mathbf{r}_0$ is the initial projected latent representation, $\hat{\mathbf{q}}_l = \text{VQ}(\mathbf{r}_{l-1})$ is the quantized feature at level $l$ (the nearest codebook entry to the residual from the previous level), $\mathbf{r}_l = \mathbf{r}_{l-1} - \hat{\mathbf{q}}_l$ is the residual error after $l$ levels of quantization, $L$ is the total number of RVQ levels (8 for both vision and audio), and $\hat{\mathbf{z}}$ is the final quantized representation — the sum of all level-wise quantized features.

What this computes in operational terms: at level 1, the system finds the single best codebook entry to approximate the original feature. Then it computes the difference between the original and this approximation (the residual), and at level 2, it finds a codebook entry to approximate that residual. This process repeats level by level, each time approximating the remaining error from all previous approximations. The final reconstruction is the sum of the approximations from all 8 levels. This produces 8 discrete token IDs per spatial position (one per level), each drawn from a codebook of size 16,384 for vision and decreasing sizes for audio.

Why this form over single-stage VQ: a single large codebook with enough entries to achieve high fidelity would be computationally infeasible (the nearest-neighbor search cost grows linearly with codebook size). RVQ achieves the same representational capacity exponentially more efficiently: with $L$ codebooks of size $K$ each, the total number of possible quantized representations is $K^L$ . For the vision tokenizer with $L = 8$ and $K = 16,384$ , this is $16,384^8 \approx 2^{112}$ possible representations — a combinatorially vast space — while only requiring $8 \times 16,384 = 131,072$ codebook entries total. This is the "exponential representation space" the paper refers to in Section 1.

The commitment loss (part of Equation 5):

$L_{\text{commit}}$

This is the standard VQ commitment loss that encourages the projected representation $\mathbf{r}_0$ to stay close to the assigned codebook entries, averaged across all $L$ residual levels. It prevents the encoder from drifting arbitrarily far from the codebook, which would make quantization meaningless.

The semantic reconstruction loss (part of Equation 5):

$L_{\text{semantic}}$

This is a cosine similarity loss between the original SAE features and the features reconstructed from the quantized representation. A lightweight decoder reconstructs the pre-quantization semantic features from $\hat{\mathbf{z}}$ , and the cosine distance between the reconstruction and the original SAE features is minimized. This ensures that the quantized representation $\hat{\mathbf{z}}$ preserves the semantic content of $\mathbf{z}$ — it is the mechanism by which semantic completeness is explicitly enforced during tokenizer training.

The full quantization objective (Equation 5):

$L_{\text{quant}} = \lambda_c L_{\text{commit}} + \lambda_s L_{\text{semantic}}$

where $\lambda_c$ and $\lambda_s$ are scalar weights balancing the two loss terms. $\lambda_c$ controls how strongly the encoder features are pulled toward the codebook (too high and the features collapse to the codebook; too low and the codebook doesn't learn), while $\lambda_s$ controls the emphasis on preserving semantic information in the quantized representation.

Codebook update mechanism (Equation 4): the codebook entries are not updated via gradient descent (which would require backpropagation through the non-differentiable argmin operation). Instead, they use an Exponential Moving Average (EMA) update:

$e_k \leftarrow \frac{m_k}{N_k}$

where for each codebook entry $e_k$ , the system maintains a running cluster size $N_k \leftarrow \gamma N_k + (1 - \gamma) |\mathcal{S}_k|$ and an embedding sum $m_k \leftarrow \gamma m_k + (1 - \gamma) \sum_{j \in \mathcal{S}_k} r_j$ , with $\mathcal{S}_k = \{r_j : \arg\min_i \|r_j - e_i\| = k\}$ being the set of residual vectors assigned to entry $k$ , and $\gamma$ being the decay factor. Laplace smoothing is applied to $N_k$ for numerical stability, and inactive entries (those with $N_k < 1$ ) are re-initialized from the current batch to maintain codebook utilization.

Why EMA over gradient descent: the quantization operation $\hat{\mathbf{q}}_l = \text{VQ}(\mathbf{r}_{l-1})$ is non-differentiable, so gradients cannot flow through it to the codebook entries. EMA provides a direct, stable alternative: each codebook entry becomes the weighted moving average of all feature vectors assigned to it, which naturally converges to the centroid of its assigned cluster without the need for gradient-based optimization.

Tokenization training stages (Stage 1, Section 4.1.1, Figure 4): the tokenizer is trained in two phases. First, an initial fixed-resolution stage for fast convergence. Second, an any-resolution stage with RVQ to adapt the quantization process to variable token lengths, with maximum training sequence length set to 8192. The training corpus comprises approximately 50M images from LAION, COYO, DataComp, TextAtlas, and a subset of in-house visual understanding data, plus high-fidelity synthetic data (e.g., MidJourney) to enhance generation quality. Training is at native resolution until convergence, with maximum image resolution set to 1,736 × 1,736. The resulting discrete tokens are designed to preserve both discriminative and generative information.

Visual De-Tokenizer: Reconstructing Images from Discrete Codes

After the tokenizer maps images to discrete IDs, the de-tokenizer must perform the inverse mapping — reconstructing pixel-level images from the discrete token sequences. This is needed both for evaluating reconstruction fidelity (as a diagnostic of semantic completeness) and for the generation pipeline (where the language model predicts discrete tokens that must be rendered as images).

The pixel decoder (Section 2.2.4) is a 400M-parameter Vision Transformer trained from scratch. It processes the discrete code embeddings through a series of steps:

Patch unmerging: a learnable MLP-based module reverses the spatial merging applied by the SAE encoder (which compresses the image by a factor of 28×), restoring the original patch count. This is necessary because the SAE's spatial compression means the discrete tokens correspond to a coarse spatial grid — the unmerger expands this back to the original ViT patch resolution.
Transformer processing: the unmerged features pass through a stack of transformer layers with 2D Rotary Position Embeddings (RoPE), which encode spatial position in a way that naturally handles arbitrary resolutions.
Pixel projection: a linear head projects the final hidden states to pixel space, producing the reconstructed image.

The pixel decoder training objective (Equation 6):

$L_{\text{dec}} = \lambda_1 L_{\text{pixel}} + \lambda_2 L_{\text{percep}} + \lambda_3 L_{\text{align}}$

where $L_{\text{pixel}}$ is a pixel-level reconstruction loss (likely MSE or L1 between the reconstructed and original images), $L_{\text{percep}}$ is a perceptual loss (likely based on features from a pretrained network, capturing structural similarity beyond raw pixel differences), and $L_{\text{align}}$ is an alignment loss (likely ensuring semantic consistency between the reconstruction and the original).

The image refiner: the pixel decoder alone produces structurally coherent but "overly smooth" reconstructions with "diminished high-frequency details and perceptual sharpness" (Section 2.2.4). To address this, a lightweight image refiner is trained with a flow-matching objective $L_{\text{flow}}$ . The refiner takes two conditioning signals: (1) the pixel decoder's reconstruction, concatenated with the noise latent along the channel dimension for spatial guidance, and (2) the projected discrete code embeddings for semantic conditioning. The reconstruction from the pixel decoder serves as a structural anchor — the refiner focuses on enhancing fine-grained textures and high-frequency details without deviating from the semantic content established by the pixel decoder.

The refiner is initialized from OmniGen2 weights and continued trained with flow-matching loss. The training data for the de-tokenizer reuses the same image corpus as the tokenizer, supplemented with SAM-1B, RenderedText, IDL, and a collection of high-resolution in-house images (Section 4.1.1). Both the SAE encoder and the codebook remain frozen throughout de-tokenizer training.

Why a two-stage decoder-plus-refiner design: a single-stage pixel decoder would need to simultaneously recover global structure and fine-grained details, which is difficult because the SAE's compression discards high-frequency information that the decoder must hallucinate — and hallucinated details often look unnatural. The refiner separates concerns: the pixel decoder handles semantic reconstruction (layout, objects, text), and the refiner handles perceptual enhancement (textures, sharpness). Because the refiner is conditioned on the pixel decoder's output, it can focus on detail without risking semantic drift. The paper emphasizes that the refiner "is designed to focus on detail restoration" and plays a "critical role in faithfully recovering the original image at the semantic level" — within the LLM autoregressive framework, the predicted discrete tokens inherently encode semantic content (layout, structure), and the refiner ensures perceptual quality.

Information Recovery from Residual Architectures: A Structural Justification

The paper provides a theoretical analysis (Section 2.2.4) of why the SAE representation retains sufficient information for reconstruction even though the SAE was never trained with a reconstruction objective. This analysis is central to justifying that semantic completeness is achievable without explicitly training the encoder for generation.

Residual pathway for low-level signal propagation (Equation 7). The encoder is formulated as a sequence of $L$ residual blocks, where each block's output is the sum of its input and a learned transformation:

$x_l = x_{l-1} + F_l(x_{l-1})$

The final latent representation $\mathbf{z}_p$ can be expressed as the additive accumulation of the initial projection and all subsequent feature increments:

$\mathbf{z}_p = x_0 + \sum_{l=1}^{L} F_l(x_{l-1}) = x_0 + F_1(x_0) + F_2(x_1) + \cdots + F_L(x_{L-1})$

where $x_0$ is the initial patched embedding, $F_l$ is the transformation at block $l$ , and $x_l$ is the output of block $l$ .

What this equation reveals: the identity mapping $x_{l-1}$ in each residual connection means that fine-grained visual signals from earlier layers are not overwritten by higher-level semantic abstractions — they are additive to the representation. The final feature $\mathbf{z}_p$ contains contributions from every layer, including the early layers that encode pixel-level information. This means the representation retains high mutual information $I(I; \mathbf{z}_p)$ with the original image even though the training objective optimizes only for semantic tasks.

Empirical evidence (Section 3.2.2, Figure 5, Table 6): the paper tests this hypothesis by attaching a lightweight ViT-based decoder to various frozen encoders and measuring reconstruction quality. The key findings:

A randomly initialized ViT-B/16 achieves the best reconstruction performance (PSNR 30.52, SSIM 0.887), outperforming both its pretrained counterpart (PSNR 21.86, SSIM 0.581) and pretrained ResNet-50 (PSNR 20.88). This is surprising: a network that has never seen real images reconstructs better than one trained on ImageNet. The paper hypothesizes that random weights produce noise-like signals that are easier for the decoder to "denoise," while the residual pathways preserve pixel information regardless of weight values.
QwenViT without the merger module achieves moderate reconstruction (PSNR 18.16), while QwenViT with the merger (which further downsamples from 14× to 28× compression) shows noticeable degradation. This confirms that aggressive spatial compression reduces the information available for reconstruction.
Qualitative visualizations show that SAE-style encoders can recover coarse image-level structures (layout, object positions) but are less effective at reconstructing fine-grained, high-frequency details — which is exactly why the flow-matching refiner is needed.

Why this matters: the analysis demonstrates that semantic completeness is structurally grounded in the residual architecture of modern vision encoders, not solely dependent on training objectives. This provides a theoretical justification for why an SAE trained purely for semantic tasks can still serve as the foundation for a generative tokenizer — the architecture itself preserves a latent pathway for information recovery. This insight is presented as "previously underexplored" (Section 3.2.2) and provides a new perspective on the trade-off between pixel-level and semantic-level information in vision encoders.

Additive Encoding and DepthTransformer: Efficient Multi-Level Autoregressive Modeling

The RVQ produces multiple levels of discrete tokens per spatial position: for vision, 8 levels, each from a 16,384-entry codebook. Naively treating each level as a separate token in the autoregressive sequence would multiply sequence length by 8, making generation 8× more expensive. The paper instead uses two complementary mechanisms: additive encoding for input and parallel decoding for output.

Multi-level additive encoding: when feeding vision tokens into the language model, the embeddings from all 8 levels at each spatial position are summed together before entering the backbone. Formally, if a spatial position has level-wise embeddings $\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_8$ (each drawn from a separate embedding table for that level), the input to the backbone is:

$\mathbf{e}_{\text{input}} = \sum_{l=1}^{8} \mathbf{e}_l$

This means each spatial position occupies a single token slot in the autoregressive sequence, not 8. The backbone sees a single vector per position that integrates information from all quantization levels. The paper states that these embeddings "are not shared across levels, allowing each level to capture complementary information" (Section 2.5.1) — the separate embedding tables mean that level 1 can learn to encode coarse structure while level 8 can learn to encode fine details, and the additive combination lets the backbone attend to all levels simultaneously.

DepthTransformer for parallel decoding: during generation, the backbone produces a single hidden state per spatial position. The DepthTransformer — a specialized decoder head — takes this hidden state and produces predictions for all 8 quantization levels in parallel:

$\{p^{(1)}, p^{(2)}, \ldots, p^{(8)}\} = \text{DepthTransformer}(\mathbf{h})$

where $p^{(l)}$ is the probability distribution over the 16,384 codebook entries at level $l$ , and $\mathbf{h}$ is the hidden state from the backbone at that position.

What this enables operationally: a single autoregressive step in the backbone produces all 8 tokens for that spatial position simultaneously. Without this, generation would require 8 sequential steps per spatial position (predict level 1, condition on it to predict level 2, condition on both to predict level 3, etc.), which would be 8× slower. The DepthTransformer architecture — described as a "task-aware" variant of the original DepthTransformer design from prior work — enables parallel multi-level prediction while maintaining the quality benefits of hierarchical quantization.

Multi-level supervision during training: the training objective computes loss across all levels. The backbone's single autoregressive step is supervised by cross-entropy losses on all 8 level-wise predictions from the DepthTransformer:

$L = \sum_{l=1}^{8} w_l \cdot \text{CrossEntropy}(p^{(l)}, y^{(l)})$

where $y^{(l)}$ is the ground-truth discrete token at level $l$ and $w_l$ are level-specific weights. The paper does not specify the exact weight values but notes that this design allows the model to learn to distribute information appropriately across levels — early levels capture coarse structure while later levels refine details.

The Pre-Buffer module (Section 3.2.1): the paper identifies a practical issue with the additive encoding: after summing multi-level embeddings, the resulting vector may lose the structured separation between levels that the RVQ training established. To address this, a lightweight Pre-Buffer module (implemented as a single-layer FFN) is inserted after the codebook lookup and before the backbone:

$\mathbf{e}_{\text{buffered}} = \text{FFN}\left(\sum_{l=1}^{8} \mathbf{e}_l\right)$

This remapping operation "substantially accelerates convergence and improves the expressiveness of the discrete tokens" (Section 3.2.1). The ablation in Figure 9 (Experiment II vs. III) shows that adding the Pre-Buffer reduces the pre-alignment loss gap between discrete and continuous models.

Why additive encoding plus DepthTransformer over sequential level prediction: the alternative — treating each level as a separate autoregressive step — would increase sequence length by 8× and make the effective context window 8× shorter for a given computational budget. The additive encoding collapses 8 levels into 1 token, preserving context efficiency. The DepthTransformer recovers the multi-level structure at the output, enabling the model to generate all levels without sequential dependency. This design "unlocks an exponential representation space for multi-level tokens, while maintaining the computational efficiency of a single autoregressive step" (Section 1) — the representational capacity scales as $K^L$ due to the RVQ combinatorics, but the computational cost scales only as $O(1)$ per spatial position.

Audio Tokenizer: Applying the Same Principles to Temporal Signals

The audio tokenizer follows the same RVQ-based discrete representation principles as the vision tokenizer but is adapted for temporal (1D) rather than spatial (2D) signals.

Architecture (Section 2.3, Figure 6):

Whisper encoder: a pretrained Whisper-large-v3 model extracts audio features from the input waveform. Whisper was trained on large-scale speech recognition data and captures both semantic content (what is being said) and paralinguistic information (emotion, tone, speaker characteristics, environmental context). The encoder acts as the audio equivalent of the SAE — providing semantically rich features as input to quantization.
Downsampling: the Whisper features are downsampled by a factor of 4 before quantization. This temporal compression reduces the sequence length, making autoregressive modeling more efficient.
8-layer RVQ: the downsampled features are quantized through 8 residual levels, with decreasing codebook sizes: 8K, 4K, 2K, 1K, 1K, 1K, 1K, and 1K. The decreasing sizes reflect the expectation that later levels capture progressively finer and less information-rich residuals. The resulting token rate is 12.5 Hz (12.5 tokens per second of audio after compression).
Dual-branch training: the discrete tokens are forwarded along two parallel branches:
- Understanding branch: tokens are fed into a frozen pretrained LLM (Qwen3-1.7B) that is trained on diverse audio understanding tasks. This branch encodes both semantic and acoustic information and aligns the audio tokens with the LLM's textual embedding space. The paper notes that these audio representations are "transferable across LLM architectures and vocabularies" — once trained, the tokenizer's understanding branch can be discarded.
- Reconstruction branch: tokens pass through a decoder (architecture symmetric to the encoder) that reconstructs coarse Mel spectrograms. A flow-matching model then refines these coarse spectrograms into high-fidelity Mel spectrograms, which are converted to audio waveforms by a HiFi-GAN vocoder.

Training objective (Equation 8):

$L_{\text{audio}} = \lambda_1 L_{\text{recon}} + \lambda_2 L_{\text{commit}} + \lambda_3 L_{\text{llm}}$

where $L_{\text{recon}}$ is the Mel spectrogram reconstruction loss (training the decoder to recover the frequency-domain representation from discrete tokens), $L_{\text{commit}}$ is the standard RVQ commitment loss, and $L_{\text{llm}}$ is the language model loss for audio understanding tasks (training the understanding branch to use the discrete tokens for tasks like ASR, audio QA, etc.).

Training stages (Section 4.1.2):

Stage 1 (Decoder Warm-up): encoder and LLM are initialized from Whisper-large-v3 and Qwen3-1.7B respectively, decoder is randomly initialized. Only the decoder is trained on Mel spectrogram reconstruction. The encoder and LLM remain frozen.
Stage 2 (Semantic-Acoustic Joint Training): all modules except the LLM and flow-matching module are updated. The training objective combines all three losses. The RVQ module is enabled during this stage.
Stage 3 (Decoder Fine-tuning): the decoder is reconfigured as a Diffusion Transformer (DiT) to denoise artifacts introduced by RVQ. High-quality 24kHz audio data is used. This stage produces 24kHz Mel-spectrogram representations well-adapted for downstream vocoders.

Training data: approximately 2.5 million hours total, consisting of (i) large-scale web-collected Chinese and English speech data cleaned and automatically transcribed using open-source ASR models, and (ii) a curated dataset of high-quality multilingual and dialectal ASR data, in-house synthetic speech, and music/sound captioning datasets (Section 4.1.2).

Why Whisper as encoder: Whisper provides a strong starting point because it was trained on 680,000 hours of weakly supervised speech data, capturing a wide range of acoustic conditions, languages, and speaking styles. The paper notes that leveraging scaling laws observed in audio tokenizers, the Whisper encoder provides "strong semantic comprehension of speech, environmental sounds, and music, while maintaining robust acoustic reconstruction fidelity."

Why a separate LLM during tokenizer training: the frozen Qwen3-1.7B in the understanding branch serves as an alignment target — it ensures that the discrete audio tokens are compatible with the representational space of language models. By training the tokenizer's encoder to produce tokens that are useful for the LLM's understanding tasks, the paper ensures that the audio representations will integrate well into the final multimodal model. This LLM is discarded after tokenizer training — it serves as a training signal, not as part of the deployed system.

Language Model Backbone: Modality-Agnostic Mixture-of-Experts

The decoder-only backbone is the LongCat-Flash-Lite A3B model — a Mixture-of-Experts architecture with 68.5B total parameters and an average of 3B activated parameters (range: 2.9B to 4.5B depending on context). The underlying architecture uses the Zero-Expert and Shortcut MoE design from prior work (Section 2.4).

Modality-agnostic design: unlike many multimodal architectures that introduce modality-specific modifications — modality-aware MoE routing, 3D Rotary Position Embeddings for video, or bidirectional attention for vision inputs — LongCat-Next processes all tokens identically:

All tokens (text, vision, audio, special tokens) pass through the same transformer layers with the same attention mechanism (presumably causal, based on the decoder-only description).
The MoE routing is modality-agnostic: experts are selected based on token content, not token type, and there is no architectural enforcement of modality specialization.
There are no modality-specific attention masks or position encodings — vision tokens at different spatial positions are treated as a flat sequence with standard position embeddings.

Why modality-agnostic: the paper argues that this design forces the model to develop a genuinely unified representation space. If vision and audio tokens truly share semantics with language tokens, they should not need separate processing pathways — the same attention mechanism and expert routing should work for all modalities. This is the architectural manifestation of the DiNA philosophy: multimodality should be handled by the tokenizer, not the backbone.

Training dynamics analysis (Section 3.2.5, Figure 11): the paper provides empirical evidence that modality-agnostic MoE training naturally induces functional specialization without architectural enforcement. Comparing the pure text model with the natively trained multimodal counterpart:

Expert specialization emerges: a subset of experts gradually develops preferences for specific modalities (vision, audio, or text), even though the routing mechanism was never told about modalities. This is visible in Figure 11 (right) as distinct clusters of experts that are preferentially activated by tokens of a particular modality.
Routing becomes more structured: the routers develop "increasingly distinct and stable selection patterns over experts" compared to the pre-multimodal training state.
Capacity utilization increases: the average number of routed tokens per expert increases from 507.1 to 584.6 after multimodal training, indicating that the model is using more of its available expert capacity when handling diverse modalities.

What these dynamics imply: the MoE architecture functions as a multi-task learner without explicit task conditioning. The model learns to allocate different experts to different modalities based purely on the token content and the optimization pressure from the next-token prediction objective. This is presented as evidence that modality-agnostic design is not just simpler — it's sufficient and possibly optimal, because the model can discover modality-specialized computation patterns through standard training.

End-to-end multimodal embedding (Section 2.5.1): all multimodal embeddings are randomly initialized and trained jointly with the backbone. The visual embedding uses a codebook of size 8 × 16,384 = 131,072 total entries, with multi-level tokens combined through additive encoding. The audio embedding follows the same design with decreasing codebook sizes. Importantly, these embeddings "are learned end-to-end to support both multimodal understanding and generation" — the embedding table is optimized jointly with the backbone's parameters, allowing the model to learn embedding representations that are optimal for both types of tasks.

Relationship to tokenizer embeddings: the pre-quantization features from the SAE are used "solely for establishing the discrete clustering assignments in the RVQ process, rather than directly dictating the embedding values themselves" (Section 2.5.1). This means the continuous semantic features determine which discrete token a visual patch maps to (via nearest-neighbor lookup in the codebook), but the actual vector representation of that token is learned from scratch during language model training. The model is free to develop its own "native language" for vision, reinterpreting the discrete codes in terms that are optimal for the autoregressive task.

Modality heads (Section 2.5.2): the model has separate output heads for text and for multimodal tokens:

Text head: a standard MLP that maps hidden states to vocabulary probabilities. Used for text generation during understanding tasks.
Multimodal heads: task-aware DepthTransformers that decode hidden states into multi-level token predictions. Used for visual and audio generation. The DepthTransformer generates all levels in parallel at each autoregressive step.

During training, multi-level supervision is applied: the language model performs a single autoregressive prediction step, and the DepthTransformer simultaneously predicts all levels for that step, with a loss term for each level.

Internal Linguistic Guidance for Speech Generation

A key challenge in speech-language models is that the model must produce coherent, semantically accurate speech — not just plausible-sounding audio. The paper addresses this through internal linguistic guidance, where the model simultaneously generates text and audio tokens, with the text tokens serving as an internal "script" that guides the speech generation (Section 2.5.3, Figure 7).

Two modalities of audio: the paper distinguishes between:

Pure audio modality: audio tokens generated or received without accompanying text. This is typical for user input in real-time voice interaction scenarios, where textual guidance is unavailable.
Text-guided audio modality: audio tokens paired with aligned text tokens at the segment level. The text and audio embeddings for each time step are embedded separately through dedicated input embedding layers and then fused via element-wise summation. This is used for the model's speech output.

Two generation strategies for text-guided audio:

Parallel generation: at each decoding step, the model predicts both a text token and an audio token simultaneously. To maintain alignment, the first audio token is intentionally delayed by a specified number of steps relative to the text. Figure 7 illustrates this with a delay of 1 — the model outputs text token T1 and audio token A1 at the same step, but A1 corresponds semantically to an earlier time point. This strategy eliminates response latency and is suitable for full-duplex (simultaneous bidirectional) dialogue.
Serial generation: the model first generates the complete guided text segment, then generates the corresponding audio segment. At each step, the model only predicts tokens from a single modality, which "simplifies the process and avoids conflicts between modality representations" and "ensures high linguistic quality in the generated speech."

Unified training with random delays: to train the model for both strategies simultaneously, the paper proposes a general training paradigm: for each aligned text-audio segment, the delay between text and audio is randomly selected from a uniform distribution over $\{1, 2, \ldots, \text{text_segment_length}\}$ . This means:

When the delay equals the text segment length, the model experiences serial generation (all text tokens are generated before any audio tokens).
When the delay is a small number, the model experiences parallel generation (text and audio overlap significantly).
The model learns to be robust to arbitrary delays, enabling it to generate text-guided speech at any alignment offset.

Computational loss design: for text-guided audio modality, the loss is computed jointly for both text tokens and audio tokens:

$L_{\text{text-guided}} = L_{\text{text}} + L_{\text{audio}}$

For pure audio modality (used in ASR and interleaved data), loss computation is omitted for the audio tokens following prior work — the model processes these tokens as conditioning context but is not trained to reproduce them.

Special tokens for modality conversion: the paper introduces three special tokens to indicate modality boundaries and facilitate segment alignment: AS (Audio Start), AE (Audio End), and TE (Text End). These mark the beginning and end of audio segments and the end of text segments, respectively, providing structural cues for the model to track modality transitions.

Experimental validation (Section 3.2.4): the paper evaluates whether parallel generation can match serial generation in semantic accuracy. On LlamaQuestions, parallel achieves 79.33 vs. serial's 81.67. On ReasoningQA, parallel achieves 74.95 vs. serial's 80.30. The gaps are presented as relatively small, demonstrating that the random-delay training "significantly reduces discrepancies between the two generation strategies" and that parallel generation can "preserve response quality while offering improved efficiency."

Why internal linguistic guidance over separate TTS: a two-stage approach (generate text, then feed it to a separate TTS model) introduces latency, requires maintaining two separate models, and prevents end-to-end optimization for the text-speech relationship. Internal linguistic guidance makes speech generation a first-class capability of the unified model, trained under the same autoregressive objective as all other tasks. The text tokens serve as an explicit scaffold for the audio tokens, ensuring semantic accuracy while allowing the model to learn the acoustic realization jointly with the linguistic content.

Training Pipeline: From Tokenizer to Native Multimodal Model

The full training pipeline (Section 4.1, Figure 13, Table 7) consists of two phases and four stages:

Phase I: Modality-Specific Tokenizer Training

Visual tokenizer (dNaViT) is trained independently through Stage 1 (tokenization with RVQ and semantic reconstruction) and Stage 2 (de-tokenization with pixel decoder and refiner). Training uses approximately 50M images at arbitrary resolutions.
Audio tokenizer is trained independently through three stages (decoder warm-up, semantic-acoustic joint training, decoder fine-tuning). Training uses approximately 2.5M hours of audio data.

Phase II: Native Multimodality Training (Table 7)

Stage 1: Pre-Align — batch size 8192, sequence length 8K. Only codebook embeddings and DepthTransformer decoders are trainable; the language backbone and tokenizers remain frozen. Uses image-caption pairs for vision, pure audio/ASR/TTS/interleaved data for audio. Purpose: warm up the embedding and decoding components to align discrete tokens with the backbone's representational space before full end-to-end training.
Stage 2: Pre-training — batch size 8192, sequence length 8K. All components (backbone, embeddings, decoders) are trainable; tokenizers remain frozen. Extends data to include interleaved image-text data, OCR, grounding, fixed-resolution generation, pure text. This is the main phase where the model learns multimodal capabilities across understanding and generation.
Stage 3: Mid-training — batch size 1024, sequence length 32K. All components trainable. Introduces long chain-of-thought reasoning data for understanding, arbitrary-resolution feature training for visual generation, high-quality data, multi-turn dialogue. For visual generation, this stage applies cluster-based rebalancing (Section 4.3.2) to address distributional imbalance: images are encoded with SigLIP2, clustered via large-scale distributed K-Means using FAISS, aggressively deduplicated within high-density clusters, and reweighted via power-law rebalancing to increase sampling probability of sparse clusters.
Stage 4: Supervised Fine-Tuning (SFT) — batch size 128, sequence length 64K. All components trainable. Uses high-quality instruction data, reasoning data, arbitrary-resolution generation data, long pure text data. The SFT stage refines instruction-following capability and aligns the model with human preferences across all modalities.

Total training data: over 2 trillion tokens across all stages.

Why a staged pipeline: the pre-align stage addresses the fact that discrete visual and audio embeddings are "learned entirely from scratch" while the language model backbone starts from a pretrained state. Without this warm-up phase, the randomly initialized embeddings would present a training signal mismatch — the backbone would need to simultaneously learn to interpret the embedding space and use it for downstream tasks. The Pre-Align stage allows the embeddings to converge toward a meaningful organization before the backbone begins adapting to them, which the paper shows "substantially accelerates convergence" (Section 3.2.1).

Infrastructure (Section 5): the paper describes a custom pipeline parallelism strategy called VHalf to handle the heterogeneous compute patterns in multimodal training. The key insight is that the embedding layer and modality-specific loss modules (like DepthTransformer) have very different execution times than the standard LLM transformer layers, and this latency fluctuates based on the modality mix in each batch. The VHalf design:

Co-locates the embedding layer and modality loss modules on the same physical device in a V-shaped pipeline.
Creates a shared buffer at this "anchor" device so the loss modules can directly access the hidden states needed for RVQ multi-level token loss computation, eliminating cross-device communication.
Decouples the LLM head from the modality-specific loss modules, assigning it to a separate pipeline stage with a small number of LLM transformer layers.
Distributes the remaining LLM transformer layers evenly across the remaining devices.

The paper claims this configuration achieves "near-perfect load balance" and "fundamentally eliminates the high cross-stage communication overhead" typically required for boundary modules (Section 5.1).

4. Key Insights and Innovations

Innovation 1: Semantic Completeness as a Diagnostic Criterion — Not Just a Tokenizer, but a Sufficiency Standard

The most conceptually distinctive contribution of this paper is not the dNaViT architecture itself but the semantic completeness criterion (Section 2.2.1) that defines what it means for a discrete tokenizer to be adequate. Prior work on discrete multimodal modeling treated tokenizer quality as a matter of reconstruction fidelity (does the decoded image look like the original?) or downstream performance (does the model trained on these tokens achieve good accuracy?). The paper reframes the problem by introducing a formal conditional independence: the discrete tokens z are sufficient if, for any downstream query Q about the image, conditioning on z yields the same answer distribution as conditioning on the raw pixels I. This is a probabilistic sufficiency standard, not an architectural prescription or an empirical benchmark metric.

Why this matters: it converts the debate about continuous vs. discrete representations from an ideological argument into a testable hypothesis. The widespread belief that "compressing rich visual information into a finite codebook inevitably hinders representation capacity" (Section 1) implicitly assumes that discrete tokens cannot achieve semantic completeness — that the information loss from quantization necessarily creates a performance ceiling. The paper's innovation is to treat this as an empirical question rather than a premise, and to provide both a formal framework for evaluating it and experimental evidence that it is false under the right conditions (Section 3.2.1, Figure 9 showing discrete models approaching continuous parity with sufficient data). This is a fundamental reframing, not an incremental improvement: it shifts the field's burden of proof from "prove discrete tokens can work" to "identify what conditions are necessary for discrete tokens to work."

The criterion is also generative — it decomposes into discriminative invariance and generative sufficiency, two properties that prior work typically achieved with separate architectures (e.g., Janus used one encoder for understanding and a separate VQ-VAE for generation). The paper's claim that an SAE can satisfy both simultaneously under a single tokenizer is a substantive hypothesis about representation learning, not just an engineering convenience. The experiment in Section 3.2.3 showing that unified training produces lower generation loss than pure-generation training (while matching pure-understanding performance) provides indirect evidence for this claim — it suggests that the representations learned for understanding are not just compatible with generation but actively beneficial to it.

Innovation 2: The Performance Ceiling Is an Artifact of Insufficient Representation Capacity and Training Scale, Not Discretization

The paper's most empirically aggressive claim is that the historically observed gap between discrete and continuous visual modeling is not fundamental — it is a consequence of two addressable bottlenecks: (i) insufficient capacity in the pre-quantization encoder, and (ii) insufficient training data for the discrete embeddings. Prior discrete models (Chameleon, Emu3, early EVE variants) consistently underperformed continuous-projection models on understanding benchmarks, leading to a widespread assumption that discretization imposes an inherent upper bound. The paper's counterargument, supported by the experiments in Section 3.2.1 and Table 9, is that with an SAE providing semantically complete features and sufficient data scale (hundreds of billions of tokens), the discrete model achieves near-parity with its continuous counterpart.

This matters because it redirects the research trajectory for unified multimodal models. If the ceiling were truly inherent, then progress on discrete modeling would be fundamentally capped, and the field should invest in hybrid architectures or continuous representations. If the ceiling is instead a data-and-capacity problem, then the path forward is straightforward: build better SAEs, scale training, and optimize tokenizer training objectives. The paper's own results — LongCat-Next matching Qwen3-VL-A3B (a specialized continuous-projection model) on visual understanding while maintaining generation capability (Tables 1, 2, 3) — provide the first industrial-scale evidence for the latter interpretation. This is a negative result turned positive: the ceiling was never about discretization; it was about not having solved the representation and scaling problems that continuous models had already addressed.

The practical implication is significant. If discrete tokens are a drop-in replacement for continuous features (given sufficient tokenizer quality), then the entire ecosystem of LLM training infrastructure, optimization techniques, and serving frameworks becomes immediately applicable to multimodal models without the engineering complexity of hybrid architectures. This shifts the cost-benefit analysis for organizations building multimodal systems — the simplification gains from full discretization are no longer offset by a performance penalty.

Innovation 3: Residual Architecture Provides Structural Completeness — Information Recovery Without Reconstruction Supervision

The paper's analysis of intrinsic information recovery from residual vision encoders (Section 2.2.4, Equation 7, Figures 5 and Table 6) represents a genuinely underexplored insight about representation learning. The standard assumption — implicit in approaches like RAE that add explicit reconstruction objectives to semantic encoders — is that semantic training discards pixel-level information, and recovering that information for generation requires additional supervision. The paper demonstrates empirically that this assumption is overly pessimistic: even a randomly initialized ViT achieves strong reconstruction (PSNR 30.52, Table 6), and semantically trained encoders retain coarse structural information that can be recovered with a lightweight decoder.

The theoretical contribution is a structural argument for completeness: the identity mappings in residual connections ensure that early-layer information is additively preserved in the final representation, not overwritten by later semantic abstractions. Formally, zp = x0 + Σ Fl(xl-1), where x0 contains pixel-level information and each Fl adds semantic refinements. This means the representation inherently retains high mutual information with the original image, even without reconstruction supervision. The paper does not claim this is a proof — it's an empirical observation with a structural interpretation — but it provides a principled explanation for why SAE-based tokenization works for generation despite never being optimized for it.

This finding challenges the prevailing two-encoder paradigm (separate encoders for understanding and generation, as in Janus) by suggesting that a single encoder — with sufficient residual depth and appropriate training — can naturally support both. It also provides guidance for future tokenizer design: residual connections are not just an optimization tool for training deep networks; they are an architectural mechanism for preserving representational completeness across levels of abstraction.

Innovation 4: Understanding and Generation Are Synergistic Under a Single Objective — The "Conflict" Is an Artifact of Separate Objectives

The paper's most provocative finding about multimodal learning dynamics is that understanding and generation do not conflict when unified under a single next-token prediction objective — and may even be synergistic. The experiment in Section 3.2.3 (Figure 10) shows that a unified model trained on a 1:1 mixture of understanding and generation data achieves lower generation loss than a pure-generation model (by 0.02 despite having half the task-specific data) while matching the pure-understanding model's performance (within 0.006 loss). This directly contradicts the prevailing narrative — reinforced by the Janus architecture's explicit decoupling of understanding and generation encoders, and by the performance gaps in prior unified models (Table 3 showing Janus-Pro, Show-o2, OneCAT all substantially weaker on understanding than specialized models) — that these objectives compete for model capacity.

The mechanism is not fully explained, but the implication is clear: understanding tasks provide a rich training signal that transfers to generation (the model learns what images contain and can use that knowledge to decide what to produce), while generation tasks may provide less benefit to understanding. This asymmetrical synergy suggests that the "conflict" in prior work arose from having separate optimization objectives or separate representational spaces for the two tasks, forcing the model to learn two incompatible mappings rather than a single unified one. The DiNA framework collapses this by making both tasks instances of the same autoregressive process — understanding is image tokens → text tokens, generation is text tokens → image tokens — differing only in conditional prior, not in architecture or objective.

This is a conceptual reframing with practical consequences. If the conflict is an artifact of architectural choices rather than an inherent tension, then unified discrete models have a structural advantage over hybrid architectures that cannot be recovered by scaling either component independently. It also suggests that training on both understanding and generation data is not a zero-sum tradeoff but a positive-sum combination — each modality benefits from the other's supervision.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates across a wide range of benchmarks spanning three modalities. For visual understanding, the primary datasets include MMMU and MMMU-Pro (multi-discipline reasoning), MathVista and MathVision (mathematical reasoning), VisuLogic and BabyVision (visual logic), OCRBench, OCRBenchV2, DocVQA, and OmniDocBench (OCR and document understanding), ChartQA, CharXiv, and InfoVQA (chart and infographic), MMStar, RealWorldQA, and CountBench (general VQA), and OSWorld-G and ScreenSpot-V2 (GUI grounding). For visual generation, the evaluation uses GenEval (compositional alignment), DPG-Bench (prompt following), LongText-Bench (long-text understanding for T2I), WISE (world knowledge), TIFF and CVTG-2K (text rendering). For audio, the benchmarks include LibriSpeech, AISHELL-1/2, FLEURS, and WenetSpeech (ASR), SeedTTS (TTS), MMAU, VocalSound, TUT2017, and ClothoAQA (audio understanding), and OpenAudioBench (audio-to-text chat). For text, evaluation spans MMLU, MMLU-Pro, C-Eval, and CMMLU (general knowledge), Tau2-Bench and VitaBench (agentic tool use), and SWE-Bench and TerminalBench (coding). All results are reported on the respective standard test splits or evaluation protocols for each benchmark.
Base model(s). The primary model is LongCat-Next, instantiated on the LongCat-Flash-Lite A3B backbone — a Mixture-of-Experts architecture with 68.5B total parameters and an average of 3B activated parameters (range 2.9B–4.5B depending on context). This model was trained from scratch on over 2T tokens across all modalities. The paper also conducts ablation studies using a reduced setting with Qwen-7B as the language backbone (Section 3.2) due to full-scale computational costs. The choice of the A3B MoE architecture reflects the paper's emphasis on industrial-scale, deployment-ready multimodal modeling rather than proof-of-concept demonstrations at smaller scale.
Metrics. The paper uses modality-specific metrics throughout. For visual understanding, the primary metric is accuracy (%) on benchmark test sets, with some benchmarks using specific grading functions (e.g., MATH's released grading function for MathVista, Kimi-Audio-Evalkit with gpt-4o-2024-08-06 as judge for audio QA). For visual generation, metrics vary by benchmark: GenEval reports compositional alignment scores, DPG-Bench and LongText-Bench report prompt-following accuracy, WISE reports world knowledge scores, and TIFF/CVTG-2K report text rendering accuracy. For audio, Word Error Rate (WER, ↓) is used for ASR and TTS tasks, while accuracy (%) is used for audio understanding and audio-to-text chat. For text, accuracy is the primary metric for knowledge benchmarks and coding tasks, with specific protocol-defined scores for agentic benchmarks (e.g., avg@8 for Tau2-Bench).
Baselines. The paper compares against a comprehensive set of models grouped by capability. For unified omni-modal systems, the primary baseline is Qwen3-Omni-A3B-Instruct. For visual understanding, specialist MLLMs include InternVL3.5-A3B-Flash and Qwen3-VL-A3B-Instruct, plus GPT5-minimal and Gemini2.5-Flash-Lite as frontier baselines. For visual generation, baselines include unified multimodal models (Janus-Pro, Show-o2, OneCAT, Mogao, BAGEL, NEO-unify, Ovis-U1, Lumina, OmniGen2, UniWorld-V1, X-Omni, InternVL-U, BLIP3-o) and specialized T2I models (Emu-3.5, Qwen-Image 2507, Gemini 2.5 Flash Image, FLUX.1-dev, Seeddream 3.0). For audio, baselines include Gemini-3.1-Flash-Lite-preview, Gemini-2.5-Flash-Lite-preview, Qwen3-Omni-A3B-Instruct, MiMo-Audio, Kimi-Audio, and Step-Audio-2-mini. For text, baselines include Kimi-Linear-48B-A3B, Qwen3-Next-80B-A3B-Instruct, and Qwen3-Omni-A3B-Instruct. Values from baselines marked with * in the paper's tables are sourced from public reports rather than reproduced by the authors.
Generation budget / compute accounting. For the ablation studies in Section 3.2, compute is measured in training tokens (e.g., 0.1B, 5B, 300B tokens for pre-training/mid-training/SFT stages). The pre-alignment experiments in Section 3.2.1 control for data scale by comparing models trained on identical token counts. For the RL experiments, generation budget is implicit in the number of rollout samples (4 rollouts per instance for data filtering). The main results in Tables 1–5 do not report per-sample inference cost — they compare final benchmark performance after full training, with all models evaluated under their standard inference protocols (greedy decoding or best-of-N as specified by each baseline's documentation).
Cross-validation / statistical protocol. The ablation study in Section 3.2.1 uses a controlled comparison protocol: all modules except the frozen language backbone and vision encoders are optimized under identical settings, with captioning loss as a proxy metric. The discrete vs. continuous comparison in Figure 9 reports training loss curves and downstream benchmark scores under carefully matched token budgets. For the RL experiments (Section 6.1, Appendix 9.2), data is filtered by performing 4 rollouts with the base model and removing instances that are either entirely correct or entirely incorrect, retaining approximately 30K samples for understanding and 40K for generation. The paper does not describe formal statistical significance testing (confidence intervals, error bars, or multiple-run variance) for the main benchmark results in Tables 1–5. For audio evaluation, standardized protocols from Kimi-Audio-Evalkit are adopted to ensure reproducibility.

Main Quantitative Results

Visual Understanding

Headline results (Table 1). LongCat-Next achieves competitive or superior performance against specialized visual understanding models across most benchmarks, while substantially outperforming the primary omni-modal baseline (Qwen3-Omni-A3B-Instruct) on reasoning and OCR tasks. On MMMU, LongCat-Next scores 70.6 vs. Qwen3-Omni's 69.1* and Qwen3-VL-A3B's 74.2*. On MMMU-Pro, it scores 60.3 vs. 57.0* for Qwen3-Omni and 60.4* for Qwen3-VL. On MathVista (mini), the margin is substantial: 83.1 vs. 75.9* (Qwen3-Omni) and 80.1* (Qwen3-VL). On MathVision, LongCat-Next achieves 64.7 vs. 56.3* (Qwen3-Omni) and 60.2* (Qwen3-VL). These results are notable because Qwen3-VL-A3B is a specialist MLLM optimized purely for visual comprehension, with no generation or audio capability — LongCat-Next matches or exceeds it on several reasoning benchmarks while being a full omni-modal model.

STEM and reasoning benchmarks (Table 1, top section). On VisuLogic, LongCat-Next achieves 29.4, outperforming all listed baselines including InternVL3.5-A3B-Flash (28.4) and Gemini2.5-Flash-Lite (26.1). On BabyVision, it scores 14.4, surpassing InternVL3.5-A3B-Flash (11.3) and Qwen3-Omni (11.9) but trailing Gemini2.5-Flash-Lite (19.6) and Qwen3-VL-A3B (16.0). The paper notes these gains on VisuLogic and BabyVision as "surprisingly strong" given that the model was "not specifically optimized towards these datasets," interpreting them as evidence of "robust, emergent generalization capabilities for decoding complex visual logic puzzles."

OCR and document understanding (Table 1, middle section). LongCat-Next shows particularly strong performance on text-heavy benchmarks. On OmniDocBench (English), it achieves a lower score of 0.152 (↓ indicates lower is better for this metric), substantially outperforming Qwen3-Omni (0.289), GPT5-minimal (0.174*), and Qwen3-VL (0.183*). On the Chinese variant (OmniDocBench-zh), it scores 0.226 vs. 0.406 (Qwen3-Omni), 0.312 (Gemini2.5-Flash-Lite), and 0.253* (Qwen3-VL). On CharXiv (reasoning questions), LongCat-Next achieves 60.1, the best among all models, while on CharXiv (descriptive questions) it scores 89.9, second only to Gemini2.5-Flash-Lite (91.2). On ChartQA, it achieves 88.0, outperforming all baselines including specialist MLLMs. On DocVQA (test), it scores 94.2, matching InternVL3.5-A3B-Flash (94.2*) and approaching Qwen3-VL (95.0*). On OCRBench, it scores 86.5, behind the specialist MLLMs (InternVL3.5 at 88.0*, Qwen3-VL at 90.3*) but ahead of Qwen3-Omni (85.4*) and Gemini2.5-Flash-Lite (84.8).

General VQA and GUI (Table 1, bottom sections). On general-domain benchmarks, LongCat-Next performs competitively but slightly behind the strongest specialist models. On MMStar, it scores 69.3 vs. Qwen3-VL-A3B's 72.1 and Gemini2.5-Flash-Lite's 74.93. On RealWorldQA, it scores 72.0, surpassing Gemini2.5-Flash-Lite (70.5) but trailing Qwen3-VL (73.7). On CountBench, it scores 82.1, behind the top performers (Gemini2.5 at 90.0, Qwen3-VL at 90.6). On GUI benchmarks, it achieves 58.3 on OSWorld-G and 88.3 on ScreenSpot-V2, maintaining "highly competitive performance, maintaining parity with Qwen3-Omni-A3B-Instruct and Qwen3-VL-A3B-Instruct."

Comparison with unified models (Table 3). When compared against other unified multimodal models (those combining understanding and generation), LongCat-Next establishes a dominant position. On MMMU, its 70.6 dwarfs the next-best unified model (NEO-unify at 68.9, BAGEL at 55.3). On MathVista, its 83.1 far exceeds BAGEL (73.1) and Ovis-U1 (69.4). On OCRBench, its 86.5 is competitive with Ovis-U1 (88.3) and ahead of all others. On DocVQA, its 94.2 matches BAGEL (92.2) and NEO-unify (91.6). On generation metrics (Table 3, right columns), LongCat-Next's 84.44 on GenEval and 84.66 on DPG are competitive with the best unified models (NEO-unify at 85/86.71, Show-o2 at 76/86.14). On text rendering (TIFF, CVTG), LongCat-Next achieves 82.85/84.38 and 76.36 respectively, substantially outperforming nearly all unified baselines. This combined performance — strong understanding AND strong generation — is the paper's central claim about unified models, and Table 3 provides the most direct evidence that prior unified models consistently sacrificed one capability for the other, while LongCat-Next achieves both simultaneously.

Visual Generation

Comparison with specialized T2I models (Table 2). Despite being a unified model with a fraction of the parameters of specialized T2I systems, LongCat-Next achieves competitive results. On GenEval, it scores 84.44, surpassing FLUX.1-dev (66.00*), Emu-3.5 (72.67), Gemini 2.5 Flash Image (79.67), and matching the range of Qwen-Image 2507 (87.00*). On DPG-Bench, its 84.66 is competitive with Seeddream 3.0 (94.31*) and Qwen-Image (88.32*). On LongText-Bench, for English it scores 93.15 vs. Emu-3.5's 97.60* and Qwen-Image's 94.30*; for Chinese, it scores 89.08 vs. Emu-3.5's 92.80* and Qwen-Image's 94.60*. On WISE (world knowledge), it scores 57.00, behind Gemini 2.5 Flash Image (76.27) but competitive with Emu-3.5 (57.64) and FLUX.1-dev (50.00*). On text rendering benchmarks, TIFF scores are 82.85/84.38 (vs. Gemini's 90.53/90.80 and Qwen-Image's 86.10*/86.80*), and CVTG-2K is 76.36 (vs. Emu-3.5's 91.23* and Qwen-Image's 82.88*). The paper emphasizes that these results are achieved with a "significantly smaller model" and that the unified architecture's tight integration of language understanding with image generation is responsible for the strong text rendering and compositional performance — areas where prior unified models have historically struggled.

Comparison with unified multimodal models on generation (Table 3, right columns). LongCat-Next excels at long-text understanding and text rendering compared to other unified models. On LongText-Bench (English/Chinese), its 93.15/89.08 vastly exceeds the next-best unified model (NEO-unify at 91.40/75.50) and completely dominates models like Janus-Pro (1.90/0.60), Ovis-U1 (3.00/5.10), Lumina (43.70/4.70), and OmniGen2 (56.10/5.90) — these near-zero scores for older unified models highlight the historical difficulty of text rendering in discrete autoregressive frameworks. On TIFF, LongCat-Next's 82.85/84.38 is competitive with InternVL-U (74.90/73.90) and OmniGen2 (74.70/72.00). On CVTG-2K, its 76.36 outperforms InternVL-U (62.30), Lumina (59.00), and BAGEL (35.60). On GenEval and DPG-Bench, LongCat-Next's scores (84.44, 84.66) are in the upper range of unified models, comparable to NEO-unify (85, 86.71), Show-o2 (76, 86.14), and Bagel (89, 85.07). The paper attributes the gains on text-heavy generation to the unified architecture, which "enables stronger semantic planning before synthesis, [so] the model better preserves textual intent in complex scenarios such as multi-object composition and text rendering, where prior unified models often struggle."

Audio

Headline ASR and TTS results (Table 4, top two sections). LongCat-Next demonstrates strong but not dominant ASR performance. On LibriSpeech test-clean, it achieves WER 1.63 vs. Qwen3-Omni's 1.22* (the best), MiMo-Audio's 2.47, Kimi-Audio's 1.28*, and Step-Audio-2-mini's 1.33*. On LibriSpeech test-other, it scores 3.42 vs. Qwen3-Omni's 2.48* and Kimi-Audio's 2.42*. On Chinese ASR benchmarks, LongCat-Next shows a clearer advantage over Gemini models: on AISHELL-1, it achieves 1.47 vs. Gemini-3.1-Flash-Lite's 6.00 and Gemini-2.5-Flash-Lite's 11.64, though still behind Qwen3-Omni (0.84*). On WenetSpeech (test-meeting and test-net), LongCat-Next's 8.19 and 5.98 substantially outperform both Gemini models (20.37/16.15 and 23.04/24.83 respectively). On TTS, the evaluation uses SeedTTS: LongCat-Next achieves WER 1.90 (Chinese) and 1.89 (English), behind Qwen3-Omni (1.07*/1.39*) but competitive with MiMo-Audio (1.96*/5.37*), Kimi-Audio (13.46/29.45), and Step-Audio-2-mini (2.13*/3.18*). The TTS WER metric measures how accurately a separate ASR system can transcribe the synthesized speech — lower WER indicates more intelligible, higher-quality synthesis.

Audio understanding and chat (Table 4, bottom two sections). LongCat-Next achieves state-of-the-art performance on audio understanding benchmarks among the compared models. On MMAU, it scores 76.40, behind Qwen3-Omni (78.20) but ahead of Gemini-2.5-Flash-Lite (74.80), Gemini-3.1-Flash-Lite (71.70), and MiMo-Audio (75.80). On ClothoAQA, it scores 73.45, second only to Qwen3-Omni (75.16*) and ahead of all others. On TUT2017 (acoustic scene classification), it achieves 43.09, best among the non-Kimi models (Kimi-Audio scores 65.25*). On VocalSound, it scores 85.91, behind MiMo-Audio (87.94), Kimi-Audio (94.85*), and Qwen3-Omni (91.59*). On audio-to-text chat benchmarks, LongCat-Next is highly competitive: on AlpacaEval (86.83), it trails Qwen3-Omni (90.10) and Gemini-2.5 (89.05) but leads Gemini-3.1 (62.56) and Kimi-Audio (78.74). On ReasoningQA, it scores 87.52, essentially tied with Qwen3-Omni (87.62) and substantially ahead of all other models (Gemini-2.5 at 74.06, MiMo-Audio at 75.34). On TriviaQA and WebQuestions, LongCat-Next scores 67.60 and 69.10, competitive with Qwen3-Omni (76.60/75.90) and Gemini-2.5 (66.90/70.00).

Serial vs. parallel generation semantics (Section 3.2.4). The paper evaluates whether the model's parallel generation strategy (simultaneous text and audio token prediction) can match the semantic accuracy of serial generation (text first, then audio). On LlamaQuestions, parallel generation achieves 79.33 vs. serial's 81.67 — a gap of 2.34 points. On ReasoningQA, parallel achieves 74.95 vs. serial's 80.30 — a gap of 5.35 points. The paper characterizes these gaps as "slight" and evidence that "parallel generation can preserve response quality while offering improved efficiency." The larger gap on ReasoningQA suggests that complex reasoning tasks are more sensitive to the text-audio alignment strategy, though both modes remain functional.

Text

Agentic tool use and coding (Table 5, top sections). LongCat-Next shows a dramatic advantage in practical execution tasks. On Tau2-Bench (agentic tool use), it achieves exceptional scores: Airline 56.50 (avg@8) vs. Kimi-Linear-48B's 44.00, Qwen3-Next's 45.5*, and Qwen3-Omni's 27.00; Retail 73.68 vs. Kimi's 18.86, Qwen3-Next's 57.3*, and Qwen3-Omni's 40.80; Telecom 62.06 vs. Kimi's 15.68, Qwen3-Next's 13.2*, and Qwen3-Omni's 4.39. These margins — particularly the Telecom score where LongCat-Next achieves ~4–15× the performance of the next-best model — are the paper's strongest text-domain results, demonstrating robust capability in "dynamic environment navigation, complex tool dependency resolution, and real-world codebase manipulation." On SWE-Bench, LongCat-Next scores 43.0, outperforming Kimi-Linear-48B (32.80) and Qwen3-Next (37.60). On TerminalBench, it scores 18.75, slightly behind Kimi-Linear-48B (20.00) but ahead of Qwen3-Next (15.19). On VitaBench, it scores 5.80 (avg@4), matching Qwen3-Next (5.80).

General knowledge (Table 5, bottom section). LongCat-Next maintains strong foundational language capabilities, successfully mitigating the "multimodal tax." On MMLU, it scores 83.95, outperforming Kimi-Linear-48B (79.91) but trailing Qwen3-Next (89.28) and Qwen3-Omni (87.10). On MMLU-Pro, it scores 77.02 vs. Kimi's 67.22, Qwen3-Next's 82.93, and Qwen3-Omni's 79.89. On C-Eval, it scores 86.80 vs. Kimi's 78.48, Qwen3-Next's 90.91, and Qwen3-Omni's 88.50. On CMMLU, it scores 82.13 vs. Kimi's 76.26, Qwen3-Next's 86.50, and Qwen3-Omni's 85.76. The paper acknowledges that "the strictly text-optimized Qwen3-Next-80B-Instruct naturally sets the upper bound in these traditional academic tests" but emphasizes that LongCat-Next "maintains a highly resilient cognitive baseline, ensuring its multimodal outputs are anchored by deep logical reasoning." The key comparison is against Qwen3-Omni (also a multimodal model): LongCat-Next trails by 3–4 points on MMLU/C-Eval/CMMLU but leads on Tau2-Bench and SWE-Bench, suggesting a different capability profile where practical execution is prioritized over academic knowledge benchmarks.

Ablation Studies and Robustness Checks

Discrete vs. continuous modeling under matched conditions (Section 3.2.1, Figure 9). The paper conducts a systematic ablation comparing discrete (dNaViT) and continuous (NaViT) versions of the same architecture under controlled data scales. Under limited data (0.1B pre-training tokens, 4B mid-training+SFT tokens), the discrete model substantially underperforms the continuous model: OCRBench 598 vs. 779, DocVQA 78.0 vs. 88.2, MMMU 44.8 vs. 49.8, MathVista 47.3 vs. 59.6. However, as data scales to 300B mid-training+SFT tokens, the gap closes dramatically: OCRBench 844 (discrete) vs. 858 (continuous), DocVQA 96.0 vs. 96.0, MMMU 60.0 vs. 58.0, MathVista 74.0 vs. 75.0. The paper interprets this as evidence that "discrete visual embeddings are learned entirely from scratch" and "require more data to reach comparable performance," but with sufficient scale, the performance ceiling disappears. The continuous model achieves 58.0 on MMMU while the discrete achieves 60.0 at the highest data scale — actually slightly better, though the paper does not overinterpret this reversal.

Pre-Buffer module (Section 3.2.1, Figure 9, Experiment III vs. II). Adding a lightweight Pre-Buffer (single-layer FFN) after the multi-level embedding summation substantially accelerates convergence and improves expressiveness. Under the discrete setting with Pre-Buffer (Exp III), the pre-alignment training loss drops faster, and downstream performance improves: OCRBench 776 vs. 598 (Exp II, no Pre-Buffer), DocVQA 88.9 vs. 78.0, ChartQA 80.6 vs. 71.6, MathVista 56.1 vs. 47.3. The paper hypothesizes that the Pre-Buffer addresses "insufficient re-encoding after the sum-up operation on multi-level embeddings," providing a necessary remapping step to recover structured information after additive combination.

Training data scale (Section 3.2.1, Figure 9, Experiments IV through VII). The paper compares three data scales for the discrete model: 5B pre-training + 4B mid-training+SFT (Exp IV), 5B + 4B without Pre-Buffer (Exp V), and 5B + 300B (Exp VII). The results show that scaling mid-training+SFT data from 4B to 300B produces the largest gains: MMMU improves from 48.0–49.8 (Exps IV/V) to 60.0 (Exp VII), OCRBench from 705–720 to 844, MathVista from 54.2–56.7 to 74.0. The paper concludes that "unlike the continuous setting, discrete visual embeddings... require more data to reach comparable performance" and that "with the appropriate data scaling and improved quality, the full potential of discrete representations can be unlocked."

Information recovery from different encoder architectures (Section 3.2.2, Figure 5, Table 6). The paper compares reconstruction fidelity across four encoder types, all paired with a lightweight ViT-based decoder and measured on PSNR, SSIM, and rFID. The key finding is that a randomly initialized ViT-B/16 achieves the best reconstruction (PSNR 30.52, SSIM 0.887, rFID 0.5847), outperforming pretrained ViT-B/16 (PSNR 21.86, SSIM 0.581, rFID 0.8850) and pretrained ResNet-50 (PSNR 20.88, SSIM 0.509, rFID 0.4619). QwenViT without the merger module achieves PSNR 18.16 and SSIM 0.46; introducing the merger (which downsamples from 14× to 28×) leads to "noticeable degradation" (qualitative comparison, Figure 5). The paper hypothesizes that random weights produce "noise-like signals, which are easier for the decoder to denoise during reconstruction," while residual pathways "may still preserve a portion of the original pixel-level information" regardless of weight values.

Understanding-generation interaction under a unified objective (Section 3.2.3, Figure 10). The paper trains three models: Pure-Understanding (100B tokens understanding data), Pure-Generation (100B tokens generation data), and Unified (50B from each, total 100B). Under comparable token counts, the Unified model achieves a generation loss 0.02 lower than the Pure-Generation model (despite having half the task-specific data), while its understanding loss is only 0.006 higher than the Pure-Understanding model. The paper interprets this as evidence that "generation does not compromise understanding, whereas understanding actively enhances generation" — an asymmetrical synergy that contradicts the prevailing narrative of conflict between these objectives. The loss curves are proportionally scaled to account for the halved task-specific data in the Unified model, with both the original unscaled and scaled versions shown for reference.

Parallel vs. serial audio generation semantics (Section 3.2.4). The paper evaluates whether the random-delay unified training paradigm successfully transfers semantic fidelity from serial to parallel generation. Under audio-to-audio evaluation on LlamaQuestions, parallel achieves 79.33 vs. serial's 81.67. On ReasoningQA, parallel achieves 74.95 vs. serial's 80.30. Both results use the same model checkpoint, differing only in the delay applied at inference time. The 2.34-point gap on LlamaQuestions and 5.35-point gap on ReasoningQA are presented as evidence that the model "adaptively learns to align audio and text semantics within the context," though the larger gap on reasoning tasks suggests that complex semantic reasoning is more sensitive to the text-audio generation strategy.

Modality-agnostic MoE training dynamics (Section 3.2.5, Figure 11). Comparing MoE layer behavior before and after multimodal training: (1) expert specialization emerges without architectural enforcement, with subsets of experts developing clear preferences for vision, audio, or text tokens; (2) routing patterns become more structured and stable; (3) the average number of routed tokens per expert increases from 507.1 to 584.6, indicating more efficient capacity utilization. The paper presents this as evidence that "multimodal learning not only induces functional differentiation but also effectively expands the model's capacity usage." No ablation is provided comparing modality-agnostic MoE to a hypothetical modality-aware MoE.

Platonic representation analysis (Section 3.2.6, Figure 12). Using t-SNE visualization of 50,000 token-level embeddings from image-text inputs, the paper compares how different model families organize cross-modal representations. Qwen2.5-VL (non-native) produces "largely separated modality clusters" — text and vision tokens occupy distinct regions of the embedding space. Qwen3.5 (data-native training but not architecturally native) exhibits "only partial cross-modal alignment." LongCat-Next shows "interwoven embeddings across visual and textual tokens," with modality tokens forming an interlaced distribution rather than separate clusters. The paper interprets this as evidence that "a native discrete multimodal design more effectively promotes a unified semantic space, where features from different modalities behave like multilingual expressions of the same underlying concepts." An additional finding is that the frozen quantized mapping from the visual tokenizer "exhibits negligible performance degradation when adapted to the LongCat architecture," suggesting inherent alignment between the SAE's discrete semantic space and the language representation space.

Discrete quantization strategy comparison (Appendix 9.3, Figure 19). Using feature reconstruction loss as a proxy metric, the paper compares vanilla VQ, two-stage RVQ, and eight-stage RVQ. Two-stage RVQ slightly outperforms vanilla VQ. Eight-stage RVQ shows significantly lower reconstruction loss, "demonstrating that the residual mechanism and compositionality of RVQ are essential for achieving discrete quantization with minimal information loss." The paper adopts 8-stage RVQ as the default, based on empirical validation that it "achieves sufficiently low information loss without imposing excessive computational overhead."

Reinforcement learning for understanding and generation (Section 6.1, Appendix 9.2, Table 9, Figure 17). Using a Qwen-7B backbone for computational tractability, the paper applies GRPO-based RL to both image understanding and image generation. For understanding (Table 9, top), RL improves scores across most benchmarks: MMMU 64.22 → 66.45, MMMU-Pro 51.27 → 53.58, MathVista 80.30 → 81.90, MathVision 49.28 → 53.52, MMStar 66.33 → 71.13, RealWorldQA 66.01 → 72.54, MMVP 73.33 → 74.66, ChartQA 88.56 → 92.08, AI2D 81.57 → 85.13. For OCR, OmniDocBench-zh improves (0.187 → 0.169) while OmniDocBench-en slightly degrades (0.256 → 0.266). For generation (Table 9, bottom, GenEval), RL improves Overall score 83.94 → 87.33, with the largest gains on counting (71.25 → 78.75, +7.50), position (74.75 → 81.50, +6.75), and color attributes (72.19 → 76.75, +4.56). Figure 17 shows steadily increasing reward curves across all dimensions during RL training.

Training-inference mismatch and entropy explosion in RL (Section 6.1, Appendix 9.1, Figure 18). The paper identifies a critical failure mode during discrete RL training: as training progresses, policy entropy and training-inference mismatch increase simultaneously, creating a "positive feedback loop that exacerbates the explosive growth of entropy" as noisy, garbled tokens enter the training data and their sampling probability is amplified. The paper shows that token-level importance sampling corrections (Truncated Importance Sampling, Multiple Importance Sampling) "merely delayed the onset of entropy explosion by a few training steps without addressing the fundamental problem." The proposed solution — sequence-level filtering using both entropy-based and per-token probability difference thresholds — is shown to "effectively prevent noisy data from entering the training process." The probability difference filter flags entire sequences when any single token exhibits |π_sampler − π_actor| > δ, targeting the phenomenon where "anomalous tokens... have a catastrophic effect on overall sequence quality." Figure 18 demonstrates the positive correlation between entropy explosion, gradient norm surges, training-inference policy divergence, and reward decline.

Visual de-tokenizer components (Appendix 9.6, Figure 25). Qualitative visualization shows the effect of the pixel decoder versus the pixel decoder plus refiner. The pixel decoder alone recovers semantic content (layout, objects, text) but produces "overly smooth" images. Adding the refiner "plays a critical role in faithfully recovering the original image at the semantic level," restoring fine-grained details and perceptual sharpness. The paper notes that "within the framework of LLM autoregression, the predicted discrete tokens inherently encode semantic content, such as the layout and structural elements of the image," and that these tokens "demonstrate superior performance in OCR tasks, as they inherently contain semantically complete information."

Critical Assessment

Claim 1: Discrete modeling can achieve performance parity with continuous representations given sufficient representation capacity, appropriate quantization, and adequate training scale.

What was tested. The ablation in Section 3.2.1 (Figure 9) directly compares discrete and continuous versions of the same architecture under matched data conditions, using a Qwen-7B backbone with frozen vision encoders. At 300B tokens of mid-training+SFT data, the discrete model achieves OCRBench 844 vs. continuous 858, DocVQA 96.0 vs. 96.0, MMMU 60.0 vs. 58.0, MathVista 74.0 vs. 75.0 — essentially parity. The full LongCat-Next model (using the A3B backbone) achieves competitive performance with Qwen3-VL-A3B (a continuous-projection specialist) across visual understanding benchmarks in Table 1, winning on some (MathVista 83.1 vs. 80.1*, MathVision 64.7 vs. 60.2*) and trailing on others (MMMU 70.6 vs. 74.2*, MMStar 69.3 vs. 72.1, OCRBench 86.5 vs. 90.3*).

Assessment. The ablation provides convincing evidence that the discrete-continuous gap closes with sufficient data, but several caveats apply. First, the "continuous baseline" in the ablation is NaViT with the same SAE encoder — this is a particular type of continuous representation (continuous features from a semantically aligned encoder), not the most common continuous approach (which uses a learned projector from a vision encoder to the LLM's embedding space). It's possible that other continuous approaches would show a different scaling profile. Second, the full-scale comparison against Qwen3-VL-A3B is not a controlled ablation — the two models differ in backbone architecture, training data composition, and training recipes. LongCat-Next winning on some benchmarks and losing on others does not cleanly isolate the effect of discretization. The paper would be strengthened by a direct full-scale comparison between LongCat-Next with discrete dNaViT and LongCat-Next with continuous NaViT (the natural extension of the Qwen-7B ablation), but this experiment is not reported — likely due to the computational cost of training a full-scale continuous variant.

Conditions. The claim holds for the specific SAE choice (Qwen2.5-ViT), the specific quantization strategy (8-stage RVQ with semantic reconstruction loss), and the specific data scale (2T+ tokens total, with 300B+ in mid-training+SFT). The paper does not establish that different SAEs or different quantization approaches would achieve parity at different scales. The finding that discrete embeddings "require more data to reach comparable performance" (Section 3.2.1) means the parity claim is conditional on sufficient data scale — at lower data budgets, the gap persists. Organizations with limited training budgets may still find continuous approaches preferable.

Claim 2: LongCat-Next reconciles the traditionally competing objectives of understanding and generation, achieving strong performance on both simultaneously without the tradeoffs observed in prior unified models.

What was tested. Table 3 provides the most direct evidence: LongCat-Next scores 70.6 on MMMU (understanding) while achieving 84.44 on GenEval and 84.66 on DPG (generation) — simultaneously competitive with or exceeding all other unified models on both axes. The ablation in Section 3.2.3 (Figure 10) shows that a unified model trained on a 1:1 mix of understanding and generation data achieves lower generation loss than a pure-generation model and near-identical understanding loss to a pure-understanding model. Prior unified models in Table 3 show clear tradeoffs: Janus-Pro has strong generation (GenEval 80) but weak understanding (MMMU 41.0); NEO-unify has strong understanding (MMMU 68.9) but its generation is weaker than LongCat-Next on text rendering. Section 3.1.4 shows that LongCat-Next maintains strong text capabilities (MMLU 83.95, SWE-Bench 43.0), addressing the "multimodal tax."

Assessment. The evidence for reconciliation is strong in the context of unified models — LongCat-Next is clearly the best-performing unified model on the combined understanding-generation frontier. However, the claim that understanding and generation "do not conflict" under a single objective requires careful qualification. The ablation in Figure 10 uses a relatively small Qwen-7B backbone trained on 100B tokens. At this scale, the unified model shows 0.006 higher understanding loss than the pure-understanding model — a small but real gap. It's unclear whether this gap would grow, shrink, or remain at larger scales. Additionally, the generation quality of LongCat-Next, while strong for a unified model, still trails specialized T2I models on several benchmarks (WISE: 57.00 vs. Gemini 2.5 Flash Image's 76.27; TIFF: 82.85/84.38 vs. Gemini's 90.53/90.80; CVTG-2K: 76.36 vs. Emu-3.5's 91.23*). The claim is about reconciling objectives within a unified model, not about matching the best specialist on each task — but the practical interpretation of "strong performance on both" depends on whether these remaining gaps matter for the intended deployment context.

Missing experiment. The paper never trains a pure-understanding version of LongCat-Next at full scale to compare against the unified version. Without this, it's impossible to determine whether the unified model's understanding performance (e.g., MMMU 70.6) is lower than what a pure-understanding LongCat-Next would achieve. The ablation at Qwen-7B scale suggests the gap would be small, but this has not been verified at the A3B scale. Similarly, no pure-generation LongCat-Next is trained at full scale to establish the upper bound on generation quality within this architecture family.

Claim 3: LongCat-Next breaks the historical performance ceiling of discrete visual modeling, achieving competitive results with specialized models.

What was tested. Table 1 directly compares LongCat-Next against Qwen3-VL-A3B (a dedicated vision-language model), InternVL3.5-A3B-Flash, and several frontier models. LongCat-Next wins on MathVista, MathVision, VisuLogic, ChartQA, and OmniDocBench; is competitive on MMMU, DocVQA, and RealWorldQA; and trails on MMStar, CountBench, and OCRBench. The overall pattern is that LongCat-Next is stronger on reasoning-heavy and text-heavy benchmarks and somewhat weaker on general VQA and fine-grained counting — neither universally better nor universally worse than specialist models.

Assessment. The paper's framing is careful: it does not claim to surpass all specialist models, but to have "broken the performance ceiling" — meaning discrete models are no longer categorically inferior. The evidence supports this narrower claim. Prior discrete models (Table 3) typically scored 40-55 on MMMU, while LongCat-Next achieves 70.6. The gap between LongCat-Next and the best specialist (Qwen3-VL-A3B) is generally small (<5 points on most benchmarks), whereas the gap between prior discrete models and specialists was often 20-30 points. This is a genuine breakthrough.

Caveats. The historical ceiling was established using older discrete tokenization methods (VQ-VAE, vanilla VQ) and smaller training budgets. The paper's contribution is showing that better tokenizers (SAE + RVQ) plus more training closes the gap, but it's not self-evident that this "breaks" a fundamental ceiling rather than simply applying more resources to the same problem. A skeptic could argue that the ceiling was never about discretization per se but about the specific implementation choices made by prior work — and that continuous models would similarly underperform if given poor encoders and limited data. The paper's ablation in Figure 9 partially addresses this by controlling for data scale and showing convergence, but the full-scale comparison against Qwen3-VL-A3B does not isolate the continuous-vs-discrete variable.

Claim 4: LongCat-Next is an industrial-strength unified foundation model that excels at seeing, painting, and talking.

What was tested. The comprehensive evaluation across Tables 1-5 covers visual understanding (12+ benchmarks), visual generation (7 benchmarks), audio (14+ benchmarks across ASR, TTS, understanding, and chat), and text (10+ benchmarks). The model performs competitively or better than comparable-scale baselines across all modalities.

Assessment. The evidence for industrial-strength capability is robust in breadth but leaves some questions about depth. On visual understanding, LongCat-Next is clearly competitive with the best models at comparable scale. On visual generation, it's the best unified model but trails specialized T2I systems on several metrics. On audio, it's strong but not dominant — Qwen3-Omni leads on several ASR and understanding benchmarks. On text, it excels at practical execution tasks (Tau2-Bench, SWE-Bench) while being slightly behind on academic knowledge benchmarks.

Weaknesses in the evidence. (1) All results are from a single model family (LongCat-Flash) with a single tokenizer design (Qwen2.5-ViT as SAE). There is no evidence that the approach would work with different backbones or different SAE choices. (2) The paper does not report inference latency or memory consumption for multimodal generation — crucial metrics for industrial deployment. (3) The difficulty estimation cost for the vision tokenizer (50M images for training) and audio tokenizer (2.5M hours) represents a substantial upfront investment that is not factored into any efficiency comparison. (4) The paper does not evaluate on video understanding or generation, despite these being increasingly important industrial use cases. (5) No human evaluation is reported for generative quality — all metrics are automated benchmarks, which may not capture user-perceived quality differences.

Missing experiments and analyses that would strengthen the paper

Direct full-scale discrete vs. continuous comparison. The most important missing experiment is training LongCat-Next with both discrete (dNaViT) and continuous (NaViT) vision representations at the full A3B scale, matched for total training compute. This would directly test whether the discrete approach incurs any performance penalty at industrial scale, rather than relying on extrapolation from the Qwen-7B ablation. The paper implicitly acknowledges this gap by not claiming that discrete is better than continuous — only that it's not categorically worse.

Ablation on number of RVQ levels. The paper adopts 8-level RVQ based on a feature reconstruction loss proxy (Appendix 9.3, Figure 19), but never ablates downstream performance vs. number of levels. Would 4-level RVQ achieve similar understanding but worse generation? Would 12-level RVQ improve generation but waste capacity on understanding? The additive encoding plus DepthTransformer design handles the multi-level structure efficiently, making this a practical design choice that deserves direct evaluation.

Ablation on SAE choice. The paper adopts Qwen2.5-ViT as the SAE and notes that other candidates (MoonViT, AIMv2) could work. A comparison of at least one alternative SAE on downstream performance would validate the claim that any semantically complete encoder can serve this role, rather than the success being specific to Qwen2.5-ViT's training recipe.

Evaluation of generation diversity. All generation metrics (GenEval, DPG, TIFF, CVTG-2K) measure alignment between prompts and generated images, but none measure diversity — whether the model produces varied outputs for the same prompt or collapses to a few modes. Mode collapse is a known issue with autoregressive image generation, and its absence or presence in LongCat-Next is not evaluated.

Inference cost analysis. The paper emphasizes infrastructure-friendliness and industrial deployment, but provides no measurements of tokens-per-image (required for understanding), generation time (for image or speech synthesis), or GPU memory consumption during inference. These are essential metrics for evaluating "industrial-strength" claims.

Controlled study of the multimodal tax. The paper claims to have mitigated the multimodal tax, but the evidence is indirect — comparing LongCat-Next's text scores against Qwen3-Omni's, where the two models differ in backbone architecture, training data, and training recipe. A controlled ablation would train LongCat-Next with and without multimodal data (matched for total tokens) and compare text benchmark performance, isolating the effect of multimodal training on language capability.

Summary of evidential strengths and weaknesses

Strengths. The paper provides unusually comprehensive evaluation across three modalities with 30+ benchmarks. The ablation studies in Section 3.2 address key mechanistic questions about the discrete-continuous gap, the Pre-Buffer, data scaling, encoding strategies, and understanding-generation interaction. The RL experiments in Section 6.1 demonstrate that discrete representations integrate naturally with existing LLM RL methods. The representation analysis (Platonic hypothesis, MoE training dynamics, information recovery from residual architectures) provides theoretical grounding for the empirical results.

Weaknesses. Many of the paper's strongest claims rely on a single model family with a single SAE choice. Several key ablations are performed only at reduced (Qwen-7B) scale, leaving uncertainty about whether the findings transfer to the full A3B model. The paper does not report confidence intervals or multiple-run variance for any benchmark results. Missing experiments on inference cost, generation diversity, and direct full-scale continuous-vs-discrete comparison limit the strength of the "no performance ceiling" and "industrial-strength" claims. The paper's heavy reliance on automated benchmarks without human evaluation is a gap for generative quality assessment.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Gains

The assumption or constraint. The approach fundamentally requires a reliable estimate of prompt difficulty to route problems through the compute-optimal strategy, yet the difficulty estimation procedure itself — generating 2048 samples per question and scoring them with the PRM — is extraordinarily expensive. The paper acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. The paper suggests future work on training models to predict difficulty directly from the question text (Section 8), but no such model is developed or evaluated.

The consequence. The 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain. For a single query, generating 2048 samples for difficulty estimation costs more than the largest test-time compute budgets studied (256–512 generations), making the per-query cost of the compute-optimal approach potentially worse than simply running best-of-256 on everything. The approach becomes cost-effective only if difficulty estimation cost can be amortized across many queries with similar difficulty profiles — such as in batch evaluation or when the same questions recur — but the paper provides no amortization analysis.

What evidence exists in the paper. The paper acknowledges the cost explicitly in Section 3.2 and notes that the predicted difficulty estimation method (using PRM scores rather than ground-truth labels) still requires generating 2048 samples. Figure 4 shows that predicted difficulty bins track oracle bins closely, confirming that ground-truth access is not required, but the sample-generation cost remains. No experiment measures the total cost including difficulty estimation or compares against a baseline that spends the estimation budget on additional best-of-N samples instead.

Mitigation status. Not addressed. The paper flags this as a "key avenue for future work" and suggests training a lightweight difficulty classifier or using adaptive estimation that starts with a small number of samples and adjusts mid-computation (Section 3.2, Section 8). Neither approach is implemented.

Hard Problems Remain Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The entire compute-optimal framework rests on the premise that the base model already produces correct solutions at a non-trivial rate. For the hardest difficulty quintile (bin 5), this premise fails: the base model's pass@1 is near zero, and no allocation of test-time compute — regardless of strategy or budget — provides meaningful improvement. The paper is transparent about this:

For search: "On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3). For revisions: "Bin 5: all ratios produce roughly 2–3% accuracy. No allocation strategy helps" (Figure 7, right, described in Section 6.3). In the FLOPs-matched comparison: "Test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time" (Section 7).

The consequence. The boundary condition is sharp: test-time compute amplifies existing capability but does not create it from nothing. This means the approach offers no path forward for problems that genuinely exceed the base model's training distribution — novel reasoning, out-of-distribution tasks, or problems requiring knowledge the model does not possess. For such problems, scaling pretraining remains the only viable path. The FLOPs-matched comparison in Figure 9 shows that on bin 5, the 14× larger model (using greedy decoding with no test-time augmentation) substantially outperforms the smaller model even with unlimited test-time compute, because the larger model has acquired capabilities through pretraining that no amount of inference-time search can replicate.

What evidence exists in the paper. Figure 3 (right, bin 5) shows accuracy at 1–3% for all methods and all budgets. Figure 7 (right, bin 5) shows roughly 2–3% accuracy regardless of sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% while the 14× larger model's star sits above it. The paper explicitly notes this in the Section 7 takeaway box.

Mitigation status. Not mitigated — this is a fundamental limitation that the paper acknowledges as inherent to the approach. The paper does not claim test-time compute can solve all problems. The practical implication is that deployers must separately handle genuinely hard problems (routing to larger models, human review, or accepting that they exceed current capability), and the compute-optimal framework provides no guidance for this regime.

The 14× Larger Model Baseline Is Weakened by Non-Compute-Optimal Pretraining and Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm rather than Chinchilla-optimal training where both parameters and data would be scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Additionally, the larger model is evaluated using only greedy decoding with no test-time augmentation of its own — no majority voting, no best-of-N, no search. This creates an asymmetric comparison: the smaller model receives compute-optimal test-time strategies while the larger model is restricted to a single forward pass.

The consequence. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1 for revisions, from the Figure 1 bar chart) may be overstated relative to what a fairer baseline would show. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger. Furthermore, giving the larger model even a modest test-time compute budget (say, best-of-8) would create a much stronger baseline — the paper's FLOPs accounting would need to include this inference cost, tightening the budget for the test-time approach. The current comparison answers "test-time compute vs. strictly larger model with no test-time compute," but the practically relevant question is "how should I allocate my total FLOPs between a larger model with some test-time compute and a smaller model with more test-time compute?"

What evidence exists in the paper. The paper acknowledges the parameter-only scaling choice but does not explore how a Chinchilla-optimal or test-time-augmented larger model would change the results. Figure 9 and the Figure 1 bar charts show the comparison as-is. No sensitivity analysis is provided for the pretraining baseline.

Mitigation status. The paper explicitly flags this as future work (Section 7). The current results should be interpreted as an existence proof that test-time compute can sometimes substitute for pretraining, not as a precise quantification of the tradeoff rate under optimal pretraining.

Single Benchmark, Single Model Family — Findings May Not Generalize

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. Several aspects of the findings could be model-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference.

Additionally, the test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample, and the selected strategies may not be robust. The paper does not report confidence intervals on the compute-optimal scaling curves.

The consequence. A practitioner deploying this approach on a different model family (e.g., Llama, Claude, Gemini) or a different reasoning domain cannot assume the same difficulty thresholds, optimal strategy selections, or quantitative efficiency gains will apply. The qualitative patterns — difficulty-dependent strategy effectiveness, verifier over-optimization on easy problems, revisions helping easy problems — may transfer, but the paper provides no evidence for or against this.

What evidence exists in the paper. The paper acknowledges the single-benchmark limitation implicitly through its scope but does not test on additional reasoning benchmarks, additional model families, or additional domains. All difficulty bin analyses, all strategy comparisons, and all FLOPs-matched results are specific to MATH + PaLM 2-S*.

Mitigation status. Not addressed. The paper makes no claims about generalizability and does not suggest replication studies as future work. A practitioner would need to replicate the key analyses (difficulty-dependent scaling curves, optimal strategy selection) on their own model and domain before relying on the paper's specific findings.

Sequential Revisions Are Inherently High-Latency — the Paper Measures FLOPs but Ignores Wall-Clock Time

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency. Sequential revisions are inherently serial — each revision depends on the previous one — while parallel best-of-N can be executed simultaneously with sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel (which the compute-optimal policy selects for many medium-difficulty problems, based on Figure 7) takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously. The paper also acknowledges a 38% correct-to-incorrect reversion rate in revision chains (Section 6.1), meaning that even within a serial chain, the model frequently undoes its own correct work, wasting sequential computation.

The consequence. For latency-sensitive applications — interactive assistants, real-time decision-making, any deployment where the user is waiting for a response — the sequential-heavy strategies favored by the compute-optimal policy on easy-to-medium problems may be impractical regardless of their accuracy advantages. The 4× FLOPs efficiency gain may translate to a wall-clock slowdown if the budget is allocated to deep sequential chains rather than shallow parallel ones. The paper's framework optimizes for total computation, not response time, and these objectives can conflict sharply.

What evidence exists in the paper. The sequential-to-parallel ratio analysis in Figure 7 shows that for many difficulty bins, the optimal allocation involves significant sequential depth. The 38% reversion rate is reported in Section 6.1. No latency measurements are provided — the paper reports only generation counts. The revision inference procedure (Figure 5) shows that each step in a chain depends on the previous step, confirming the serial dependency.

Mitigation status. Not addressed. The paper does not discuss latency, does not measure wall-clock time on any hardware configuration, and does not consider latency-constrained variants of the optimization objective. The compute-optimal framework could in principle be extended to include a latency term, but this is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation about multimodal AI by shifting the fundamental question from "how do we attach vision and audio to language models?" to "how do we represent all modalities within a shared discrete token space so that language-model infrastructure applies universally?" This is a reframing of the problem rather than a point improvement on an existing approach — it converts the architectural fragmentation that has defined multimodal research into a tokenizer design problem, with the decoder-only backbone treated as a fixed, modality-agnostic substrate.

The specific shift in thinking is this: prior work treated discretization as an unfortunate compromise — you discretize vision because you want to use autoregressive generation, but you accept the understanding penalty as the cost. The paper's central empirical claim, supported by the discrete-vs-continuous ablation in Figure 9, is that this tradeoff is not fundamental. With an SAE providing semantically complete pre-quantization features and sufficient training data (300B+ tokens), the discrete model achieves near-parity with its continuous counterpart (MMMU 60.0 discrete vs. 58.0 continuous, DocVQA 96.0 vs. 96.0 at the 300B data scale). This means the field's burden of proof shifts: the null hypothesis is no longer "discrete modeling has a performance ceiling" but "discrete modeling works if the tokenizer achieves semantic completeness."

The practical consequence is a redirect of research investment. If discrete tokens can substitute for continuous features without penalty, then the entire ecosystem of LLM optimization — training infrastructure, quantization, inference serving, RL pipelines — immediately applies to multimodal models without the engineering complexity of hybrid architectures. The paper demonstrates this concretely with the GRPO experiments in Section 6.1, where standard language-model RL methods are applied directly to image understanding and generation tasks, achieving gains on MMMU (64.22 → 66.45) and GenEval (83.94 → 87.33) with minimal modality-specific adaptation. This makes hybrid architectures (continuous vision encoders with bolt-on generation heads) look increasingly like transitional designs rather than stable solutions — they solve the integration problem with architectural complexity that the DiNA approach solves through representational unification.

The paper also resolves a significant contradiction in the literature. Prior unified models (Table 3) consistently showed that generation quality and understanding quality traded off against each other — Janus-Pro achieved GenEval 80 but MMMU 41.0, while NEO-unify achieved MMMU 68.9 but weaker generation on text rendering. The interpretation has been that these objectives genuinely compete for model capacity. The paper's experiment in Section 3.2.3 contradicts this interpretation: under a single autoregressive objective, the unified model achieves lower generation loss than a pure-generation model (by 0.02 despite half the task-specific data) while matching pure-understanding performance (within 0.006 loss). This suggests that the conflict in prior work was an artifact of having separate objectives or separate representational spaces — and that a properly unified objective converts what appeared to be competition into synergy.

The MoE training dynamics analysis (Section 3.2.5) provides a mechanistic glimpse into why this works. The modality-agnostic routing naturally develops modality-specialized experts (Figure 11, right) without architectural enforcement, and capacity utilization increases (507.1 → 584.6 routed tokens per expert on average). This suggests that the model discovers an efficient internal division of labor when given a unified representational space — experts specialize functionally while the shared attention mechanism handles cross-modal interactions. This is a more elegant solution than modality-aware routing because it lets the training dynamics determine the allocation rather than encoding architectural assumptions.

What becomes more attractive: (1) Tokenizer research becomes the critical path. If the backbone is fixed, all the leverage is in building better SAEs, better quantization strategies, and better de-tokenizers. The paper's finding that representation capacity and training scale matter more than the continuous-vs-discrete distinction means investment in encoder pretraining and quantization objectives has higher expected return than investment in novel multimodal architectures. (2) RL for multimodal models becomes straightforward. The GRPO results demonstrate that discrete representations make image understanding and generation amenable to the same RL methods used for text, without the ODE-to-SDE conversion steps required by continuous diffusion models. (3) Infrastructure reuse becomes a primary motivation. The paper's VHalf pipeline parallelism (Section 5) shows that with careful scheduling, the heterogeneous compute patterns of multimodal training can be absorbed into existing LLM training infrastructure.

What becomes less attractive: (1) Hybrid architectures with separate continuous encoders for understanding and separate generation heads. If a unified discrete approach achieves competitive understanding and strong generation simultaneously (Table 3), the engineering complexity of maintaining two separate representational pipelines becomes harder to justify. (2) Modality-specific architectural modifications (modality-aware MoE, bidirectional attention for images, 3D RoPE for video). The paper's finding that modality specialization emerges naturally in a modality-agnostic MoE suggests these architectural interventions are unnecessary and may even limit the model's ability to discover optimal cross-modal representations. (3) Treating generation and understanding as separate problems requiring separate solutions. The DiNA formulation — both are autoregressive next-token prediction with different conditional priors — makes this separation look like a historical accident rather than a principled design choice.

Follow-Up Research This Work Enables

Systematic evaluation of SAE choice on downstream multimodal performance. The paper adopts Qwen2.5-ViT as the SAE and notes that MoonViT and AIMv2 are potential alternatives, but never compares them. A strong follow-up would train dNaViT tokenizers on top of three different SAEs (Qwen2.5-ViT, MoonViT, AIMv2) at controlled data scale, then train identically configured LongCat-Next models on the resulting discrete tokens and compare performance across the full benchmark suite (Table 1 + Table 2). The hypothesis is that any SAE satisfying the semantic completeness criterion should work; a negative result — finding that one SAE dramatically outperforms others despite similar standalone performance — would reveal that the SAE must satisfy additional properties (e.g., specific alignment with the LLM's pretraining distribution) that the paper's criterion does not capture. The paper's own finding that a randomly initialized ViT achieves the best reconstruction (PSNR 30.52, Table 6) but presumably terrible semantics already hints that reconstruction fidelity and semantic completeness are partially independent axes — a systematic SAE comparison would map this space.

Direct full-scale discrete-vs-continuous comparison with matched compute. The paper's strongest ablation comparing discrete and continuous models uses a Qwen-7B backbone (Figure 9), and the full-scale comparison against Qwen3-VL-A3B (Table 1) confounds the discrete-vs-continuous variable with different backbones and training recipes. A definitive experiment would train LongCat-Next-A3B with both dNaViT (discrete) and NaViT (continuous) vision representations, matched for total training tokens, and evaluate on all visual understanding benchmarks. This requires implementing a continuous variant where the SAE features are projected directly into the LLM's embedding space through a learned connector (analogous to LLaVA-style integration), then comparing benchmark scores, training loss curves, and generation quality. The Qwen-7B ablation suggests the gap should be near zero; a positive result (no gap) would definitively validate the discrete approach; a negative result (a small but persistent gap at full scale) would identify whether there is a residual cost to discretization that the 7B experiment was too small to detect.

Ablation on the number of RVQ levels in downstream multimodal tasks. The paper adopts 8-level RVQ based on feature reconstruction loss (Appendix 9.3, Figure 19), but never measures how the number of levels affects multimodal performance. This matters because more levels increase representational capacity (the effective codebook size grows as K^L) but also increase the training burden on the DepthTransformer and the embedding table size. A controlled experiment would train dNaViT tokenizers with 1, 2, 4, 8, and 16 RVQ levels (all other settings fixed), then train LongCat-Next models on each and evaluate understanding (MMMU, OCRBench, DocVQA) and generation (GenEval, TIFF, CVTG-2K). The hypothesis from the paper's framework is that understanding should saturate at relatively few levels (since high-level semantics are captured in early RVQ stages) while generation should continue improving with more levels (since fine details require later residual stages). If confirmed, this would guide practitioners in trading off tokenizer complexity against task requirements; if not, it would indicate that the level structure is not cleanly decomposable into semantics-vs-details as the paper implies.

Combining revisions with PRM-guided search in the DiNA framework. The paper studies PRM tree-search and iterative revisions as independent mechanisms in the language-only setting (Sections 5-6 of the prior analysis), but the DiNA framework makes their combination natural and potentially powerful for multimodal generation. The natural extension is to use the revision model (fine-tuned on incorrect-to-correct trajectories) as the proposal distribution within beam search over image tokens — at each step of image generation, the model conditions on previously generated and rejected image token sequences as context, potentially producing higher-quality candidates. The PRM (trained on the base model's image token outputs) would score partial generations, and beam search would prune unpromising branches. The experiment would compare: (1) standard autoregressive image generation, (2) best-of-N with PRM scoring, (3) beam search over image tokens with PRM guidance, (4) revisions applied to complete image generations, and (5) the combination of revisions within beam search. The paper's finding that understanding and generation are synergistic (Section 3.2.3) suggests that combining these mechanisms could yield gains neither achieves alone, particularly on compositional generation tasks (GenEval) and text rendering (TIFF, CVTG-2K) where the beam search could explore multiple layout strategies while revisions refine details.

Extending the semantic completeness criterion to domains without clean correctness signals. The paper's semantic completeness definition (Equation 1) relies on the existence of ground-truth responses A for image-centric queries Q during tokenizer training. For domains where such queries are unavailable or correctness is ambiguous — video understanding, 3D scene reasoning, multimodal dialogue — the criterion cannot be directly applied. A follow-up would define a self-supervised semantic completeness proxy: can the discrete tokens z preserve enough information to predict other tokens from the same or different modalities? Concretely, for video, the proxy would be: given discrete tokens from frame t, can the model predict discrete tokens from frame t+1? For multimodal dialogue, the proxy would be: given image tokens and preceding text tokens, can the model predict the next text token? The experiment would train dNaViT-style tokenizers on video data with this self-supervised objective replacing the language-conditioned supervision, then evaluate whether the resulting tokens support both video understanding (e.g., Video-MME, EgoSchema) and video generation. This would test whether semantic completeness can be bootstrapped from the autoregressive objective itself, removing the dependence on expensive vision-language pretraining for SAE construction.

Stress-testing the understanding-generation synergy claim at full scale. The paper's most provocative finding — that generation does not compromise understanding and may actively enhance it (Section 3.2.3, Figure 10) — is demonstrated only at Qwen-7B scale with 100B training tokens. The full-scale LongCat-Next was never compared against a pure-understanding variant at the A3B scale. A definitive stress test would train three full-scale models: (1) LongCat-Next as-is (unified understanding + generation, ~2T tokens total), (2) LongCat-Next trained only on understanding data (matched total tokens, generation data replaced with additional understanding data), and (3) LongCat-Next trained only on generation data (matched total tokens). If model (1) matches model (2) on understanding benchmarks and matches model (3) on generation benchmarks, the synergy claim is validated at industrial scale and has profound implications for training efficiency — unified training yields two capabilities for the price of one. If model (1) underperforms model (2) on understanding, the "multimodal tax" is real but potentially worth paying for the generation capability. If model (1) underperforms both, the synergy is an artifact of the small-scale experiment and the paper's central claim about reconciliation requires revision.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of multimodal systems in production. The paper's primary practical contribution is demonstrating that a single model can handle text, vision, and audio at competitive quality without maintaining separate specialist models. For an organization deploying multimodal AI — customer support with image understanding, document processing with OCR, voice interaction with TTS/ASR — the default architecture today involves stitching together multiple specialist models (a vision-language model for images, a separate TTS system for speech output, an ASR system for speech input). LongCat-Next shows that a single A3B model (68.5B total, 3B activated) can replace this ensemble, achieving OCRBench 86.5 (competitive with specialist InternVL3.5-A3B-Flash at 88.0*), TTS quality of WER 1.89 on English SeedTTS (matching MiMo-Audio at 5.37* and substantially better than Kimi-Audio at 29.45), and competitive ASR performance. The practical benefit is not just reduced model count — it is architectural simplification of the serving stack: one model binary, one inference pipeline, one optimization surface, one set of quantization and deployment tools. The paper's open-source release of both the model and tokenizers makes this immediately accessible.

Data generation pipelines for multimodal self-improvement. The DiNA framework's unification of understanding and generation under a single autoregressive objective makes it particularly suitable for self-improvement loops where the model generates training data for itself. A concrete deployment scenario: an organization has a collection of unlabeled images and wants to generate high-quality captions for training a downstream model. With LongCat-Next, the pipeline is: (1) for each image, generate multiple captions using the understanding pathway (best-of-N with PRM scoring, if a verifier is available); (2) for each generated caption, regenerate the image using the generation pathway and check reconstruction consistency; (3) retain captions where the regenerated image is semantically similar to the original (measured by the SAE's feature similarity); (4) fine-tune the model on these retained image-caption pairs. This consistency filtering — possible because the same model does both directions — provides a self-supervised quality signal that would require a separate model in a hybrid architecture. The paper's RL results (Table 9, Appendix 9.2) provide preliminary evidence that this kind of iterative refinement works: RL training on model-generated data improved MMMU from 64.22 to 66.45 and GenEval from 83.94 to 87.33. The key efficiency gain is that the consistency check uses the model's own representations rather than requiring external verifiers.

On-device and edge deployment of multimodal assistants. The A3B activated parameter count (average 3B, range 2.9B–4.5B) puts LongCat-Next in a regime where on-device deployment is plausible with aggressive quantization, particularly given that the MoE architecture activates only a fraction of total parameters per token. The practical scenario: a mobile device runs a quantized LongCat-Next that handles real-time voice interaction (parallel generation mode for low latency, achieving 79.33 on LlamaQuestions in the parallel setting), takes photos and answers questions about them (MMMU 70.6, competitive with cloud-scale models), and generates images from text descriptions (GenEval 84.44, exceeding FLUX.1-dev's 66.00). The paper's VHalf pipeline parallelism (Section 5) demonstrates training efficiency, but the inference implications are equally important: the modality-agnostic backbone means standard LLM inference optimizations (KV-cache compression, speculative decoding, weight-only quantization) apply to all modalities without modification. The key metric missing from the paper is inference latency and memory consumption — a deployment team would need to benchmark these on target hardware — but the architectural simplicity makes the engineering path clear compared to hybrid systems requiring separate inference pipelines for each modality.

Unified RL fine-tuning for multimodal alignment. The paper's GRPO experiments (Section 6.1) demonstrate that discrete representations make multimodal RL straightforward: the same policy optimization methods used for text RLHF apply directly to image and audio generation tasks. A practical deployment scenario is aligning a multimodal assistant with user preferences across modalities. The setup: deploy LongCat-Next as an interactive assistant; collect user feedback (ratings, preferences, corrections) on text responses, generated images, and synthesized speech; define a unified reward function that combines these modality-specific signals; apply GRPO with the filtering mechanisms described in Section 6.1 (entropy filter, probability difference filter) to optimize the model end-to-end. The paper's finding that multi-dimensional reward models prevent reward hacking (Section 6.1, comprehensive capability + OCR + semantic alignment + image quality) provides a template for constructing robust reward signals. The key advantage over hybrid systems is that a single RL training run optimizes all modalities simultaneously — the understanding-generation synergy shown in Figure 10 suggests that improvements in one modality may transfer to others.

When to Prefer This Method

The paper explicitly positions the DiNA framework against the prevailing language-plus-auxiliary paradigm (continuous vision encoders projecting features into LLM embedding spaces) and against hybrid unified architectures (separate encoders for understanding and generation, as in Janus). It also implicitly contrasts with encoder-free approaches (EVE, NEO) and specialist models (dedicated T2I systems like Flux, dedicated MLLMs like Qwen3-VL). The decision criteria below are drawn from the paper's empirical results and stated design goals:

Prefer DiNA-based native multimodal modeling (LongCat-Next) when:

You need both understanding and generation across multiple modalities in a single deployed system, and the engineering simplification of one model outweighs any residual per-modality performance gaps. The paper shows LongCat-Next matching Qwen3-VL-A3B on many understanding benchmarks (MathVista 83.1 vs. 80.1*) while adding image generation (GenEval 84.44, competitive with specialized T2I models) and speech capabilities — a capability combination that would otherwise require 3+ separate models.
Text rendering and OCR-heavy document understanding are critical requirements. LongCat-Next substantially outperforms prior unified models on text-heavy benchmarks (LongText-EN 93.15 vs. next-best unified NEO-unify at 91.40; OmniDocBench-en 0.152 vs. Qwen3-Omni's 0.289), suggesting the SAE + RVQ tokenizer captures fine-grained textual detail that discrete models historically missed.
You have sufficient training budget (2T+ tokens) to train the discrete embeddings from scratch, and you need the deployment simplicity of a single modality-agnostic backbone. The paper shows that discrete embeddings require more data than continuous projections (Figure 9, the gap closes at 300B mid-training+SFT tokens but persists at 4B), so the approach is not suitable for low-data regimes.
You plan to apply RL fine-tuning across modalities, and want to use standard LLM RL methods (GRPO, PPO) without modality-specific modifications. The paper's GRPO results (Table 9) demonstrate that discrete representations make multimodal RL as straightforward as text RL.

Prefer continuous-projection approaches (e.g., Qwen3-VL, InternVL3.5) when:

Visual understanding is the sole or dominant requirement, and generation is not needed. The paper shows that LongCat-Next trails Qwen3-VL-A3B on some understanding benchmarks (MMStar 69.3 vs. 72.1; CountBench 82.1 vs. 90.6), so a pure-understanding continuous model may achieve slightly higher accuracy on these tasks.
Training data budget is limited. The discrete-vs-continuous ablation (Figure 9, 4B mid-training+SFT tokens) shows a clear gap favoring continuous representations at lower data scales — the discrete model needs more data to converge its randomly initialized embeddings.
Inference latency for generation is a hard constraint, and you cannot amortize the cost of the DepthTransformer decoding step. The paper does not measure generation latency, but the multi-level parallel decoding plus flow-matching refiner adds computation beyond a standard LLM forward pass.

Prefer specialist T2I models (e.g., Flux, Qwen-Image) when:

Maximum image generation quality is the only objective, particularly on benchmarks requiring world knowledge (WISE: LongCat-Next 57.00 vs. Gemini 2.5 Flash Image 76.27) or pixel-level text rendering fidelity (TIFF: 82.85/84.38 vs. Gemini's 90.53/90.80). Specialist models optimized purely for generation still hold an edge on these metrics.
You are willing to maintain a separate generation pipeline and do not need the architectural simplicity of a unified system. The paper does not claim to beat the best specialists on generation quality, only to be competitive while adding understanding and speech capabilities.

Prefer encoder-free approaches (EVE, NEO) when:

Absolute architectural simplicity is valued over training efficiency, and you have the compute budget to handle extremely long token sequences (raw pixel tokenization produces much longer sequences than the 28× compression from dNaViT's SAE). The paper acknowledges encoder-free approaches "suffer from pixel redundancy" (Section 2.2.1) but celebrates their conceptual purity — in regimes where compute is abundant and simplicity is prioritized, they remain an attractive alternative.
You want to avoid dependence on a pretrained SAE, which may introduce biases from the SAE's training data or architecture. LongCat-Next's tokenizer inherits whatever biases exist in Qwen2.5-ViT's training — encoder-free approaches avoid this dependency at the cost of learning all visual semantics from scratch.