MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction

🎯 Pitch

MiniCPM-o 4.5 overturns the standard turn-taking paradigm by letting an AI see, listen, and speak simultaneously—interrupting itself with live updates mid-sentence. Its 9B model even proactively initiates conversation based on the unfolding scene, matching Gemini 2.5 Flash visually while producing faster, better speech than a 30B rival on just 12GB of edge-device RAM.

1. Executive Summary

This paper introduces MiniCPM-o 4.5, an open-source 9B-parameter MLLM designed for real-time full-duplex omni-modal interaction—the ability to see, listen, and speak simultaneously while also exhibiting proactive behaviors. The core technical contribution is Omni-Flow, a unified streaming framework that aligns multimodal inputs and outputs along a shared temporal axis, converting conventional turn-based interaction into a continuous, time-aligned process where perception and response unfold in parallel. MiniCPM-o 4.5 achieves state-of-the-art open-source vision-language performance at its scale, approaching Gemini 2.5 Flash (e.g., 77.6 vs. 78.5 on OpenCompass in instruct mode), and surpasses the much larger Qwen3-Omni-30B-A3B in both omni-modal understanding and speech generation quality (e.g., 0.86 vs. 1.41 CER on SeedTTS Test-ZH), all while running efficiently on edge devices with less than 12GB RAM—establishing that human-like continuous full-duplex interaction is feasible at practical deployment scales, but only when perception, response, and control decisions are explicitly time-aligned rather than serialized into alternating phases.

2. Context and Motivation

The Core Problem: Interaction Architecture Limits Multimodal AI, Not Just Modality Coverage

The fundamental question this paper tackles is not simply "how do we build a model that processes vision, speech, and text"—multimodal models already exist for that. Rather, the question is: how should a multimodal AI system be architected at the interaction level so that it behaves more like a human conversational partner, capable of perceiving and responding in parallel rather than serialized turns?

This distinction is crucial. As the paper argues in Section 1, progress in multimodal LLMs has been rapid along three axes—modality coverage has expanded, output quality has improved, and latency has decreased. Yet these improvements all operate within the same fundamental interaction paradigm: a strict alternating phase structure where the model first perceives inputs (listens/watches), then generates a response, then waits for the next input. This creates what the paper calls the "Blocked-I/O Limitation of Existing Paradigms" (Figure 3), where:

Information flow is blocked during generation: While the model is speaking, new environmental information—the user interrupting, a change in the visual scene, an important sound—cannot influence the ongoing utterance. The model is effectively deaf and blind while it talks.
Behavior is strictly reactive: Models only respond to explicit user requests. They lack the ability to initiate speech based on their continuous observation of the environment, such as offering a reminder when they notice a timer has elapsed or commenting on a relevant visual event unfolding in a shared viewing session.
Temporal alignment is lost: In long interactions, the model's understanding of "what just happened" becomes decoupled from its speech output. A model might reference events that occurred seconds ago as if they are current, because its perception and response timelines are misaligned.

This matters because human conversation is fundamentally full-duplex: we process incoming auditory and visual signals continuously while formulating and delivering responses. We interject, we adjust our speech mid-utterance when we notice confusion, we react to environmental events without being prompted. The paper argues that the gap between current AI interaction and human interaction is now dominated not by what the model can understand or generate, but by when and how it can do so relative to the unfolding environment.

Why Full-Duplex Interaction Matters: Real-World Impact

The practical implications span several critical deployment scenarios:

Continuous assistance and ambient interaction (Section 1). In long-horizon assistance settings—a user cooking while following a recipe, a technician repairing equipment with AI guidance, an elderly person receiving daily activity support—the AI needs to provide timely, context-aware input without requiring explicit queries. A turn-based model requires the user to constantly trigger interactions, breaking the flow of the primary task. A full-duplex model can offer proactive reminders ("the oven timer has 2 minutes remaining"), warnings based on visual scene understanding ("the pot is boiling over"), or relevant information triggered by environmental cues rather than explicit prompts.

Natural conversation dynamics. Real human dialogue involves overlapping speech, backchanneling ("uh-huh," "I see"), and mid-utterance adjustments. A model that must finish generating before processing new input cannot handle these dynamics. If a user says "I need to go to the... actually, never mind," a turn-based model will likely respond to the incomplete or cancelled request. A full-duplex model, by continuously perceiving, can detect the self-correction and adjust its planned response before speaking.

Real-time scene commentary and collaboration. Consider two people watching a live sports event while an AI provides commentary. In a turn-based system, the AI must be explicitly queried ("what just happened?") and by the time it responds, the described event is already in the past. A full-duplex system, as shown in Figure 3's bottom panel, can provide continuous commentary ("The player in red is dribbling... OH! He SHOOTS!") where the perception and response timelines are tightly coupled, and the model can interject spontaneously when significant events occur.

Edge deployment feasibility (Section 7). The paper emphasizes that MiniCPM-o 4.5 achieves this full-duplex capability on edge devices with less than 12GB RAM. If full-duplex interaction required datacenter-scale compute, its practical impact would be limited. The fact that it runs on consumer GPUs and even mobile-class hardware (via llama.cpp-omni) means that always-on, proactive AI assistants become feasible in privacy-sensitive, connectivity-limited, or cost-constrained environments where cloud dependence is undesirable.

Where Prior Approaches Fall Short

The paper identifies several distinct limitations of existing work:

1. Turn-based streaming models are still alternating-phase systems. Recent models like GPT-4o (2024), Gemini Live (2025), and Qwen3-Omni (2025) support streaming audio and video input with streaming speech output. However, the paper's analysis (Figure 3) reveals that these systems still internally serialize perception and response into alternating phases. The model finishes processing a chunk of input, generates a chunk of output, then processes the next chunk of input. There is no mechanism for the input stream to interrupt or modify generation mid-utterance. The perception window is effectively closed during speech production.

This is not a trivial implementation detail—it is an architectural constraint. Standard causal language model inference is inherently serial: the model must complete its autoregressive generation for a response before it can ingest new tokens that might change what it should say. Omni-Flow addresses this by partitioning time into fine-grained chunks where perception and generation are interleaved at the token level within each chunk (Section 3.2).

2. Existing omni-modal models lack proactive behavior. Models like Qwen3-Omni-30B-A3B and Kimi-Audio provide strong single-turn audio-visual understanding, but they only speak when spoken to. The paper explicitly notes in Section 1:

"model behaviors remain strictly request-driven, rather than being proactively initiated from the evolving multimodal environment"

This is partly a training data problem—most instruction-tuning datasets consist of explicit query-response pairs where the assistant only responds to direct prompts—and partly an architectural problem. Without a mechanism for the model to continuously observe the environment and decide autonomously when to speak, proactive behavior cannot emerge. Omni-Flow's serialization scheme (Section 3.2) addresses this by making the listen/speak decision an explicit prediction within each time chunk. The model is trained to output a [listen] token when it should remain silent and regular text tokens when it should speak, making the decision to initiate interaction part of the model's learned behavior rather than an external trigger.

3. Speech generation is temporally decoupled from ongoing perception. Existing streaming speech systems generate speech tokens well ahead of playback. The paper identifies this as a fundamental problem for full-duplex interaction (Section 3.4): if the model generates 5 seconds of speech based on its current state, but the environment changes 1 second into playback, the remaining 4 seconds of speech are stale—they reflect an outdated understanding of the world. The paper's proposed Time-Aligned Interleaving (TAIL) mechanism addresses this by ensuring that text generation stays temporally aligned with speech playback, so the model's speech output at any moment reflects its most recent perceptual state.

4. Voice activity detection (VAD) introduces latency and rigidity. Many existing streaming systems rely on external VAD modules to determine when the user has finished speaking, at which point the model begins generating a response. The paper notes this introduces unnecessary latency and makes interaction brittle—the model cannot interrupt, overlap, or initiate speech. By training the model itself to predict listen/speak tokens (Section 3.3), Omni-Flow eliminates the dependency on external VAD and makes turn-taking decisions a learned, context-dependent behavior.

5. Disjoint training pipelines struggle to unify modalities. Prior omni-modal models often train speech, vision, and text capabilities separately and then combine them, or use separate modules that are not jointly optimized. This leads to several issues the paper addresses:

Catastrophic forgetting: When speech capabilities are added to a strong vision-language model, the vision-language performance often degrades (a phenomenon the paper references in citing Hsiao et al., 2025; Section 2, caption to architecture description).
Shallow cross-modal connections: If the speech decoder only sees the LLM's final text output (not intermediate hidden states), it cannot leverage the LLM's rich contextual understanding for prosody, emotion, and speaking style.
Lack of end-to-end gradient flow: Without differentiable connections from speech output back through the LLM backbone, the LLM cannot learn to produce representations that are useful for speech generation.

MiniCPM-o 4.5 addresses all three through end-to-end token-level continuous connections (Section 2): the speech token decoder receives both its own embedding and the LLM backbone's hidden state (via an MLP projector) at each position, and all components are jointly trained with gradient propagation throughout the entire architecture.

6. No general framework exists for full-duplex multimodal interaction. Most critically, prior work lacks a unified conceptual framework for full-duplex interaction. Each system implements its own ad-hoc serialization, buffering, and timing strategies. The paper introduces Omni-Flow as a general formulation: identify time-aligned streams (visual, audio, output), partition time into fixed-duration chunks, serialize tokens within each chunk as [visual; audio; output], and concatenate chunks into a single sequence for a causal language model. This framework abstracts away the implementation details and provides a principled foundation for designing and analyzing full-duplex systems.

How This Paper Positions Itself

The paper's positioning can be understood along several dimensions:

Relative to the MiniCPM family lineage. This is explicitly the next step in the MiniCPM series, building on MiniCPM-V 4.5 (vision-language) and MiniCPM-o 2.6 (initial omni-modal). The authors frame MiniCPM-o 4.5 as advancing the interaction paradigm, not just improving benchmark scores. Figure 2 tells this story visually: the evolution from text interaction (ChatGPT 2022), to multimodal understanding (GPT-4V 2023), to omni live streaming (GPT-4o 2024), to the target of human-like full-duplex interaction (MiniCPM-o 4.5 2026). The paper positions itself at the frontier of this trajectory.

Relative to Qwen3-Omni-30B-A3B. This is the paper's primary open-source comparator, and the comparison is deliberately asymmetric: MiniCPM-o 4.5 has 9B parameters versus 30B (with 3B active). The paper argues that architectural efficiency—specifically the end-to-end token-level connections and the delegation of speech token generation to lightweight decoders rather than the LLM backbone—enables competitive or superior performance at a fraction of the parameter count and compute budget. This positions MiniCPM-o 4.5 as the efficient path to omni-modal capability, consistent with the MiniCPM family's emphasis on edge deployment.

Relative to proprietary models (Gemini 2.5 Flash, GPT-5). The paper is careful not to claim superiority over these much larger proprietary systems. Instead, it uses "approaches Gemini 2.5 Flash" language (abstract, Section 6.2) and the Figure 1 radar plot to show it is competitive in vision-language capabilities while being fully open-source and edge-deployable. The implicit argument is: full-duplex omni-modal interaction is not a capability that requires massive proprietary models; it can be achieved, and made openly available, at practical scales.

Relative to interaction paradigm research. The paper positions Omni-Flow not just as an implementation technique for MiniCPM-o 4.5, but as a general framework for full-duplex interaction that could be adopted by other models. The explicit analogy to "time-division multiplexing" (Section 3, opening paragraph) and the ablation studies on design choices (Table 1) suggest the authors view this as a contribution to the broader understanding of multimodal interaction architecture, not just a model-specific optimization.

The proactive behavior positioning. The paper explicitly identifies proactive behavior as a key differentiator. Current models are "reactive"—they respond only when prompted. MiniCPM-o 4.5 is "proactive"—it can initiate speech based on its continuous understanding of the environment. This is not presented as a separate module or capability, but as a natural emergent property of the Omni-Flow framework: because the model makes listen/speak decisions independently in each time chunk, and because it is trained on data where appropriate proactive behavior is demonstrated, the behavior emerges from the same architecture and training objective. The paper is careful to note this is still "relatively simple" (Section 8, Limitations), but it establishes proactive interaction as a core research direction.

The efficiency positioning. Throughout the paper, efficiency is not treated as an afterthought but as a primary design constraint. The visual resampler achieves 16× compression (vs. the common 4×), the audio projector achieves 5× temporal compression, the LLM generates only text (not speech tokens), and the speech decoder is a lightweight 0.3B Llama model. The inference results in Table 11 and Table 12 demonstrate that these choices pay off: full-duplex interaction runs on consumer hardware. This positions MiniCPM-o 4.5 as proof that full-duplex omni-modal interaction is not just a research demo but a deployable capability.

3. Technical Approach

3.1 Reader Orientation

MiniCPM-o 4.5 is an end-to-end trainable system that takes in continuous streams of video and audio as input and produces a stream of spoken text as output, all while running in real-time on consumer hardware. The core problem it solves is the architectural mismatch between how humans communicate (continuously, in parallel, with proactive initiations) and how current AI models process interaction (as a serialized pipeline of alternating listen-then-speak phases). The "shape" of the solution is to reimagine multimodal interaction not as discrete turns but as a continuous, time-aligned process where the model receives and processes new sensory information while it is generating speech, and where the decision to speak or remain silent is an explicit learned behavior that emerges from the same architecture, not an externally triggered event.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in an end-to-end differentiable chain:

Visual Encoder (SigLIP ViT, 0.4B parameters): Takes image frames from a video stream and converts each one into a compact set of visual tokens. Uses a high-compression resampler (16×) to keep the token budget manageable.
Audio Encoder (Whisper Medium, 0.3B parameters): Processes raw audio in streaming chunks and produces a time-stamped sequence of audio feature tokens. A 5× temporal compressor further reduces these before entering the LLM.
LLM Backbone (Qwen3-8B, ~8.2B parameters): The central reasoning engine. It receives interleaved visual tokens, audio tokens, and special control tokens organized along a shared timeline. It autoregressively generates text tokens representing the assistant's response, along with hidden states that drive speech generation.
Speech Token Decoder (Llama-based, ~0.3B parameters): A lightweight decoder that takes each text token's LLM hidden state (via an MLP projector) and produces discrete speech tokens (S3 tokens at 25 tokens/second). Because prosody and style decisions are pre-encoded in the LLM's hidden states, this decoder can be small.
Streaming Flow-Matching Decoder: Converts the discrete speech token stream into raw audio waveforms for playback, using a reference audio clip from the system prompt for voice identity.

Information flows through these components as a continuous stream: video frames and audio chunks arrive at their respective encoders → compressed tokens are inserted into a unified sequence ordered by their millisecond-level timestamps → the LLM processes one "time chunk" at a time (1.0 second granularity), first ingesting new perceptual tokens, then deciding whether to output a [listen] token (remain silent) or begin generating text tokens → each generated text token's hidden state is sent to the speech token decoder → speech tokens are synthesized into waveforms → the cycle repeats continuously.

3.3 Roadmap for the Deep Dive

The technical explanation proceeds in five stages, ordered to build understanding from the architectural substrate through the interaction framework to the training procedure:

First, the end-to-end multimodal architecture (Section 2) — how visual encoding, audio encoding, text generation, and speech generation are all connected through differentiable token-level interfaces, and the critical design choices (high compression ratios, hidden-state injection into speech decoding) that enable efficiency without sacrificing quality.
Second, the Omni-Flow framework (Section 3) — the conceptual core of the paper: how time-aligned streams are defined, serialized into a unified sequence, and used to convert turn-based interaction into continuous full-duplex behavior. This includes the design tradeoffs explored through ablation studies (chunk size, boundary explicitness, control formulation).
Third, the Time-Aligned Interleaving (TAIL) mechanism for speech generation (Section 3.4) — why existing streaming speech generation causes temporal misalignment between perception and speech playback, and how adaptive chunk-wise text generation with bounded look-ahead resolves this.
Fourth, the data pipeline and training strategy (Sections 4–5) — what data is used and how it is constructed, particularly the novel full-duplex interaction data with millisecond-level time indexing, and the staged training procedure (speech pretraining → joint pretraining → supervised fine-tuning → reinforcement learning) designed to progressively integrate speech into a pre-existing vision-language model without catastrophic forgetting.
Fifth, the efficient inference implementation (Section 7) — how the system achieves real-time full-duplex interaction on consumer hardware, including the custom llama.cpp-omni inference framework, quantized deployment, and cross-platform support.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that full-duplex omni-modal interaction requires rethinking the temporal structure of model inference, not just adding modalities or reducing latency. The key innovations are: (1) a unified time-aligned serialization that enables parallel perception and response, (2) learned listen/speak control that enables proactive behavior, and (3) adaptive speech generation timing that keeps spoken output aligned with the live environment.

Visual Encoding: High-Compression Streaming Image Processing

The visual encoding pipeline transforms raw video frames into a compact sequence of tokens that the LLM can process alongside text and audio tokens. The design is driven by a tension: high-resolution images contain rich detail necessary for OCR, document understanding, and fine-grained visual reasoning, but full-duplex streaming requires processing multiple frames per second within a tight token budget.

Image partitioning strategy. MiniCPM-o 4.5 adopts the LLaVA-UHD (Guo et al., 2024) partitioning approach. The key idea is that instead of downscaling a high-resolution image to a fixed small size (which loses detail), or encoding it as a single massive grid of patches (which consumes too many tokens), the image is divided into multiple overlapping "slices" that each get encoded independently. Each slice covers a portion of the original image at a resolution that preserves fine detail. This is conceptually similar to how a human might examine a detailed diagram by focusing on different regions sequentially, but here all regions are processed in parallel and the results are concatenated.

The paper specifies two resolution regimes. For the full-duplex streaming mode, the maximum resolution is limited to 448×448 pixels per slice. This is a deliberate constraint: full-duplex interaction requires processing multiple frames per second, and higher resolutions would blow out the token budget. For traditional turn-based inference (instruct mode and thinking mode), the maximum resolution is 2240×2240, enabling the model to leverage fine-grained visual details when interaction latency is less critical. This dual-resolution design means the same model can operate efficiently in both streaming and non-streaming contexts without architectural changes—only the input preprocessing differs.

Encoding pipeline details. Each image slice is first passed through a SigLIP ViT (Zhai et al., 2023) with 417.8M parameters (27 layers, hidden dimension 1,152, 16 attention heads, FFN dimension 4,304, GELUtanh activation, patch size 14×14). The ViT outputs a grid of patch embeddings: each 14×14 pixel region of the input becomes a 1,152-dimensional vector. For a slice of size 448×448, this produces a 32×32 grid = 1,024 patch tokens.

Next, a visual resampler module compresses these 1,024 tokens down to just 64 tokens. The resampler (88.9M parameters) uses a set of 64 learned query tokens that cross-attend to the ViT's patch tokens, producing 64 output embeddings of dimension 4,096 (matching the LLM's hidden dimension). This achieves a 16× compression ratio (1,024 → 64).

Why 16× compression matters. The paper notes that this is "higher than the common 4× compression" used in models like Qwen2.5-VL, InternVL3.5, and Qwen3-VL. The 4× compression typically compresses each 256 (16×16) patch grid to 64 tokens, while MiniCPM-o 4.5 compresses 1,024 tokens to 64—four times more aggressive. This is not an arbitrary choice; it is motivated by the streaming use case. In full-duplex mode, the model processes approximately 1 frame per second. If each frame produced 256 visual tokens (as with 4× compression), a 10-second interaction would consume 2,560 visual tokens—a significant fraction of the LLM's context window just for visual processing. At 64 tokens per frame, the same 10-second interaction uses only 640 visual tokens, leaving far more capacity for audio tokens, text history, and generation. The paper implicitly argues, through this design choice, that modern vision encoders produce redundant patch-level representations and that aggressive compression via learned cross-attention can preserve task-relevant information while dramatically reducing token count.

The 64 compressed tokens per slice are the final visual embeddings that get inserted into the LLM's input sequence at the appropriate temporal position, as described in the Omni-Flow serialization below.

Audio Encoding: Streaming Chunk-Based Processing with Temporal Compression

The audio encoding pipeline converts raw audio waveforms into a sequence of feature vectors that represent the acoustic content over time, with careful attention to streaming operation and temporal compression.

Encoder architecture. MiniCPM-o 4.5 uses the encoder portion of Whisper Medium (Radford et al., 2023), a 307.2M-parameter transformer (24 layers, hidden dimension 1,024, 16 attention heads, FFN dimension 4,096, GELU activation). Whisper processes audio represented as 80-channel Mel-frequency spectrograms, which convert raw audio into a time-frequency representation that emphasizes perceptually relevant frequency bands. The encoder outputs a sequence of feature vectors at a rate of 50 tokens per second of audio. This means that 1 second of input speech produces 50 feature vectors, each of dimension 1,024.

Streaming operation. The paper specifies that audio encoding operates "in a chunk-based streaming fashion" following the approach of Wenet (Yao et al., 2021). This means the encoder does not wait for a complete utterance before producing output; instead, audio arrives in small time windows (chunks), and the encoder processes each chunk as it arrives, producing output tokens with minimal latency. The chunk size is not explicitly specified in the paper, but typical streaming ASR systems use chunks of 100–200 milliseconds. This streaming design is critical for full-duplex interaction: the model cannot wait for the user to finish speaking before processing their audio, because it needs to potentially interject or adjust its ongoing speech based on what it hears mid-utterance.

Temporal compression. The 50 Hz output rate (50 tokens per second) would be expensive for the LLM backbone. If the model processes 1-second time chunks (as specified in Section 3.3), each chunk would contribute 50 audio tokens, which adds up quickly over long interactions. To address this, the paper introduces a two-layer MLP projector (21.0M parameters, dimensions 1024 → 4096 → 4096) that performs a 5× temporal compression, reducing the rate from 50 tokens per second to 10 tokens per second. The MLP takes a window of 5 consecutive Whisper feature vectors (each 1,024-dimensional), concatenates or aggregates them, and projects to a single 4,096-dimensional vector matching the LLM's hidden space.

The choice of a simple two-layer MLP rather than a more complex architecture (e.g., an attention-based aggregator, or a convolutional downsampler) reflects an efficiency-driven design philosophy: the projector's job is primarily to align dimensions and reduce token rate, not to perform complex reasoning. Any sophisticated audio understanding should happen in the LLM backbone, not in the projector. The ReLU activation provides non-linearity for the dimensional transformation, but the architecture is deliberately minimal.

Why 10 tokens per second? This rate balances two concerns. If the rate were too high (e.g., the original 50 Hz), the LLM would be overwhelmed by audio tokens, leaving less capacity for visual context, conversation history, and generation—and making training less efficient because the model would need to learn long-range dependencies over many more tokens. If the rate were too low (e.g., 2–5 Hz), the temporal resolution would be insufficient to capture rapid acoustic events like consonant transitions, word boundaries, or overlapping speech—degrading the model's ability to understand and respond in real-time conversations. The 10 Hz rate (one token every 100 milliseconds) roughly matches the temporal resolution of linguistic events: syllables last about 100–200 milliseconds, and phoneme transitions occur on similar timescales. This means each audio token carries information at approximately the syllable level, which is a natural granularity for language understanding.

LLM Backbone: Text-Centric Generation with Hidden-State Speech Conditioning

The central reasoning engine is a Qwen3-8B model (Team, 2025) with 8,189.2M parameters. Its architecture follows standard decoder-only transformer design: 36 layers, hidden dimension 4,096, 32 attention heads with 8 key-value heads (Grouped-Query Attention, GQA), head dimension 128, FFN dimension 12,288 with SiLU activation, RMSNorm normalization with ϵ=10⁻⁶, vocabulary size 151,748 tokens, maximum context length 40,960 tokens, and RoPE position encoding with base frequency θ=10⁶. The model uses no weight tying between input embeddings and output projection.

Why the LLM generates only text, not speech tokens. This is one of the most consequential architectural decisions in the paper. Some recent omni-modal models (e.g., Mini-Omni, Qwen2.5-Omni) have the LLM backbone directly generate discrete speech tokens (typically ~25 tokens per second) in addition to text. The paper argues this approach has two major drawbacks:

Efficiency impediment: Generating 25 speech tokens per second means the LLM must perform 25 autoregressive decoding steps for every second of speech, even though natural speech only conveys semantic content at roughly 3–4 words per second (about 3–4 text tokens per second at typical tokenization rates). The remaining ~21 decoding steps per second are dedicated to acoustic detail (prosody, timbre, pronunciation) that does not require the LLM's full reasoning capacity. In full-duplex interaction, where the model must also continuously process new visual and audio inputs, this 6–8× inflation in decoding steps is a significant computational burden.
Language capability degradation: The paper cites prior work (Hsiao et al., 2025; Xu et al., 2025) showing that requiring the LLM backbone to directly model low-level acoustic features tends to degrade its core language understanding and reasoning capabilities—a form of catastrophic interference where the model's representational capacity becomes partially diverted to speech acoustics at the expense of semantic processing.

MiniCPM-o 4.5's design avoids both problems by having the LLM generate only text tokens (at approximately 3–4 tokens per second for natural speech). The acoustic detail is handled by a separate, lightweight speech token decoder (0.3B parameters) that receives the LLM's hidden states as conditioning. This means the LLM does not need to spend capacity modeling low-level speech features, but it does need to encode prosodic and stylistic information in its hidden states so the speech decoder can produce natural-sounding speech. The paper's phrase "with prosodic decisions pre-encoded by the LLM backbone, the small speech decoder can devote its capacity to speech modeling" captures this division of labor.

How hidden states reach the speech decoder. For each text token that the LLM generates and that should be spoken, the LLM produces a hidden state vector of dimension 4,096 (the output of the final transformer layer at that token position). This hidden state is passed through a backbone-to-decoder projector—a two-layer MLP with ReLU activation, dimensions 4096 → 768 → 768, totaling 10.5M parameters. The projector serves two purposes: it reduces the dimensionality from the LLM's 4,096 to the speech decoder's 768, and it learns to extract the subset of the LLM's representation that is relevant for speech generation (prosody, emotion, speaking style, pronunciation context) while discarding information specific to text semantics that the speech decoder does not need.

The projected hidden state is then summed with the speech token decoder's own token embedding at each position. This additive combination means the speech decoder sees both the linguistic and acoustic context from its own token embeddings and the rich prosodic conditioning from the LLM's hidden states, allowing it to generate speech tokens that are both phonetically accurate and expressively appropriate.

System prompt design for voice cloning. The paper notes that MiniCPM-o 4.5 supports "multimodal system prompts that contain both text and reference audio." This works because the end-to-end architecture allows reference audio tokens (from a short clip of a target speaker's voice) to be inserted into the input sequence alongside text instructions. The LLM can attend to these reference audio tokens when generating hidden states for speech output, encoding the target speaker's vocal characteristics (timbre, pitch range, speaking rhythm) into the hidden states that condition the speech decoder. This enables zero-shot voice cloning without a separate speaker embedding module—the LLM's cross-attention over the reference audio does the speaker adaptation implicitly.

Speech Token Decoder: Lightweight Autoregressive Speech Generation

The speech token decoder converts text-token-aligned hidden states from the LLM into discrete speech tokens that can be synthesized into audio waveforms.

Architecture. The decoder is a Llama-based transformer with approximately 0.3B total parameters, split into two components:

Text embedding layer (116.8M parameters): An embedding table of size 152,064 × 768. The vocabulary size (152,064) is much larger than the LLM's text vocabulary (151,748) because it includes both standard text tokens and special tokens used during speech generation.
Transformer decoder (188.8M parameters): 20 layers, hidden dimension 768, 12 attention heads, 12 key-value heads (no GQA savings here—all heads are full), FFN dimension 3,072, SiLU activation. Maximum context length is 4,096 tokens. This is a relatively small transformer by modern standards—comparable to a GPT-2 Medium-sized model—which is possible because it only needs to model the local acoustic structure of speech given strong conditioning from the LLM.

Speech token representation. The decoder generates discrete tokens from a speech codebook of size 6,562, with a single codebook (no multi-codebook quantization). This uses S3 tokens (Supervised Semantic Speech tokens; Du et al., 2024), which are semantic-level discrete representations of speech learned through self-supervised or supervised training. The token frame rate is 25 tokens per second, meaning every token represents approximately 40 milliseconds of speech audio. This is the standard frame rate for discrete speech token systems—it is high enough to capture fine acoustic detail (phoneme transitions, voice quality) but low enough to be generated autoregressively with reasonable computational cost.

Why a single codebook? Many speech generation systems use multiple codebooks (e.g., 4–8 codebooks each generating tokens at 25 Hz, with the codebooks representing different aspects of the audio like coarse vs. fine spectral features). Using a single codebook simplifies generation (no need for codebook interleaving or hierarchical decoding) and reduces the token rate, but places more burden on the codebook to capture all acoustic detail in a single discrete token per timestep. The S3 token design is chosen to make this feasible by focusing on semantic-level features that are more compressible than raw acoustic features.

Maximum context window. The 4,096-token context window at 25 tokens/second corresponds to approximately 164 seconds (2 minutes 44 seconds) of speech context. This is sufficient for typical conversational turns and short monologues but would be insufficient for very long-form speech generation (e.g., audiobook narration). For such use cases, the streaming architecture of the flow-matching decoder (described next) handles the long-form generation by processing tokens in a sliding window.

Waveform Synthesis: Streaming Flow-Matching Decoder

The final stage converts the discrete S3 speech tokens into continuous audio waveforms that can be played through speakers.

Flow-matching decoder. The paper uses a streaming flow-matching decoder following CosyVoice 2 (Du et al., 2024). Flow matching is a generative modeling technique where a simple base distribution (e.g., Gaussian noise) is gradually transformed into the target distribution (a speech waveform) through a learned velocity field. At inference time, the decoder starts with random noise and iteratively denoises it, guided by the conditioning signal (the speech tokens and the reference audio), to produce a waveform. The "streaming" aspect means the decoder can begin generating audio output before all speech tokens have been received—critically important for low-latency full-duplex interaction where speech playback must start within milliseconds of the LLM producing text.

Reference audio conditioning. The decoder uses the reference audio from the multimodal system prompt for voice timbre conditioning. The reference audio is encoded and used to condition the flow-matching process, ensuring the generated speech matches the target speaker's voice characteristics. This is the mechanism that enables voice cloning: by providing a short reference clip (typically 3–10 seconds) of the desired speaker, the decoder produces speech that sounds like that speaker while saying the content determined by the speech tokens.

Integration with the streaming pipeline. The speech token decoder produces tokens at 25 Hz. These are immediately passed to the flow-matching decoder, which begins synthesizing audio in small chunks (typically 40–80 millisecond chunks for streaming operation). The end-to-end latency from text token generation to audible speech is designed to be small enough (ideally under 200 milliseconds) that the full-duplex interaction feels natural—the model's speech output tracks the evolving environment in near-real-time.

Omni-Flow: The Core Full-Duplex Framework

Omni-Flow is the paper's central conceptual and technical contribution. It is not a single algorithm but a framework for organizing multimodal interaction that defines: (1) what the input and output streams are, (2) how they are aligned in time, (3) how they are serialized into a sequence consumable by a causal language model, and (4) how the model decides when to speak.

Motivation: the Blocked-I/O problem. Existing streaming multimodal models (including GPT-4o, Gemini Live, and Qwen3-Omni) operate in what is essentially a turn-based mode at the architectural level, even if the user experience feels "streaming." The model receives audio/video input, processes it, generates a complete response, and only then begins processing the next chunk of input. During generation, the model is effectively deaf and blind—new visual or auditory information cannot influence the ongoing speech output. This is what the paper calls "Blocked-I/O" (Figure 3). The consequence is not just latency but a fundamental limitation on interaction quality: the model cannot interject, adjust mid-utterance when the user looks confused, or notice and comment on events that occur while it is speaking.

Omni-Flow solves this by partitioning continuous time into fine-grained windows and interleaving perception and generation within each window, so the model repeatedly updates its understanding of the environment while producing output.

Three time-aligned streams. The paper identifies three information streams that coexist during full-duplex interaction:

env-visual: Live visual observations of the environment, arriving as a sequence of video frames with timestamps.
env-audio: The acoustic scene, including user speech when present and ambient sounds, arriving as a waveform with timestamps.
out-stream: The assistant's text and speech output, which is generated by the model but must be temporally coordinated with perception.

A critical philosophical shift described in Section 3.1: under Omni-Flow, "user requests are no longer treated as a privileged conversational role, but instead become part of the continuously observed world state, entering primarily through env-audio." This reframing removes the need for explicit turn-detection. The model observes the audio stream continuously and, when it detects a question or a conversational opening, can choose to respond—not because a VAD module triggered, but because the ongoing context indicates that speaking is appropriate.

Time chunking and serialization. The continuous interaction is partitioned into time windows of fixed duration t. The paper evaluates three values: 1.0 second, 0.2 seconds, and 0.1 seconds, eventually selecting 1.0 second as the optimal balance between responsiveness and stability (Table 1). For the $k$ th time chunk (spanning time $[(k-1)t, kt)$ ):

Visual encoding: Any video frames with timestamps falling in this chunk are encoded by the SigLIP ViT and resampler into a sequence of visual tokens $\mathbf{v}_k$ . Each frame contributes 64 tokens (plus slice tokens if the image is partitioned).
Audio encoding: Any audio within this chunk is processed by the Whisper encoder and MLP projector into a sequence of audio tokens $\mathbf{a}_k$ , at 10 tokens per second of audio.
Output generation: The model generates an output token sequence $\mathbf{o}_k$ . When the model determines it should remain silent, $\mathbf{o}_k$ contains only the special [listen] token. When it should speak, $\mathbf{o}_k$ contains text tokens representing the assistant's speech for this time chunk.
Group formation: These three token sequences are concatenated into a single group $\mathbf{g}_k = [\mathbf{v}_k; \mathbf{a}_k; \mathbf{o}_k]$ .
Sequence construction: Groups for consecutive time chunks are concatenated to form the full interaction sequence: $[\mathbf{g}_1, \mathbf{g}_2, \mathbf{g}_3, \ldots]$ .

The critical ordering within each group—visual tokens first, then audio tokens, then output tokens—is not arbitrary. It ensures the model conditions its output on the most recent observations. When generating $\mathbf{o}_k$ , the model has already processed $\mathbf{v}_k$ and $\mathbf{a}_k$ (due to causal attention), so its speech is grounded in the current chunk's perceptual information. In the next chunk $k+1$ , the model first processes $\mathbf{v}_{k+1}$ and $\mathbf{a}_{k+1}$ (new observations) before generating $\mathbf{o}_{k+1}$ , so its output incorporates any changes in the environment that occurred during the previous chunk.

What happens as chunk size decreases. The paper's ablation study (Table 1) reveals a fundamental tradeoff. Reducing t from 1.0s to 0.2s or 0.1s increases temporal responsiveness—the model updates its understanding of the environment more frequently and can react faster. However, shorter chunks leave less "modeling budget" per chunk. With a 0.1-second chunk, the model receives only about 1 audio token (10 Hz × 0.1s) and perhaps a partial visual observation, and it must decide whether to speak and what to say based on very limited context. The paper reports substantial degradation: on MMLU, performance drops from 0.65 at 1.0s to 0.32 at 0.1s (a 51% relative decline). The degradation on AlpacaEval is even steeper (3.56 → 2.40). The 1.0-second chunk provides enough contextual information in each window for the model to make stable decisions while still enabling reasonable responsiveness for most interaction scenarios.

Boundary explicitness. The ablation compares two approaches for separating consecutive groups $\mathbf{g}_k$ and $\mathbf{g}_{k+1}$ :

Explicit boundaries: Special delimiter tokens are inserted at group boundaries, explicitly marking where one time chunk ends and the next begins.
Implicit boundaries: No special tokens; the model must infer group boundaries from content patterns.

Explicit boundaries consistently outperform implicit boundaries across all metrics in Table 1 (e.g., AlpacaEval 3.60 vs. 3.31 with 1.0s chunks under LT control). The paper interprets this as evidence that "distinguishing newly observed inputs from newly generated outputs is a nontrivial problem, and making this structure explicit can reduce the burden on the model." In other words, the model benefits from knowing unambiguously where perceptual observations end and where its own generation begins—this temporal structure is not easily inferred from content alone, especially when the model is simultaneously making decisions about whether to speak.

Control formulation: Listen-Speak vs. Listen-Text. This is arguably the most nuanced design choice in Omni-Flow. The paper explores two formulations for how the model decides to speak:

Listen-Speak (LS): The model first predicts a binary control token—either [listen] or [speak]—at the beginning of each chunk's output section. If [listen], no further tokens are generated for that chunk. If [speak], the model proceeds to generate text tokens. The decision of whether to speak is explicitly separated from what to say.
Listen-Text (LT): The model directly predicts either [listen] or regular text tokens in a shared output space. There is no explicit binary decision; the model's choice to remain silent is just one possible token among many (like a word), and the choice to speak is simply the generation of a text token instead of [listen].

The results in Table 1 show LS consistently outperforming LT. With 1.0s chunks and explicit boundaries, LS achieves MMLU 0.65 vs. LT's 0.56, and AlpacaEval 3.56 vs. 3.60 (roughly comparable on AlpacaEval but much better on knowledge-intensive tasks).

Why LS outperforms LT. The paper's explanation is that "deciding whether to speak should be decoupled from deciding what to say, and entangling both in a single prediction step makes full-duplex interaction harder to learn." This makes sense from a representation learning perspective. In the LT formulation, the model's output distribution at each position must simultaneously model: (a) the probability that speaking is appropriate given the context, and (b) the probability distribution over possible things to say if speaking. These are fundamentally different kinds of decisions—one is a binary gating decision, the other is open-ended language generation—and combining them in a single softmax over the full vocabulary forces the model to allocate probability mass between the [listen] token and all possible text tokens, which may create conflicting gradients during training. The LS formulation, by separating the binary control prediction from text generation, allows each sub-task to be modeled with an appropriate inductive bias: the control token can be learned as a simple binary classifier over the LLM's hidden state, while the following text tokens benefit from the full generative language modeling objective.

Proactive behavior as an emergent property. The paper emphasizes that proactive behavior arises naturally from this formulation. Because the model is trained on data where appropriate behavior includes initiating speech (e.g., offering reminders, commenting on scene changes), and because the LS formulation gives the model explicit control over when to speak in every time chunk, the model learns to output [speak] tokens when the ongoing context warrants proactive intervention. There is no separate "proactive behavior module"—the ability to initiate interaction is simply the learned policy for the listen/speak decision.

Time-Aligned Interleaving (TAIL): Keeping Speech Temporally Aligned with Perception

Even with Omni-Flow's time-chunked serialization, a subtle but critical problem remains: temporal alignment between what the model says and what is happening in the environment. The issue, described in Section 3.4, arises from the mismatch between text generation time and speech playback time.

The fundamental mismatch. When the LLM generates text for a 1-second time chunk, it might produce text that takes more than 1 second to vocalize. For example, if the model generates "the player in the red jersey is dribbling the ball towards the goal post" during chunk k, this text might take 2.5 seconds to speak at natural speed. During those 2.5 seconds, the environment continues to evolve—the player might shoot, the ball might be intercepted, the crowd might react. But the speech the user hears during those 2.5 seconds was generated based on observations from chunk k and earlier, which are now stale. By the time the user hears the end of the sentence, the world might look very different.

Prior approaches and their failures. The paper identifies two common strategies in existing streaming speech systems (Figure 5):

Non-interleaving (Figure 5a): The model generates all text first, then synthesizes speech. This maximizes speech quality (the full sentence context is available for prosody) but completely decouples speech playback from ongoing perception. The speech heard at any moment could correspond to text generated many seconds ago. In a full-duplex setting where the environment is changing, this is unacceptable—the model would constantly be saying things that are no longer true or relevant.
Fixed text-speech ratio interleaving (Figure 5b): The model alternates between generating N text tokens and M speech tokens, where N:M is a fixed ratio. This keeps text and speech closer together than the non-interleaving approach, but assumes a fixed relationship between text tokens and speech duration. In reality, different text tokens have vastly different vocalization durations: the word "a" might take 0.1 seconds, while "dribbling" might take 0.5 seconds. A fixed ratio cannot capture this variability, so speech will inevitably drift relative to the ideal timeline.

Both approaches share a common failure mode: they allow the text stream to run ahead of the speech playback, so the content being spoken reflects an outdated model state.

TAIL: Adaptive chunk-wise text generation. The paper's proposed solution, Time-Aligned Interleaving, generates text on a per-chunk basis but with an adaptive constraint: the amount of text generated in chunk k should be calibrated so that, after vocalizing all text generated up through chunk k, the speech playback time approximately equals k × t (the end of the kth time chunk). In other words, the speech stream should not lag behind or run ahead of real time by more than a small margin.

Formally, for chunk k, the model considers the accumulated playback progress—how much speech has been generated and how much of it has been vocalized so far. If previous chunks have already introduced a slight playback delay (because their text took longer to speak than the chunk duration allowed), the model adaptively generates fewer text tokens in chunk k to let speech catch up. Conversely, if speech playback is ahead of schedule (because previous chunks contained short, fast-to-speak utterances), the model can generate more text without causing misalignment.

The paper describes the supervision construction for TAIL:

"We construct TAIL supervision from full-duplex streaming training data by collecting the start and end times of each text token. Tokens whose start times fall into $[(k-1)t, kt)$ , together with their corresponding speech tokens, are assigned to the $k$ th Omni-Flow chunk."

This means the training data itself encodes the natural temporal alignment between text and speech. When a human speaker naturally paces their speech to the visual scene, their text tokens have start times that align with the environment. By training on this alignment, the model learns to produce a similar pacing—generating text at a rate that keeps speech playback synchronized with perception.

The look-ahead mechanism. A complication: accurate speech generation sometimes requires limited future text context. The paper gives the example of "the apple" vs. "the car"—the pronunciation of "the" depends on whether the following word begins with a vowel or consonant. If speech tokens were generated strictly aligned with text tokens (speech for "the" generated at the same time as the text token "the"), the model would not know which pronunciation to use until it sees the next word.

TAIL addresses this with a bounded look-ahead mechanism: the speech tokens for the last few text tokens in chunk k are deferred to chunk k+1. This provides a small window of future context (the identity of the upcoming word) without letting text run substantially ahead of speech. The paper states:

"the speech tokens of the last few text tokens in chunk k are deferred to chunk k + 1, while the remaining tokens are spoken in chunk k."

The result is that speech generation maintains high quality (correct coarticulation, natural prosody) while still keeping the text stream closely aligned with speech playback and, consequently, with ongoing perception.

Empirical comparison of interleaving modes (Table 10). The paper evaluates three modes on the SeedTTS test set:

No interleaving: Full text generation, then speech synthesis. Achieves 1.44 CER on Chinese, 2.70 WER on English.
Fixed-text interleaving: Chunked generation with fixed text-to-speech ratio. Achieves the best CER/WER: 0.86 CER on Chinese, 2.38 WER on English. The improvement over no-interleaving suggests that chunked processing somehow benefits speech quality—possibly because the speech decoder sees shorter, more coherent segments.
Dynamic-text interleaving (TAIL) : The adaptive approach designed for full-duplex alignment. Achieves 1.04 CER on Chinese, 3.93 WER on English. The slight degradation in recognition accuracy (especially English WER) compared to fixed-text interleaving reflects the constraint TAIL imposes: the model cannot always generate as much text as it "wants" in each chunk; it must sometimes shorten its output to maintain temporal alignment. The paper positions this as "a practical trade-off between streaming interaction and speech generation quality."

Why this trade-off is acceptable. In full-duplex interaction, temporal relevance of speech content is more important than perfect pronunciation. A model that speaks with 3.93 WER but says things that are temporally aligned with the current environment is more useful than a model with 2.38 WER that is constantly commenting on events that happened 5 seconds ago. TAIL prioritizes alignment over acoustic perfection, which is the right priority for the full-duplex use case.

The data pipeline for MiniCPM-o 4.5 involves constructing training examples that span vision, speech, text, and their combinations, with particular attention to the full-duplex interaction data that enables the Omni-Flow framework.

Speech data (Section 4.1). The speech data comes from two sources designed to provide both broad coverage and high-quality controllable generation:

Large-scale natural speech data: Millions of hours of unlabeled speech from diverse sources are processed through a pipeline integrating multiple open-source components:

Silero VAD (Team, 2024) for voice activity detection—identifying where speech is present vs. silence.
Whisper (Radford et al., 2022) for automatic speech recognition—transcribing speech to text.
Paraformer (Gao et al., 2023) as an alternative ASR system, likely for ensemble or verification.
Speaker diarization (Han et al., 2024) for identifying who is speaking when in multi-speaker audio.
Music source separation (Défossez et al., 2021) for isolating speech from background music/noise.

The output of this pipeline includes training sets for zero-shot TTS (text-to-speech with unseen voices), ASR (speech-to-text), and multi-turn multi-speaker dialogue. The paper emphasizes that the corpus "encompasses a broad range of different speakers, accents, and conversational patterns"—this diversity is crucial for training a model that can understand and generate speech from any speaker, not just those in a curated dataset.

Spoken dialog data: This is a smaller but higher-quality dataset explicitly designed for instruction-following speech interaction. The construction process involves:

A text-based LLM generates colloquial, instruction-following dialogues from diverse seed queries. These dialogues are designed to cover various interaction patterns: question answering, instruction execution, multi-turn conversation, emotional expression, and proactive behavior.
A subset of these dialogues is re-recorded by professional voice actors under studio conditions. Critically, the voice actors are instructed to "deliver in a conversational style rather than reading scripts verbatim, balancing structured content with improvised expression while varying emotion, speaking rate, and emphasis under a consistent vocal identity." This means the audio is not simply a reading of the text—it contains natural disfluencies, prosodic variation, and emotional expression that make it more representative of real human speech.

The resulting corpus covers "instruction-following TTS, question answering, and multi-turn natural dialogue." By combining large-scale diverse data (for broad coverage) with high-quality studio-recorded data (for controllable, natural-sounding generation), the training covers both the breadth and depth needed for a production-quality speech model.

Vision-language data (Section 4.2). Building on the data system of MiniCPM-V 4.5, the vision-language data is expanded in several dimensions:

High-quality knowledge and alignment data: The CapsFusion pipeline (Yu et al., 2024) is updated with a better generator model to produce more informative image captions, and the filtering process is improved by "improving image-text relevance estimation"—likely using a CLIP-style or reward-model-based relevance scorer to filter out caption-image pairs where the caption does not accurately describe the image content.

Complex document and OCR data: The paper introduces a relevance-aware masking strategy for document understanding. Instead of randomly masking text regions in document images (a common augmentation), regions that are "more relevant to figures and charts in document images" are prioritized for masking. The intuition: if a question about a document chart can be answered purely from the surrounding text without looking at the chart, the training signal for visual grounding is weak. By masking text that is highly correlated with the visual content, the model is forced to attend to the chart or figure to answer the question. This "reduces the proportion of training cases that can be solved primarily from textual context alone," strengthening the model's genuine multimodal reasoning rather than text-only shortcut learning.

Real-world scenarios data: The data curation focuses on "natural and diverse query patterns"—likely meaning that instead of templated questions like "What is in this image?", the queries reflect how real users actually interact with multimodal systems: open-ended descriptions, follow-up questions, comparisons, and requests that mix modalities. Two quality improvements are applied:

Short, direct-answer responses are rewritten into "detailed, chain-of-thought-style rationales"—teaching the model to show its reasoning rather than just providing answers.
A reward-model-based filtering pipeline is applied to ensure "overall data quality and alignment with human preferences"—likely using a model like UltraRM or a custom reward model to score and filter training examples.

Dense video perception data: A "dense video captioning dataset" is constructed that provides "continuous, fine-grained descriptions of temporal events, human actions, and complex scene transitions." Unlike standard video captioning datasets that provide a single caption for the entire video, dense captioning provides captions at regular intervals or at event boundaries, enabling the model to learn the temporal structure of video content rather than treating a video as a bag of frames.

Text-only data: High-quality text instruction data from MiniCPM 4.1 post-training is incorporated to "maintain robust linguistic capabilities." This is crucial because multimodal training can cause catastrophic forgetting of text-only capabilities, and maintaining a text data mixture during training ensures the model remains a capable language model in addition to its multimodal abilities.

Omni-modal full-duplex data (Section 4.3). This is the most novel data component, designed specifically to train the Omni-Flow interaction paradigm:

Large-scale web audio-video data: A large corpus of web videos (e.g., from platforms like YouTube) is collected to provide "broad coverage of real-world full-duplex scenarios." Natural videos often contain exactly the kind of continuous multimodal interaction that Omni-Flow models: people talking while something happens in the visual scene, ambient sounds that relate to visual events, multi-speaker conversations with overlapping speech. The data is filtered to remove:

Segments dominated by single-speaker speech (no rich audio-visual interaction).
Segments with "weak audio-visual relevance" (where the audio and video are unrelated, e.g., background music over unrelated footage).
Segments with OCR-based subtitles (removed using PaddleOCR 3.0, Cui et al., 2025), which would provide text shortcuts.
Segments failing talking-head detection (Chen et al., 2025), which filters out static talking-head videos that lack dynamic visual content.
Segments with low-information or noisy ASR-derived transcripts.

Each training sample from this corpus contains the full visual input, audio input, output text, and output speech, "where each piece of information is tagged with a time index." This millisecond-level time indexing is what enables the TAIL supervision described earlier: the model can learn the precise temporal alignment between visual events, speech content, and speech playback timing.

Full-duplex task data: A smaller set of high-quality, manually constructed samples designed to teach specific full-duplex capabilities that may not appear naturally in web data. The paper mentions "multiple scenarios" with corresponding instruction-following data, supporting capabilities like "continuous scene description and proactive reminding." These are likely carefully scripted interactions where the model must:

Provide ongoing commentary as a scene unfolds.
Notice specific events and initiate speech without being prompted.
Adjust its speech based on visual changes.
Decide when to remain silent vs. when to interject.

These task-specific samples provide the "target full-duplex capabilities that require more precise interaction" that large-scale web data alone cannot guarantee.

Training: Staged Integration of Speech into a Vision-Language Foundation

The training strategy is designed around a central challenge: how to add speech input and output capabilities to a strong pre-existing vision-language model without degrading existing capabilities or training instability.

Starting point. The model is initialized from a pretraining checkpoint of MiniCPM-V 4.5, which already has strong vision-language capabilities. The Whisper encoder is initialized from pretrained weights, while new modules—the audio projector, the LLM-to-speech projector, and the speech token decoder—are randomly initialized.

Stage 1: Speech Pretraining (Section 5.1). In this first stage, only the newly added speech-related modules are trained; all pretrained components (visual encoder, LLM backbone, etc.) are frozen. Training uses a mixture of speech understanding data (ASR, speech translation, audio understanding) and speech generation data (TTS, spoken dialogue).

The objective for speech understanding tasks is standard next-token prediction: the LLM receives audio tokens and must generate the corresponding text transcription. Because the LLM backbone is frozen, the audio projector must learn to map Whisper features into a representation that the LLM can interpret as if they were text tokens representing the same content. This is effectively training an adapter that translates between the audio modality and the LLM's text-centric representational space.

The objective for speech generation is similarly next-token prediction, but applied to the speech token decoder: given the LLM's hidden states (from a frozen backbone), the speech decoder must generate the correct speech tokens. The LLM-to-speech projector must learn to extract prosodic and acoustic information from the LLM's hidden states, and the speech decoder must learn to translate this conditioning into natural-sounding speech.

The technical challenge at this stage is preventing catastrophic forgetting in the pretrained components while learning the new modality mappings. By keeping the LLM backbone and visual encoder frozen, the speech modules are forced to adapt to the existing representational space rather than pulling the LLM's representations toward speech-specific features that might conflict with vision-language processing.

Stage 2: Joint Pretraining (Section 5.2). All parameters are unfrozen, and the model is trained on a balanced mixture of vision-language, speech, and omni-modal data. The key innovation in this stage is data organization:

"To stabilize optimization, we assign different modality combinations to different data-parallel ranks, ensuring a fixed data ratio at every training step."

In distributed training, each GPU processes a subset of the batch. By assigning specific modality combinations to specific GPUs (e.g., GPU 0 always gets vision-language samples, GPU 1 always gets speech samples), the ratio of modalities in each training step's gradient update is guaranteed to be constant, regardless of random sampling. This prevents training runs where some steps are mostly vision-language and others are mostly speech, which could cause oscillatory optimization as the model adapts to different modality distributions in successive steps.

The training data mixture at this stage includes not just conventional turn-based samples but also "proactive and full-duplex interaction data, where text tokens are aligned with speech and visual signals on a shared timeline." This is the stage where the Omni-Flow serialization format is introduced, and the model learns to process time-aligned multimodal sequences and generate listen/speak control tokens.

The training objective is a unified next-token prediction objective across all modalities. The LLM predicts text tokens (including the [listen] / [speak] control tokens); the speech token decoder predicts speech tokens; and the whole system is trained end-to-end with gradients flowing through all components.

Stage 3: Joint Supervised Fine-Tuning (Section 5.3). This stage activates omni-modal capabilities and strengthens instruction following through two phases:

Large-scale instruction tuning: The model is fine-tuned on a broad set of instruction-following data across modalities. This teaches the model to respond appropriately to user queries, follow formatting instructions, and produce helpful, harmless responses.
High-quality human-annotated tuning: A smaller set of carefully curated, human-annotated examples further refines behavior. This phase focuses on the nuances of natural interaction: appropriate tone, helpfulness, safety, and the specific conversational patterns expected in full-duplex settings.

Resolution and frame rate augmentation. A notable training detail: omni-modal data is augmented with "varying resolutions and frame rates, randomly setting the maximum frame resolution to 0.2–0.4 megapixels and sampling the frame rate uniformly from 1–5 FPS." This teaches the model to operate correctly across a range of visual input qualities, which is essential for deployment where network conditions, camera quality, and available compute may vary. At inference time, the resolution and frame rate can be explicitly set to trade off between visual detail and computational cost—a "flexible quality-efficiency trade-off" that the model has learned to handle through its training augmentation.

Stage 4: Reinforcement Learning (Section 5.4). The final stage applies reinforcement learning to improve reasoning, instruction following, and hallucination reduction:

GRPO (Group Relative Policy Optimization) : The paper uses GRPO (Shao et al., 2024), a variant of policy gradient methods where advantage is computed relative to the mean reward of a group of sampled responses to the same prompt. This is more stable than standard PPO because the advantage normalization within each group removes the need for a separate value function (critic).

Length reward design. A major emphasis of this stage is on response length efficiency—preventing the model from producing unnecessarily verbose responses, especially in "thinking" mode where chain-of-thought reasoning can become extremely long. The paper introduces a smooth length reward adapted from Kimi K1.5 (Team, 2025):

$r_{\text{len}}(i) = \begin{cases} s_i, & r_i = 1, \\ \min(0, s_i), & r_i = 0, \end{cases} \quad \text{where} \quad s_i = \left(0.5 - \frac{\ell_i - \ell_{\text{min}}}{\ell_{\text{max}} - \ell_{\text{min}}}\right) \times \min\left(1, \frac{\ell_{\text{max}} - \ell_{\text{min}}}{\tau}\right)$

where $r_i \in \{0, 1\}$ is the correctness indicator (1 if the response is correct, 0 otherwise), $\ell_i$ is the length of response $i$ (in tokens), $\ell_{\text{min}}$ and $\ell_{\text{max}}$ are the minimum and maximum response lengths across all sampled responses to the same prompt, and $\tau$ is a temperature parameter that controls the scaling of the reward.

What it computes: The length reward is a function that penalizes long responses and rewards short ones, but only when length differences are meaningful.

The core term $0.5 - \frac{\ell_i - \ell_{\text{min}}}{\ell_{\text{max}} - \ell_{\text{min}}}$ maps each response's length to a normalized score between +0.5 (shortest response) and -0.5 (longest response). This is a linear interpolation based on where the response falls in the min-max range.
The $\min\left(1, \frac{\ell_{\text{max}} - \ell_{\text{min}}}{\tau}\right)$ factor downscales the reward when the difference between the longest and shortest response is small relative to $\tau$ . If all responses to a prompt are similar in length, the length penalty is reduced—there is no point in strongly penalizing length differences that are essentially noise.
The piecewise structure handles correctness: if the response is correct ( $r_i = 1$ ), the full $s_i$ is applied—short correct responses are rewarded, long correct responses are penalized. If the response is incorrect ( $r_i = 0$ ), only the negative portion of $s_i$ is applied—long incorrect responses are penalized, but short incorrect responses are not rewarded (because $\min(0, s_i)$ will be 0 if $s_i$ is positive). This prevents the model from learning to game the system by producing very short wrong answers.

Why this form over the Kimi K1.5-style reward (Table 9, Figure 6). The paper's ablation reveals the problem with the original Kimi K1.5 length reward. The K1.5-style reward "aggressively reduces the response length in thinking mode by 50.7%, but also decreases the benchmark average from 73.5 to 73.0." The training curves in Figure 6 show the mechanism: the K1.5 reward causes a "clear slowdown and even slight degradation in training accuracy in the later stage, suggesting that an overly aggressive length reward can conflict with the accuracy reward and suppress further optimization."

The proposed smooth length reward avoids this by being less aggressive. It achieves "a more moderate length reduction of 35.3% on thinking tasks, while improving the benchmark average to 74.3." The training curve remains closer to the baseline without length reward, indicating that the length reward does not interfere with the accuracy optimization. The smoothing factor $\min\left(1, \frac{\ell_{\text{max}} - \ell_{\text{min}}}{\tau}\right)$ is key: when length differences are small, the reward is downscaled, preventing the model from chasing tiny length differences that are irrelevant to actual efficiency.

The paper also notes that the length reward is not applied for the first 480 training steps. This allows the model to first learn to produce correct answers without length pressure, after which the length reward is introduced to prune unnecessary verbosity. Starting length pressure too early could prevent the model from learning to reason at all.

Accuracy reward and judge model. For accuracy rewards, the paper combines "rule-based verification with an efficient judge model" (CompassVerifier, Liu et al., 2025). Rule-based verification works for math problems (checking final answers), code (running test cases), and factual QA (exact match). The judge model handles more subjective criteria like response quality, helpfulness, and format compliance where rule-based checking is impossible.

RLAIF-V for hallucination reduction. The final RL stage applies RLAIF-V (Yu et al., 2024) to reduce hallucinations in visual scenarios. RLAIF-V uses AI feedback—a separate model evaluates whether the generated response is faithful to the visual input—to train the model to avoid generating content not supported by the image. The paper notes an interesting transfer property: "hallucination mitigation learned from image-text data transfers effectively to omni-modal full-duplex interaction, reducing hallucinations in streaming settings as well." This suggests that visual faithfulness is a generalizable skill that, once learned on static images, improves performance even in temporally complex streaming scenarios.

Efficient Inference: Edge Deployment of Full-Duplex Interaction

Section 7 describes the inference optimization that makes real-time full-duplex interaction practical on consumer hardware. This is not a minor engineering detail but a critical component of the paper's contribution: full-duplex interaction is only impactful if it can run without datacenter-scale compute.

Standard vLLM efficiency (Table 11). Using the standard vLLM serving framework (Kwon et al., 2023) on a single NVIDIA RTX 4090 (24GB VRAM):

BF16 precision: Qwen3-Omni-30B-A3B runs out of memory ("OOM")—its 30B parameters (even with 3B active) cannot fit. MiniCPM-o 4.5 achieves 154.3 tokens/second throughput with 19 GB memory usage, and first-token latency (with 64-frame visual input) of 0.59 seconds.
INT4 precision: Both models fit. Qwen3-Omni achieves 147.8 tokens/second and 0.98 seconds first-token latency using 20 GB. MiniCPM-o 4.5 achieves 212.3 tokens/second (1.44× faster) and 0.58 seconds first-token latency (1.69× faster) using only 11 GB (1.82× less memory).

The throughput advantage of MiniCPM-o 4.5 (212.3 vs. 147.8 tokens/s) despite having 9B vs. 30B total parameters demonstrates the efficiency of the architecture: the LLM backbone generates only text tokens, not speech tokens, so the number of autoregressive decoding steps per second of interaction is much lower.

llama.cpp-omni: Custom inference for full-duplex streaming (Table 12). The standard PyTorch implementation is too slow for real-time full-duplex interaction on edge devices. The paper develops llama.cpp-omni, an extension of the llama.cpp (ggml-org, 2023) inference framework—a C/C++ implementation optimized for CPU and GPU inference on consumer hardware.

The key metric for real-time speech interaction is Real-Time Factor (RTF) —the ratio of processing time to audio duration. An RTF of 1.0 means processing 1 second of audio takes 1 second (barely real-time); values below 1.0 mean faster than real-time, which is necessary to absorb occasional spikes in computation.

On an RTX 4090:

PyTorch INT4: RTF = 1.26 (slower than real-time—unusable for full-duplex), 14 GB memory.
llama.cpp-omni FP16: RTF = 0.27 (3.7× real-time), 19 GB memory.
llama.cpp-omni INT4: RTF = 0.21 (4.8× real-time), 11 GB memory.

On DGX Spark (a datacenter GPU like an H100 or similar):

PyTorch BF16: RTF = 2.43 (very slow), 26 GB memory.
PyTorch INT4: RTF = 1.27 (barely real-time), 14 GB.
llama.cpp-omni FP16: RTF = 0.46, 19 GB.
llama.cpp-omni INT4: RTF = 0.20, 11 GB.

The INT4 RTF of 0.20–0.21 means the model can process input and generate speech 5× faster than real-time, leaving ample headroom for the visual encoding, network communication, and waveform synthesis that are also part of the full-duplex pipeline. The memory footprint of 11 GB means the entire model fits within the 12 GB RAM budget that the paper advertises for edge deployment.

Cross-platform compatibility. The paper notes that llama.cpp-omni has been "validated across different operating systems, including macOS, Windows, and Linux." This is significant because it means the full-duplex capability is not locked to Linux servers with NVIDIA GPUs—it can run on MacBooks (with Apple Silicon's unified memory providing high-bandwidth access), Windows desktops with consumer GPUs, and standard Linux workstations. A "lightweight demo system" allows users to deploy the model on their own hardware without cloud dependencies.

The practical significance of 12GB RAM. Modern smartphones have 8–16 GB of RAM. High-end laptops commonly have 16–32 GB. MiniCPM-o 4.5's 11 GB memory footprint in INT4 means it can plausibly run on: a MacBook Pro with 16 GB RAM (leaving 5 GB for the OS and other applications), a high-end Android phone with 12 GB RAM, or a mid-range gaming PC. This is the hardware that real users have, not the A100 clusters that most large models require. The paper's emphasis on edge deployment is not just an efficiency flex—it enables always-on, proactive AI in privacy-sensitive environments where data cannot be sent to the cloud, or in connectivity-limited settings where cloud dependence would make the system unreliable.

4. Key Insights and Innovations

Innovation 1: Interaction Paradigm as a First-Class Architectural Problem, Not Just a Modality or Latency Problem

The central intellectual move of this paper is reframing the challenge of building human-like multimodal AI. Before MiniCPM-o 4.5, the dominant narrative in the MLLM literature was that progress toward natural interaction required improvements along two axes: adding more modalities (text → images → video → audio → speech output) and reducing latency (faster inference, streaming outputs). Models like GPT-4o, Gemini Live, and Qwen3-Omni embody this philosophy—they achieved impressive reductions in response time and added speech generation, but all operate within what the paper identifies as the same fundamental interaction architecture: an alternating perception-then-response cycle.

The paper's diagnosis (Figure 3) is that this architecture creates a qualitative ceiling that no amount of modality expansion or latency reduction can breach. The "Blocked-I/O" problem is not a performance shortfall but a structural constraint: if the model cannot process new environmental information while generating speech, it is fundamentally incapable of behaviors that define human conversation—interrupting, adjusting mid-utterance based on listener feedback, noticing and commenting on environmental events that occur during one's own speech, or initiating interaction proactively rather than in response to an explicit trigger. These are not "nice-to-have" features; they are constitutive of what we mean by real-time interaction.

What makes this framing distinctive is that it identifies the interaction paradigm itself—the temporal logic governing when perception and response occur relative to each other—as the primary bottleneck, not modality coverage (everyone has audio I/O now) or speed (streaming is table stakes). The paper argues, implicitly but clearly, that the field has been optimizing within the wrong search space. Adding more modalities to a turn-based architecture yields a more capable turn-based model, not a model that can truly participate in continuous conversation. This is analogous to the distinction between making a faster horse versus inventing a car—the alternating-phase paradigm is not something you can improve your way out of.

The significance of this reframing extends beyond this specific model. If the interaction paradigm is the bottleneck, then the research agenda shifts from "how do we add modality X faster" to "how do we design temporal architectures that couple perception and response in principled ways?" Omni-Flow is one answer to that question, but the paper explicitly positions it as a general framework (Section 3), not a model-specific optimization. This suggests a new subfield: interaction architecture design, analogous to how network protocol design emerged as a distinct discipline once it became clear that fast hardware alone couldn't solve the problems of reliable communication.

Evidence for this reframing's validity comes from the observation that prior models with strong multimodal capabilities and low latency (like Qwen3-Omni) still cannot engage in full-duplex interaction—the limitation is not capability or speed, it is architecture. MiniCPM-o 4.5's ability to outperform Qwen3-Omni-30B-A3B on omni-modal benchmarks (Table 7, e.g., JointAVBench 60.0 vs. 53.1) despite having ~1/3 the parameters is consistent with the claim that getting the interaction structure right matters as much as raw model scale.

Innovation 2: The Listen/Speak Decision as a Learned Behavior, Not an External Trigger

A quiet but profound shift in this paper is the elimination of explicit turn-detection mechanisms. In virtually all existing conversational AI systems—from Alexa to ChatGPT Voice to Qwen3-Omni—the model's decision to speak is governed by an external module, typically a Voice Activity Detector (VAD) that detects when the user has stopped talking, plus a set of rules about when responses are appropriate. The model itself has no agency over when it speaks; it is purely reactive, generating a response only when triggered.

Omni-Flow's Listen-Speak (LS) formulation makes the decision to speak an explicit prediction within the model's autoregressive generation loop. In every time chunk, the model first predicts a binary control token—[listen] or [speak]—and then, if it predicted [speak], generates text. This is a subtle architectural change with deep implications.

First, it endogenizes turn-taking. The model learns from data when speaking is appropriate—not from a hard-coded rule about silence duration, but from understanding conversational context. This means the model can learn culturally-specific turn-taking norms (different cultures have different inter-speaker gap durations), context-specific behavior (interrupting is appropriate in collaborative brainstorming but not in formal presentations), and speaker-specific patterns (some people prefer more backchanneling than others).

Second, and more importantly, it enables proactive behavior without a separate architecture. Because the model decides autonomously in each time chunk whether to speak, it can output [speak] tokens when the ongoing environmental context—not an explicit user prompt—warrants it: noticing a timer has elapsed, observing a safety hazard in a visual stream, detecting confusion in a user's facial expression. These proactive behaviors are not implemented as a separate planning module or rule engine; they emerge from the same listen/speak policy that governs all interaction decisions, trained on data where such proactive behavior is demonstrated.

This is a fundamental departure from the dominant assumption in conversational AI, which treats initiative as a separate research problem (proactive dialogue systems, mixed-initiative interaction) requiring explicit user modeling and planning. The paper suggests, through its architecture and results, that initiative can be learned as part of the same end-to-end objective that governs all interaction behavior, provided the architecture gives the model the opportunity to initiate—which turn-based architectures structurally prevent.

The ablation comparing LS and LT formulations (Table 1) provides evidence for the depth of this insight. The LT formulation, where the listen/speak decision shares a prediction space with content tokens, significantly underperforms LS (MMLU 0.56 vs. 0.65 at 1.0s chunks). The paper interprets this as evidence that "deciding whether to speak should be decoupled from deciding what to say"—but the deeper implication is that control decisions and content generation engage fundamentally different cognitive mechanisms, and conflating them in a single prediction step creates interference. This mirrors findings in cognitive science about the separation of action selection and motor planning in biological brains, and suggests that future interaction architectures should treat the decision of when to act as a distinct computational problem from what action to take.

Innovation 3: Temporal Alignment Between Speech Playback and Perception as a First-Class Design Constraint

The paper identifies and solves a problem that, to its knowledge, has not been formalized in the streaming speech literature: the temporal misalignment between when text is generated and when the corresponding speech is heard, and its consequences for full-duplex interaction. This is not a latency problem—it is a coherence problem.

In any streaming speech system where text generation runs ahead of speech playback (which is all existing systems, as shown in Figure 5), there is a growing temporal gap between the model's current understanding of the environment and the content being vocalized. If the model generates a 10-word sentence based on what it saw at time t, but that sentence takes 4 seconds to speak, then at time t+4 the user hears the end of a sentence that reflects the world state of 4 seconds ago. In a static setting (reading a prepared text), this gap is irrelevant. In a dynamic full-duplex setting where the environment is continuously changing, this gap means the model is constantly saying things that are temporally stale—describing events that have already passed, responding to questions that have been superseded, or missing new information that appeared during the playback period.

The Time-Aligned Interleaving (TAIL) mechanism (Section 3.4) is the paper's solution, but the intellectual contribution is not TAIL per se but the identification of temporal alignment as a design constraint that sits at a higher level than latency optimization. Before this paper, the field optimized for two related but distinct metrics: time-to-first-token (how fast the model starts responding) and throughput (how fast it generates tokens). TAIL introduces a third, orthogonal metric: temporal coherence—the degree to which the speech output at time t reflects the model's perceptual state at or near time t. This is not reducible to either latency or throughput; you can have ultra-low-latency generation that is still temporally misaligned if the generation is not paced to match speech playback duration.

The paper's ablation of interleaving modes (Table 10) makes the tradeoff explicit. Fixed-text interleaving achieves the best speech quality (0.86 CER on Chinese, 2.38 WER on English), while TAIL sacrifices approximately 0.18 CER and 1.55 WER to maintain temporal alignment. The paper frames this as "a practical trade-off between streaming interaction and speech generation quality," but the deeper point is that temporal alignment is a distinct axis of quality that does not appear on standard speech generation benchmarks. A system with perfect WER but poor temporal alignment would fail at full-duplex interaction; a system with slightly degraded WER but tight temporal alignment would succeed. The field's evaluation methodology, which focuses exclusively on acoustic quality and content accuracy, has been blind to this dimension.

This insight has implications beyond speech generation. Any AI system that operates in a dynamic real-time environment—robotics, autonomous driving, live commentary, real-time translation—faces a version of this problem: the time between when a decision is made and when it is executed creates a gap between the world-state assumed by the decision and the actual world-state at execution time. TAIL's adaptive pacing mechanism, which considers accumulated playback progress when deciding how much text to generate, suggests a general principle: real-time systems should model their own action latencies and adjust their planning horizon accordingly, rather than treating planning and execution as decoupled phases. This principle has not been widely adopted in the LLM literature but could become important as models move from offline batch processing to embedded real-time control.

Innovation 4: The Computational Efficiency Argument as a Capability Enabler, Not Just Cost Reduction

The paper's emphasis on efficiency—16× visual compression, 5× audio compression, text-only LLM generation, 0.3B speech decoder, INT4 edge deployment at 11 GB—can appear at first glance to be an engineering optimization story. But the intellectual contribution is subtler: the paper demonstrates that efficiency is a capability enabler for full-duplex interaction, not just a nice-to-have for deployment.

The reasoning chain is as follows. Full-duplex interaction requires the model to continuously process visual and audio inputs while generating speech output, all within real-time constraints (RTF < 1). If the model is too computationally expensive, it cannot keep up with the real-time stream, and full-duplex interaction is impossible regardless of the interaction architecture. But the converse is also true: if efficiency considerations are ignored during architecture design, the resulting model may be architecturally capable of full-duplex interaction but computationally incapable of executing it in real time. Efficiency is therefore a binding constraint on the interaction paradigm; you cannot separate "what the model can do" from "how efficiently it can do it" because for full-duplex interaction, the doing must happen in real time.

This perspective contrasts with the prevailing view in the LLM scaling community, where efficiency is typically treated as a downstream optimization—first build a model that works, then quantize and distill it for deployment. The paper's architecture reflects a different philosophy: efficiency constraints should shape the architecture from the start, because some architectural choices (like having the LLM generate speech tokens directly) create computational burdens that no amount of post-hoc optimization can fully recover.

The decision to have the LLM generate only text tokens (~3-4 tokens/second) rather than speech tokens (~25 tokens/second) exemplifies this. The paper explicitly notes that when models generate speech tokens directly, "efficiency can be significantly impeded, and the core language capabilities also tend to degrade" (Section 2). This is not just a throughput issue—the 6-8× increase in decoding steps means the model spends most of its compute budget on low-level acoustic features rather than semantic reasoning, and the representational interference degrades language understanding. By delegating speech token generation to a lightweight decoder, the LLM preserves both its reasoning capacity (for better interaction quality) and its computational efficiency (for real-time feasibility).

The cross-platform edge deployment results (Table 12) validate this philosophy. An RTF of 0.21 on consumer GPUs means the model processes input and generates speech 4.8× faster than real-time, leaving substantial headroom for visual encoding, network latency, and other overhead. If the architecture had been less efficient—for example, by generating speech tokens in the LLM backbone—the RTF would likely exceed 1.0, making real-time full-duplex interaction impossible on edge hardware regardless of how clever the Omni-Flow serialization might be.

The broader implication is that for AI systems that operate under real-time constraints in dynamic environments, architectural efficiency is not orthogonal to capability but constitutive of it. A model that cannot run in real time on target hardware is, for all practical purposes, incapable of real-time interaction—its architectural capability is theoretical, not actual. This principle, obvious in robotics and control systems, has been underappreciated in the LLM literature where the dominant assumption is that models will run in datacenters with effectively unbounded compute. The paper's explicit connection between efficiency design choices (high compression ratios, text-only LLM generation) and the feasibility of full-duplex interaction on edge devices makes this principle concrete and measurable.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans multiple benchmarks organized by modality, with no single unified test set. For vision-language evaluation (Table 2 and Table 3), the paper uses approximately 20 benchmarks drawn from standard open-source collections, including OpenCompass (a composite of 8 popular VLM benchmarks; Contributors, 2023), MMBench V1.1 (Liu et al., 2024), MMVet (Yu et al., 2024), MMStar (Chen et al., 2024), MMMU (Yue et al., 2024), MathVista (Lu et al., 2024), AI2D (Kembhavi et al., 2016), MMT-Bench (Ying et al., 2024), MM-IFEval (Ding et al., 2025), OCRBench (Liu et al., 2024), TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021), OmniDocBench (Ouyang et al., 2024), HallusionBench (Guan et al., 2024), MMHal-Bench (Sun et al., 2023), Mantis-Eval (Jiang et al., 2024), MUIRBench (Wang et al., 2024), MMSI-Bench (Yang et al., 2025), Video-MME (Fu et al., 2025), LVBench (Wang et al., 2024), MLVU (Zhou et al., 2025), LongVideoBench (Wu et al., 2024), and MotionBench (Hong et al., 2024). Speech evaluation (Table 4 and Table 5) uses ASR benchmarks (AISHELL-1/2, WenetSpeech, LibriSpeech, GigaSpeech, VoxPopuli), speech translation (CoVoST 2), audio understanding (MMAU, MELD), speech QA (VoiceBench, Speech TriviaQA, Speech Web Questions, Speech CMMU), and speech generation benchmarks (SeedTTS Test, LongTTS, Expresso, ESD). Text evaluation (Table 6) uses IFEval, BBH, CMMLU, MMLU, HumanEval, MBPP, MATH-500, and GSM8K. Omni-modal evaluation (Table 7) uses Daily-Omni, WorldSense, Video-Holmes, JointAVBench, AVUT-Human, FutureOmni, and Video-MME-Short with audio. Full-duplex evaluation (Table 8) uses LiveSports-3K-CC. Most benchmarks are standard in the field; the omni-modal and full-duplex benchmarks are more recently introduced and specifically designed to evaluate temporal alignment between modalities. The paper does not report dataset sizes for individual benchmarks, relying on the standard splits from the original benchmark publications.

Base model. The principal model is MiniCPM-o 4.5, a 9B-parameter end-to-end MLLM built on a Qwen3-8B backbone with a SigLIP ViT visual encoder (0.4B), Whisper Medium audio encoder (0.3B), and a Llama-based speech token decoder (~0.3B). The model is initialized from a MiniCPM-V 4.5 pretraining checkpoint and trained through the staged pipeline described in Section 5. Primary open-source baselines include InternVL3.5-8B (Wang et al., 2025), Qwen3-VL-8B (Bai et al., 2025), Qwen3-Omni-30B-A3B (Xu et al., 2025), Kimi-Audio (9B), CosyVoice2 (Du et al., 2024), and Qwen3-8B-Instruct (Team, 2025). Proprietary baselines include Gemini 2.5 Flash (Comanici et al., 2025) and GPT-5 (Singh et al., 2025). For full-duplex streaming, baselines include LiveCC-8B (Chen et al., 2025) and StreamingVLM-8B. The model scale choices reflect the paper's emphasis on edge-deployable efficiency: MiniCPM-o 4.5 at 9B is compared against both similar-scale models (InternVL3.5-8B, Qwen3-VL-8B) and much larger models (Qwen3-Omni-30B-A3B, which has ~3.3× more total parameters but only 3B active) to demonstrate that architectural efficiency can compensate for parameter count. The Qwen3-8B-Instruct comparison (Table 6) serves as a retention check: does omni-modal training preserve the backbone's text capabilities?

Metrics. Vision-language benchmarks use accuracy (%), except OCRBench (a scaled score, maximum 1000), OmniDocBench (error rate, lower is better), MMHal-Bench (both a score and a hallucination rate), and video benchmarks where metrics vary by dataset. Speech ASR uses Character Error Rate (CER) for Chinese and Word Error Rate (WER) for English (both lower is better); speech generation additionally uses speaker similarity (SIM-o, higher is better, scored by a speaker verification model) and emotion/style control accuracy (%). Text benchmarks use standard accuracy or pass@1 (HumanEval, MBPP). Omni-modal and full-duplex benchmarks report accuracy or win rate. The evaluation protocol uses each benchmark's standard grading procedure; for speech generation, the paper specifies that "Neutral reference audio is used for evaluation" on Expresso and ESD (Table 5 footnote), meaning the evaluation isolates content and style control from voice cloning quality.

Baselines. The paper compares against a large set of baselines that vary by evaluation domain. For vision-language (Table 2, Table 3): Gemini 2.5 Flash, InternVL3.5-8B, Qwen3-VL-8B, Qwen3-Omni-30B-A3B, and GPT-5 (thinking mode only). For speech understanding (Table 4): Kimi-Audio (9B) and Qwen3-Omni-30B-A3B. For speech generation (Table 5): CosyVoice2 and Qwen3-Omni-30B-A3B. For text (Table 6): Qwen3-8B-Instruct (the LLM backbone before omni-modal training). For omni-modal understanding (Table 7): Gemini 2.5 Flash and Qwen3-Omni-30B-A3B. For full-duplex streaming (Table 8): LiveCC (8B) and StreamingVLM (8B). Not all baselines appear in all tables—for example, GPT-5 appears only in thinking-mode vision-language results, and Kimi-Audio appears only in speech understanding. This is because some proprietary models do not support certain modalities (e.g., GPT-5's audio capabilities are not fully released at the time of writing) or data access is limited.

Generation budget / compute accounting. For standard turn-based benchmarks, generation follows the standard protocol: models receive the prompt (image, audio, video, or text) and generate a complete response with no explicit budget constraint beyond the model's maximum context length. For the full-duplex evaluation on LiveSports-3K-CC, the interaction is continuous and unbounded in time, though the benchmark itself likely constrains the duration. The paper does not compare methods at matched generation budgets (unlike the reference paper's approach of controlling for number of samples or search steps); instead, comparisons are at the model level—each model uses its standard inference configuration. For efficiency comparisons (Table 11, Table 12), compute is measured in tokens/second throughput, first-token latency (seconds), memory usage (GB), and Real-Time Factor (RTF, the ratio of processing time to audio duration). These metrics are measured on specific hardware (single NVIDIA RTX 4090 for Table 11; RTX 4090 and DGX Spark for Table 12) with specified precision (BF16 or INT4).

Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, confidence intervals, or error bars. All benchmark results are point estimates from single evaluation runs. This is standard practice in the MLLM literature for benchmark reporting but limits the ability to assess whether differences between models (especially small differences like MiniCPM-o 4.5's 77.6 vs. Qwen3-VL-8B's 76.5 on OpenCompass) are statistically reliable. For the length reward ablation (Table 9, Figure 6), the paper reports results from a "lightweight RL training experiment" with training curves showing accuracy over training steps, but does not specify the number of runs or any variance estimates.

Main Quantitative Results

Vision-Language Understanding: State-of-the-Art Open-Source at 9B Scale

Headline numbers (Table 2, instruct mode). MiniCPM-o 4.5 achieves an average of 77.6 on OpenCompass, outperforming all open-source baselines at comparable or larger scales: InternVL3.5-8B (75.8, a 1.8-point gap), Qwen3-VL-8B (76.5, a 1.1-point gap), and Qwen3-Omni-30B-A3B (75.7, a 1.9-point gap despite ~3.3× more total parameters). It approaches but does not surpass Gemini 2.5 Flash (78.5), trailing by 0.9 points. On MMBench EN v1.1, MiniCPM-o 4.5 achieves 87.6—the highest among all open-source models and slightly ahead of Gemini 2.5 Flash (86.6). On MathVista, it achieves 80.1, again the highest open-source result and substantially ahead of InternVL3.5-8B (78.4) and Qwen3-Omni-30B-A3B (75.9).

Thinking mode (Table 3). With chain-of-thought reasoning enabled, MiniCPM-o 4.5 achieves 78.2 on OpenCompass, 89.0 on MMBench EN v1.1, and 88.5 on AI2D. The gap to Gemini 2.5 Flash narrows on several benchmarks: OpenCompass 78.2 vs. 79.9 (1.7-point gap, down from 0.9 in instruct mode, though note that MiniCPM-o 4.5's improvement is only 0.6 points from instruct to thinking while Gemini 2.5 Flash gains 1.4 points). On personal benchmarks where MiniCPM-o 4.5 excels in instruct mode, the thinking mode does not provide significant additional gains—MMBench EN v1.1 moves from 87.6 to 89.0, while MathVista actually drops slightly from 80.1 to 81.0 (within typical benchmark variance). This pattern is consistent with the observation that the model already performs well on these tasks without extended reasoning.

Document and OCR (Table 2). MiniCPM-o 4.5 demonstrates standout performance on document parsing. On OmniDocBench English, it achieves an error rate of 0.109—dramatically better than Gemini 2.5 Flash (0.214), Qwen3-Omni-30B-A3B (0.216), and Qwen3-VL-8B (0.255). The Chinese variant shows an even larger gap: 0.162 vs. 0.290 for Gemini 2.5 Flash and 0.363 for Qwen3-Omni. On OCRBench, TextVQA, and DocVQA, MiniCPM-o 4.5 is competitive but not dominant: OCRBench 876 vs. Qwen3-VL-8B's 896 (a small deficit), TextVQA 83.8 vs. Qwen3-Omni's 84.1 (essentially tied), DocVQA 94.7 vs. Qwen3-VL-8B's 96.1 (a 1.4-point gap).

Multi-image understanding (Table 2). MiniCPM-o 4.5 outperforms all baselines on Mantis-Eval (79.7 vs. the next-best Qwen3-Omni at 78.3) and MMSI-Bench (16.6 vs. Qwen3-Omni's 14.2), and achieves competitive results on MUIRBench (72.0 vs. Gemini 2.5 Flash's 74.5). The MMSI-Bench result is particularly notable because all models score quite low (the maximum is 16.6 out of a possible 100), suggesting this benchmark is extremely challenging and that MiniCPM-o 4.5's advantage, while significant in relative terms (a 17% relative improvement over Qwen3-Omni), represents only a small absolute capability gain.

Video understanding (Table 2). Results are mixed but generally competitive. On Video-MME without subtitles, MiniCPM-o 4.5 achieves 70.4, close to Qwen3-Omni (70.5) and Qwen3-VL-8B (71.4) but behind Gemini 2.5 Flash (75.6). On LongVideoBench, it scores 66.0, essentially tied with Qwen3-Omni (66.9). On LVBench, it scores 50.9—well behind Gemini 2.5 Flash (62.2) but close to Qwen3-VL-8B (58.0). These results suggest that while MiniCPM-o 4.5's video understanding is competitive with other open-source models, there is still a meaningful gap to proprietary models, particularly on long-form and fine-grained temporal reasoning tasks.

Hallucination (Table 2). MiniCPM-o 4.5 achieves the best HallusionBench score among all models (63.2 vs. Qwen3-VL-8B's 61.1 and Gemini 2.5 Flash's 59.1). On MMHal-Bench, it achieves a score of 4.7 (tied with Qwen3-VL-8B and Qwen3-Omni, slightly ahead of Gemini 2.5 Flash's 4.6) with a hallucination rate of 24.3—the lowest among all models except Qwen3-VL-8B (29.9). This suggests that the RLAIF-V training (Section 5.4) is effective at reducing visual hallucinations, consistent with the paper's claim that hallucination mitigation transfers from static image-text data to streaming settings.

Pattern across benchmarks. The results exhibit a consistent pattern: MiniCPM-o 4.5 is strongest on tasks requiring fine-grained visual understanding (OCR, document parsing, multi-image comparison) and competitive-to-slightly-behind on tasks requiring broad reasoning (MMMU, MMStar, MMBench) or long-form temporal understanding (LVBench). This profile is consistent with the design emphasis on high-resolution encoding (for detail-oriented tasks) and the architectural efficiency constraints (which may limit the capacity for very long context reasoning relative to much larger models).

Speech Understanding: Broad Competitiveness with Smaller Model Scale

Headline numbers (Table 4). MiniCPM-o 4.5 demonstrates speech understanding capabilities that are competitive with much larger models. On ASR benchmarks, it achieves results close to both Kimi-Audio (9B) and Qwen3-Omni-30B-A3B, with the best performance on GigaSpeech test (8.5 WER vs. Qwen3-Omni's 8.7 and Kimi-Audio's 9.4) and VoxPopuli V1-En (6.2 vs. Qwen3-Omni's 6.4 and Kimi-Audio's 8.0). On the remaining ASR benchmarks, it is slightly behind: AISHELL-1 0.9 CER vs. 0.6 for both baselines, WenetSpeech test-net 5.9 vs. Qwen3-Omni's 4.7, LibriSpeech test-clean 1.4 vs. Qwen3-Omni's 1.2. These differences are small in absolute terms (tenths of a percentage point in CER/WER) and likely within the range of benchmark variance, but they do not establish superiority in raw ASR performance.

The model's advantages become clearer on semantic speech tasks. On speech translation (CoVoST 2 en→zh), MiniCPM-o 4.5 achieves 49.9 BLEU—substantially ahead of Qwen3-Omni (46.6) and Kimi-Audio (36.6). On the reverse direction (zh→en, 26.4), it trails Qwen3-Omni (29.4) but still exceeds Kimi-Audio (18.3). On multi-task audio understanding, it achieves 76.9 on MMAU (slightly behind Qwen3-Omni's 77.5 but ahead of Kimi-Audio's 68.4) and 60.2 on MELD (the best result, ahead of Kimi-Audio's 59.1 and Qwen3-Omni's 56.8).

Speech QA results (Table 4). On VoiceBench AlpacaEval (rated 1–5), MiniCPM-o 4.5 achieves 4.81—the highest among all models (vs. Qwen3-Omni's 4.74 and Kimi-Audio's 4.46). On Speech TriviaQA, it achieves 75.5, substantially ahead of Qwen3-Omni's 62.9 and Kimi-Audio's 41.9. However, on Speech Web Questions, it scores 70.2 versus Qwen3-Omni's 74.9—a notable deficit for a knowledge-retrieval task. On Speech CMMU (Chinese knowledge QA), it scores 59.2, ahead of Qwen3-Omni's 47.8 but behind Kimi-Audio's 67.0. The paper acknowledges these remaining gaps: "the remaining gaps on Speech Web Questions and Speech CMMU show that retrieval-like factual QA and Chinese speech knowledge QA are still challenging."

Interpretation. The speech understanding results demonstrate that MiniCPM-o 4.5's audio processing pipeline (Whisper encoder with 5× compression, two-layer MLP projector) preserves sufficient fidelity for competitive ASR and enables strong semantic speech understanding, all at roughly one-third the parameter count of Qwen3-Omni. The mixed results on speech QA benchmarks—strong on TriviaQA, weak on Web Questions—suggest that the model's knowledge retrieval capabilities, rather than its speech perception, may be the bottleneck for certain tasks. This is consistent with the text results (Table 6) showing some degradation relative to the Qwen3-8B backbone on knowledge-intensive benchmarks.

Speech Generation: Best-in-Class Clarity and Control

Headline numbers (Table 5). MiniCPM-o 4.5 achieves the best results on all speech generation metrics except speaker similarity on English. On SeedTTS Test-ZH, it achieves 0.86 CER—significantly better than CosyVoice2 (1.45) and Qwen3-Omni (1.41). On SeedTTS Test-EN, it achieves 2.38 WER, better than CosyVoice2 (2.57) and Qwen3-Omni (3.39). The speaker similarity scores are competitive: SIM-o of 74.5 on Chinese (vs. CosyVoice2's 74.8) and 64.9 on English (vs. CosyVoice2's 65.2).

On LongTTS, the advantage is dramatic for English: 3.37 WER for MiniCPM-o 4.5 versus 14.80 for CosyVoice2 and 17.33 for Qwen3-Omni. This roughly 4× lower error rate suggests that MiniCPM-o 4.5's architecture—particularly the text-only LLM generation with separate speech token decoding—is substantially more stable for long-form generation than systems that generate speech tokens directly. For Chinese LongTTS, MiniCPM-o 4.5 scores 6.58 CER versus CosyVoice2's 5.27—a small deficit that may reflect the TAIL mechanism's temporal alignment constraints (as discussed in Section 3.4).

On emotion and style control, MiniCPM-o 4.5 dramatically outperforms CosyVoice2: Expresso 29.8 vs. 17.9 (1.67× better), ESD 82.1 vs. 53.4 (1.54× better). This is a substantial margin and indicates that the LLM backbone's hidden states, which condition the speech decoder, effectively encode prosodic and emotional information that allows for nuanced style control. Qwen3-Omni does not support emotion/style control evaluation (N/A in Table 5).

The TAIL tradeoff (Table 10). The ablation comparing interleaving modes on SeedTTS reveals the explicit quality cost of temporal alignment. Fixed-text interleaving achieves the best accuracy: 0.86 CER on Chinese, 2.38 WER on English. TAIL (dynamic-text interleaving) degrades to 1.04 CER on Chinese (a 21% relative increase in error) and 3.93 WER on English (a 65% relative increase). This is a substantial tradeoff—English WER nearly doubles—but as argued in Section 3.4, the temporal alignment TAIL provides is essential for full-duplex interaction where stale speech content is more harmful than slightly degraded pronunciation. The paper frames this as "a practical trade-off between streaming interaction and speech generation quality."

Headline numbers (Table 6). MiniCPM-o 4.5 achieves an average text benchmark score of 82.1 versus 81.6 for its backbone, Qwen3-8B-Instruct—a small 0.5-point improvement. This is a significant result because it demonstrates that the extensive multimodal training (speech pretraining, joint pretraining, SFT, RL) did not degrade the model's core language capabilities—a known risk in omni-modal training that the paper explicitly references (Hsiao et al., 2025).

The breakdown reveals gains and losses. MiniCPM-o 4.5 improves on BBH (81.1 vs. 69.4, a 11.7-point gain), IFEval-P (84.7 vs. 83.0), and GSMBK (94.5 vs. 93.4). It maintains performance on HumanEval (86.6 vs. 86.6, identical) and MBPP (76.7 vs. 75.9, a small gain). It degrades on MMLU (77.0 vs. 81.7, a 4.7-point loss), CMMLU (79.6 vs. 78.7, a slight gain), and MATH-500 (77.0 vs. 84.0, a 7.0-point loss).

The MATH-500 degradation is the most concerning. A 7-point drop on a pure mathematical reasoning benchmark suggests that the multimodal training may have partially interfered with the LLM backbone's mathematical reasoning capabilities—even though reasoning benchmarks like BBH showed improvement. The paper does not analyze or explain this specific degradation. The MMLU drop (4.7 points) on a broad knowledge benchmark is also notable and may reflect capacity competition between the newly learned multimodal representations and the existing textual knowledge.

Headline numbers (Table 7). MiniCPM-o 4.5 achieves the best results on five of the seven omni-modal benchmarks, despite being the smallest model evaluated (9B vs. Qwen3-Omni's 30B-A3B and whatever scale Gemini 2.5 Flash uses, which is undisclosed but presumably much larger). On Daily-Omni, it scores 80.2 vs. Gemini 2.5 Flash's 79.3 and Qwen3-Omni's 70.7. On Video-Holmes, it achieves 64.3—a massive lead over Qwen3-Omni (50.4) and Gemini 2.5 Flash (51.3). On JointAVBench, it scores 60.0 vs. Gemini 2.5 Flash's 55.6 and Qwen3-Omni's 53.1. On AVUT-Human, it achieves 78.6 vs. Qwen3-Omni's 74.2 and Gemini 2.5 Flash's 65.4.

The two benchmarks where MiniCPM-o 4.5 does not lead are FutureOmni (56.1 vs. Qwen3-Omni's 62.1) and Video-MME-Short with audio (84.7 vs. Gemini 2.5 Flash's 85.5). Both deficits are relatively small—within a few points—and do not change the overall picture of strong omni-modal performance.

Interpretation. The omni-modal benchmarks are designed to evaluate exactly the kind of cross-modal temporal reasoning that Omni-Flow's time-aligned serialization is designed to support: understanding how visual events relate to audio events, tracking temporal correspondence between modalities, and reasoning about audio-visual scenes holistically. MiniCPM-o 4.5's strong performance on these benchmarks, especially relative to its parameter count, provides indirect evidence that Omni-Flow's time-aligned architecture improves the model's ability to integrate cross-modal temporal information—not just its interaction behavior, but its fundamental understanding of multimodal content.

Full-Duplex Streaming: Demonstrated Advantage on Continuous Visual Interaction

Headline numbers (Table 8). On LiveSports-3K-CC, an audio-free full-duplex benchmark, MiniCPM-o 4.5 achieves a win rate of 54.4, outperforming LiveCC-8B (41.5) by 12.9 points and StreamingVLM-8B (45.6) by 8.8 points. These are large margins on a 100-point scale—approximately 31% and 19% relative improvements, respectively.

Limitations of this evaluation. The paper explicitly acknowledges that "due to the limited availability of benchmarks for real-time omni-modal full-duplex interaction," this is the only quantitative full-duplex result reported. LiveSports-3K-CC evaluates only visual streaming (no audio input, no speech output), which tests only a subset of what Omni-Flow enables. The paper points to "qualitative demonstrations involving simultaneous vision, speech, and text streams" on the demo website but provides no quantitative metrics for the full omni-modal full-duplex setting—the paper's central claimed capability. This is a significant evaluation gap: the benchmark that would most directly validate the paper's core contribution (real-time see-listen-speak full-duplex interaction) does not exist in a standardized form, so the paper essentially says "trust our demo" for the primary use case.

Efficiency: Real-Time Full-Duplex on Consumer Hardware

Headline numbers (Table 11, Table 12). On a single RTX 4090 with standard vLLM inference (Table 11), MiniCPM-o 4.5 in BF16 achieves 154.3 tokens/second throughput with 0.59s first-token latency and 19 GB memory usage. Qwen3-Omni-30B-A3B cannot run in BF16 on the same hardware (OOM). In INT4, MiniCPM-o 4.5 achieves 212.3 tokens/second (44% faster than Qwen3-Omni's 147.8), 0.58s first-token latency (69% faster than Qwen3-Omni's 0.98s), and 11 GB memory (45% less than Qwen3-Omni's 20 GB).

With the custom llama.cpp-omni framework (Table 12), the efficiency picture improves further. On an RTX 4090 in INT4, MiniCPM-o 4.5 achieves RTF 0.21 (processing 4.8× faster than real-time) with 11 GB memory. On DGX Spark, the RTF is 0.20 in INT4. The PyTorch baseline on the same hardware achieves RTF 1.26–1.27 (barely real-time), indicating that the custom inference framework contributes roughly a 6× speedup.

Interpretation. These results convincingly demonstrate that MiniCPM-o 4.5 can run full-duplex interaction on consumer hardware. An RTF of 0.21 means the model has substantial headroom for visual encoding, network latency, and other overhead while maintaining real-time performance. The 11 GB memory footprint in INT4 puts the model within range of high-end laptops and potentially flagship smartphones. However, the results are based on text-only throughput and memory measurements (Table 11 specifies that "throughput and memory usage are measured on text-only tasks"), not on full end-to-end omni-modal inference including visual encoding, audio encoding, and speech decoding. The true end-to-end RTF for full-duplex interaction—including all modalities—is not reported.

Ablation Studies and Robustness Checks

Omni-Flow design choices (Table 1): The paper ablates chunk size (1.0s, 0.2s, 0.1s), boundary explicitness (explicit vs. implicit), and control formulation (Listen-Speak vs. Listen-Text). The key findings are: (1) 1.0s chunks consistently outperform 0.2s and 0.1s across all metrics—the degradation is severe (MMLU drops from 0.65 at 1.0s to 0.32 at 0.1s); (2) explicit boundaries outperform implicit boundaries (e.g., AlpacaEval 3.60 vs. 3.31 at 1.0s with LT control); (3) LS control outperforms LT control (MMLU 0.65 vs. 0.56 at 1.0s with explicit boundaries). These results were discussed in detail in Section 3. The ablation validates the paper's design choices but is conducted on a subset of benchmarks (AdvBench, AlpacaEval, IFEval, SDQA, MMLU) rather than the full evaluation suite, leaving open whether the same patterns hold for vision-language and speech tasks.

Length reward design (Table 9, Figure 6): The paper compares three conditions: no length reward, Kimi K1.5-style length reward, and the proposed smooth length reward. The proposed method achieves the best benchmark average (74.3 thinking, 70.9 instruct) while reducing thinking-mode length by 35.3% and instruct-mode length by 20.5%. The Kimi K1.5-style reward reduces thinking length more aggressively (50.7%) but degrades benchmark average from 73.5 to 73.0. The training curves in Figure 6 show that the Kimi K1.5 reward causes a slowdown and slight degradation in later training, while the proposed reward maintains a trajectory similar to the no-length-reward baseline. This is a well-designed ablation that demonstrates the specific contribution of the smooth length reward's design features (the downscaling factor, the piecewise correctness handling). A missing ablation: the contribution of the 480-step delay before applying length reward is not isolated.

Speech generation interleaving modes (Table 10): Comparing no interleaving, fixed-text interleaving, and TAIL (dynamic-text interleaving) on the SeedTTS test set. Fixed-text interleaving achieves the best recognition accuracy (0.86 CER ZH, 2.38 WER EN), while TAIL trades off accuracy (1.04 CER ZH, 3.93 WER EN) for temporal alignment. This ablation directly demonstrates the alignment-quality tradeoff that is central to the TAIL design rationale. A missing ablation: the paper does not report the actual temporal alignment achieved by each mode—the claim that TAIL improves alignment is based on the design rationale, not on quantitative measurement of alignment in the evaluated interactions.

Critical Assessment

The evaluation supports many of the paper's claims but leaves significant gaps in validating the central contribution of full-duplex omni-modal interaction.

For the claim that MiniCPM-o 4.5 achieves state-of-the-art open-source vision-language performance at its scale: The evidence in Table 2 and Table 3 is strong but qualified. MiniCPM-o 4.5 leads on a majority of benchmarks among open-source models at ~8-9B scale, and is competitive with or beats the much larger Qwen3-Omni-30B-A3B on most tasks. However, the margins are often small (1-3 points on many benchmarks), and the paper does not provide any variance estimates, making it impossible to determine whether these differences are statistically significant. On some important benchmarks (MMMU, where it scores 67.6 vs. 76.3 for Gemini 2.5 Flash), there remains a large gap to proprietary models. The claim of "state-of-the-art" is plausible but fragile—a different model at similar scale might achieve similar numbers on a different evaluation run.

For the claim that MiniCPM-o 4.5 surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and speech generation quality: The evidence is strong. On five of seven omni-modal benchmarks (Table 7), MiniCPM-o 4.5 achieves the highest score, often by substantial margins (e.g., 64.3 vs. 50.4 on Video-Holmes, a 13.9-point advantage). On speech generation (Table 5), it achieves better CER/WER on all four SeedTTS Test conditions and dramatically better LongTTS English WER (3.37 vs. 17.33). The claim of higher computational efficiency is convincingly demonstrated in Table 11, where MiniCPM-o 4.5 achieves faster throughput, lower latency, and less memory usage—when Qwen3-Omni can run at all. However, the claim of "significantly higher computation efficiency" in the abstract is based on a comparison where Qwen3-Omni uses a Mixture-of-Experts architecture (30B total, 3B active), which changes the nature of the comparison—MiniCPM-o 4.5's advantage is partly architectural (dense vs. MoE, text-only LLM generation vs. direct speech token generation) and partly scale (9B vs. 30B total). The paper does not disentangle these factors.

For the claim that MiniCPM-o 4.5 enables real-time full-duplex omni-modal interaction on edge devices: This is the paper's central contribution, and the evaluation is notably incomplete. The quantitative full-duplex evaluation (Table 8) is limited to a single audio-free benchmark (LiveSports-3K-CC) that tests only visual streaming interaction. There are no quantitative metrics for the full see-listen-speak full-duplex scenario. The efficiency results (Table 12) demonstrate that the model runs faster than real-time on consumer GPUs, but these measurements are for text-only throughput—the full end-to-end full-duplex pipeline (video encoding + audio encoding + LLM inference + speech token decoding + waveform synthesis) is not benchmarked as an integrated system. The paper provides qualitative demos on a website but no systematic evaluation of full-duplex interaction quality, latency, or robustness. The claim that the model exhibits "proactive behaviors such as issuing reminders or comments" is not evaluated quantitatively at all—there is no benchmark for proactive behavior, no human evaluation, and no ablation showing that proactive behavior emerges from Omni-Flow versus simpler alternatives. The proactive behavior is mentioned in the abstract and introduction as a key capability, but the evaluation provides no evidence for it beyond the demo.

For the Omni-Flow framework as a general contribution: The ablation in Table 1 demonstrates that design choices matter for the specific benchmarks tested (text-centric metrics like MMLU, AlpacaEval), but the ablation does not evaluate whether Omni-Flow improves full-duplex interaction quality versus a simpler alternative—it evaluates whether Omni-Flow design choices affect standard language understanding benchmarks. The relationship between MMLU performance and full-duplex interaction quality is not established. A more convincing ablation would compare Omni-Flow to a turn-based streaming baseline on a full-duplex task, but this experiment is not reported.

Key weaknesses and missing experiments:

No full-duplex omni-modal benchmark results. The paper's central claim—that MiniCPM-o 4.5 can see, listen, and speak simultaneously in real-time—is not quantitatively evaluated. LiveSports-3K-CC is audio-free and evaluates only visual interaction. There is no standardized, quantitative evaluation of the simultaneous vision+audio+speech full-duplex scenario that the paper's title and abstract emphasize.
No proactive behavior evaluation. The ability to "issue reminders or comments based on continuous understanding of the live scene" is claimed in the abstract but never quantitatively assessed. There is no ablation showing that Omni-Flow enables proactive behavior that a turn-based baseline lacks.
No ablation comparing Omni-Flow to a simpler streaming baseline on interaction tasks. The paper does not evaluate whether the full Omni-Flow framework (time-chunked serialization, LS control, TAIL) outperforms a simpler approach (e.g., standard turn-based streaming with external VAD) on a full-duplex interaction task. The Table 1 ablation evaluates text understanding benchmarks, not interaction quality.
Single full-duplex benchmark with limited scope. LiveSports-3K-CC evaluates only sports commentary, a narrow domain. The generalizability of full-duplex interaction capability to other domains (conversation, assistance, education) is not established.
No human evaluation. For capabilities like speech naturalness, interaction quality, and proactive behavior appropriateness—all of which are inherently subjective—the paper provides no human evaluation results. All speech quality metrics are automated (CER, WER, SIM-o), and interaction quality is measured only through automated benchmarks.
Missing end-to-end inference benchmarking. The RTF and memory measurements are for text-only inference (Table 11) or unspecified (Table 12 doesn't specify what the RTF measures). The actual end-to-end latency from camera frame to audible speech output, including all encoding and decoding stages, is not reported.
Data leakage concerns. The model is initialized from MiniCPM-V 4.5, which was trained on large-scale vision-language data. The paper does not discuss whether any of the evaluation benchmarks' test data may have been included in MiniCPM-V 4.5's training data, a standard concern for models that report results on widely-used benchmarks whose training data composition is often unknown for web-scraped corpora.
The text benchmark degradation is unexplained. The 7-point drop on MATH-500 and 4.7-point drop on MMLU relative to the Qwen3-8B backbone (Table 6) is noted but not analyzed. These degradations could indicate capacity competition or catastrophic forgetting from multimodal training, and understanding their cause is important for future work on omni-modal models.

In summary, the evaluation strongly supports the paper's claims about MiniCPM-o 4.5's static multimodal capabilities—vision-language understanding, speech understanding, speech generation, and text capability—relative to comparable open-source models. The evidence for the paper's central contribution—real-time full-duplex omni-modal interaction—is substantially weaker. The full-duplex interaction capability is demonstrated qualitatively on a demo website but is not rigorously evaluated quantitatively. The single full-duplex benchmark result (Table 8) is encouraging but narrow in scope, and the proactive behavior that the paper positions as a key differentiator is entirely unevaluated. The efficiency results demonstrate that real-time operation is technically feasible on consumer hardware, but the end-to-end system latency is not reported. The paper makes a compelling case for the architecture and framework, but the evaluation of the resulting full-duplex interaction capability—which is what the paper claims as its primary contribution—remains largely qualitative.

6. Limitations and Trade-offs

6.1 The Central Full-Duplex Claim Is Not Quantitatively Validated

The assumption or constraint. The paper's title, abstract, and introduction center on MiniCPM-o 4.5 as a system for "real-time full-duplex omni-modal interaction"—simultaneously seeing, listening, and speaking while acting proactively. This is presented as the paper's primary contribution and differentiator from prior turn-based streaming models. The paper acknowledges the evaluation gap explicitly in Section 6.5:

"Due to the limited availability of benchmarks for real-time omni-modal full-duplex interaction, we report results on LiveSports-3K-CC, an audio-free full-duplex benchmark. Qualitative demonstrations involving simultaneous vision, speech, and text streams are provided on our demo website."

The consequence. The single quantitative full-duplex result (Table 8, LiveSports-3K-CC) evaluates only visual streaming without audio input or speech output—it tests a narrow subset of what "full-duplex omni-modal interaction" means. The paper's central claim therefore rests on qualitative web demos with no systematic measurement of: (1) whether the model actually perceives and responds in parallel rather than in micro-turns, (2) how often temporal alignment between speech and perception is maintained, (3) the end-to-end latency from environmental event to spoken response, (4) whether proactive behaviors occur reliably and appropriately, or (5) how interaction quality compares to simpler turn-based baselines under matched conditions. A practitioner cannot determine from the paper whether MiniCPM-o 4.5 delivers a qualitatively different interaction experience from a fast turn-based model, or whether the Omni-Flow framework provides benefits that justify its complexity over simpler streaming architectures. The absence of human evaluation is particularly consequential: automated metrics cannot assess whether full-duplex interaction feels natural, whether proactive interruptions are well-timed, or whether the temporal alignment TAIL provides actually improves the user's perception of the model's situational awareness.

What evidence exists in the paper. Table 8 shows MiniCPM-o 4.5 achieving 54.4 win rate on LiveSports-3K-CC versus 41.5 (LiveCC) and 45.6 (StreamingVLM). This is a single benchmark in a single domain (sports commentary) with a single modality (vision-only). Table 1 ablates Omni-Flow design choices on text-centric benchmarks (AdvBench, AlpacaEval, IFEval, SDQA, MMLU), none of which measure full-duplex interaction quality. The paper points readers to a demo website for the primary capability claim. There is no measurement of proactive behavior frequency, appropriateness, or user satisfaction anywhere in the paper.

Mitigation status. The paper does not attempt to mitigate this gap. It acknowledges the limited benchmark availability but does not construct even a small-scale human evaluation or a custom automated metric for full-duplex interaction quality. The demo website is offered as evidence, but demos are inherently curated and unblinded. The limitation is passed to future work implicitly: the field needs better full-duplex benchmarks before systems like this can be rigorously evaluated.

6.2 Difficulty Estimation Is Computationally Prohibitive for Deployment

The assumption or constraint. The paper's evaluation of MiniCPM-o 4.5's efficiency—the claim that it runs real-time full-duplex interaction on edge devices with less than 12 GB RAM—is based on measurements of the model's inference throughput and memory usage (Tables 11 and 12). However, these measurements are for text-only inference and do not account for the full computational cost of the end-to-end full-duplex pipeline: visual encoding of multiple frames per second, audio encoding in streaming chunks, speech token decoding, and waveform synthesis. The paper acknowledges this scope limitation in Table 11's caption:

"First-token latency is evaluated with 64-frame visual inputs, while throughput and memory usage are measured on text-only tasks."

The consequence. A practitioner deploying MiniCPM-o 4.5 for full-duplex interaction cannot rely on the reported RTF of 0.21 (Table 12) as an estimate of end-to-end system performance. The visual encoder (SigLIP ViT, 0.4B parameters) must process every incoming frame; the audio encoder (Whisper Medium, 0.3B) must process every audio chunk; the speech token decoder (0.3B) must generate 25 speech tokens per second of output; and the flow-matching decoder must synthesize waveforms from those tokens. Each of these components adds latency and memory pressure not captured in text-only throughput measurements. On consumer hardware with limited memory bandwidth (e.g., a laptop with integrated graphics), the total end-to-end latency could easily exceed the real-time threshold even if the LLM backbone alone runs at RTF 0.21. The paper's claim that the model "run[s] efficiently on edge devices with less than 12GB RAM" (abstract) is therefore a statement about memory capacity for the LLM weights, not about whether the complete system achieves real-time full-duplex interaction on that hardware.

What evidence exists in the paper. Table 11 reports throughput and memory for text-only tasks, with first-token latency measured separately with 64-frame visual inputs. Table 12 reports RTF and memory for llama.cpp-omni but does not specify whether RTF includes visual encoding, audio encoding, or speech decoding. The memory figures (11 GB INT4) are end-to-end (the full model weights must be loaded), but the time measurements are not end-to-end. Nowhere in the paper is the complete "camera frame in → audio waveform out" latency reported for a representative hardware configuration.

Mitigation status. The paper provides component-level efficiency data (throughput, memory, RTF) but does not report end-to-end system latency for the full-duplex pipeline. This is a significant omission for a paper whose central contribution is a deployable real-time interaction system. The development of llama.cpp-omni and the cross-platform validation (macOS, Windows, Linux) are steps toward practical deployment, but without end-to-end timing, a practitioner cannot determine whether the system meets real-time constraints on their target hardware.

6.3 Proactive Behavior Is Claimed but Unevaluated and Likely Brittle

The assumption or constraint. Proactive behavior—the model's ability to "issu[e] reminders or comments based on its continuous understanding of the live scene" (abstract) rather than only responding to explicit user requests—is positioned as a key differentiator from existing reactive models. The paper claims that this behavior "arise[s] from ongoing context within the same interaction loop" (Section 1) and "naturally supports proactive behavior" (Section 3.2). The paper acknowledges the limitation in Section 8:

"the model's proactive behavior is still relatively simple, leaving richer context-aware planning and self-initiated assistance for future work."

The consequence. Proactive behavior in an AI assistant is a high-stakes capability. An AI that speaks without being prompted must have extremely high precision: a false positive (speaking when it shouldn't) is an interruption; a false negative (failing to speak when a reminder would have been helpful) is a missed opportunity; and an inappropriate proactive comment (misreading the social context, commenting on something private) is a trust-destroying error. The paper provides no measurement of proactive behavior precision, recall, or user-perceived appropriateness. A practitioner cannot determine whether MiniCPM-o 4.5's proactive behavior is robust enough to deploy in a real product, or whether it is a fragile capability that works only in carefully curated demo scenarios. The acknowledgment that the behavior is "still relatively simple" does not quantify what "simple" means—does the model offer timer-based reminders? Does it comment on visually salient events? Does it understand social context enough to know when silence is appropriate? Without evaluation, the proactive behavior claim functions as a vision statement rather than a demonstrated capability.

What evidence exists in the paper. None. There is no quantitative evaluation of proactive behavior in the paper—no benchmark, no ablation comparing proactive vs. reactive configurations, no human evaluation of proactive comment quality. The LiveSports-3K-CC benchmark (Table 8) evaluates continuous visual commentary, which is a form of proactive behavior (the model speaks without explicit user queries), but the benchmark does not separately measure proactive vs. reactive response quality or appropriateness. The demo website is the only evidence offered.

Mitigation status. The paper does not attempt to evaluate proactive behavior and delegates it to future work. This is a significant gap given the prominence of proactive behavior in the introduction and abstract. A minimal evaluation would have been to measure how often the model correctly decides to speak vs. remain silent in constructed scenarios with ground-truth labels for appropriate intervention points—this would not require a standardized benchmark, only a small annotated test set.

6.4 Significant Text Capability Degradation on Mathematical Reasoning and Knowledge

The assumption or constraint. The paper claims that MiniCPM-o 4.5 "retains the strong visual strengths of the MiniCPM family" (Section 1) and that the staged training pipeline is "designed to progressively integrate speech into the multimodal system in a smooth and stable manner" (Section 5). The implicit assumption is that omni-modal training preserves or improves text capabilities relative to the LLM backbone. The paper acknowledges a tradeoff only implicitly, through the data it reports.

The consequence. Table 6 reveals that MiniCPM-o 4.5 loses 7.0 points on MATH-500 (77.0 vs. 84.0 for the Qwen3-8B backbone) and 4.7 points on MMLU (77.0 vs. 81.7). These are large degradations on core capabilities. MATH-500 measures mathematical reasoning—a 7-point drop means the model fails on roughly 7% more problems that its backbone could solve. MMLU measures broad world knowledge—a 4.7-point drop across 57 subjects represents a meaningful erosion of factual knowledge. For a practitioner deciding whether to adopt MiniCPM-o 4.5, these degradations matter because many practical applications require strong text reasoning as a substrate for multimodal tasks. For example, a full-duplex AI tutor needs both real-time interaction and accurate mathematical reasoning; MiniCPM-o 4.5 may sacrifice the latter for the former. The paper's average text score (82.1 vs. 81.6) masks these specific losses because gains on BBH (+11.7) offset the losses, but BBH measures a different capability (logical reasoning with chain-of-thought) than MMLU (knowledge) or MATH-500 (mathematical problem-solving).

What evidence exists in the paper. Table 6 provides the raw numbers. The paper does not analyze or explain the MATH-500 and MMLU degradations—no ablation identifies which training stage or data mixture causes the loss, and no mitigation strategy is proposed. The paper's discussion of text results (Section 6.4) focuses on the positive finding that the average text score is maintained, stating that "a strategic balance of textual and multimodal data allows the model to retain its text capabilities," without addressing the specific degradations.

Mitigation status. Not addressed. The paper does not acknowledge the MATH-500 or MMLU degradations as limitations, does not analyze their causes, and does not propose strategies to recover the lost capabilities. A practitioner concerned about mathematical reasoning or factual knowledge in their application would need to independently evaluate whether the degradation is acceptable or whether the backbone model should be used for text-only tasks instead.

6.5 Temporal Alignment in TAIL Is Not Quantitatively Measured

The assumption or constraint. The Time-Aligned Interleaving (TAIL) mechanism is designed to solve a specific problem: the temporal misalignment between when text is generated and when the corresponding speech is heard, which the paper argues causes spoken content to become "stale with respect to the ongoing interaction" (Section 3.4). TAIL adaptively adjusts the amount of text generated in each time chunk to keep speech playback synchronized with the evolving environment. The implicit claim is that TAIL improves temporal alignment relative to fixed-text interleaving or non-interleaved generation.

The consequence. Table 10 shows that TAIL achieves worse speech recognition accuracy than fixed-text interleaving (1.04 vs. 0.86 CER on Chinese, 3.93 vs. 2.38 WER on English) but does not measure whether this accuracy sacrifice actually buys improved temporal alignment. A practitioner cannot determine whether TAIL's alignment benefit justifies its accuracy cost because the benefit is not quantified. It is possible that fixed-text interleaving already achieves sufficient temporal alignment for practical full-duplex interaction, or that TAIL's alignment improvement is too small to be perceptually meaningful, or that TAIL's adaptive mechanism occasionally makes large alignment errors that disrupt interaction. Without measuring alignment, TAIL is a mechanism with a plausible rationale but no demonstrated efficacy for its stated purpose.

What evidence exists in the paper. Table 10 provides speech quality metrics (CER, WER, SIM-o) for three interleaving modes. There is no measurement of temporal alignment—no metric for the lag between environmental events and speech content referencing those events, no measurement of how accurately speech playback time tracks real time, and no human evaluation of whether TAIL feels more temporally aligned than alternatives. The paper's entire evidence for TAIL's effectiveness is that it is designed to solve a problem the paper identifies; the evaluation does not verify that it solves it.

Mitigation status. Not addressed. The paper does not propose a metric for temporal alignment, does not measure it, and does not acknowledge the absence of this measurement as a limitation. This is a fundamental evaluation gap for a mechanism that is presented as one of the paper's technical contributions (Section 3.4) and that imposes a measurable cost on speech quality.

6.6 Evaluation Is Single-Model, Single-Family, with Undisclosed Training Data

The assumption or constraint. All experiments use MiniCPM-o 4.5, which inherits its architecture and much of its training data from the MiniCPM family (MiniCPM-V 4.5 for vision-language, MiniCPM 4.1 for text, and MiniCPM-o 2.6 for early omni-modal capabilities). The paper states (Section 4.2) that vision-language data builds on MiniCPM-V 4.5's data system, and (Section 4.2) that text data comes from MiniCPM 4.1 post-training. The implicit assumption is that the results generalize to other model families and architectures, and that the reported benchmarks are not contaminated by training data leakage from the MiniCPM family's data curation pipeline.

The consequence. A practitioner cannot determine whether Omni-Flow's benefits are specific to the MiniCPM architecture and training recipe, or whether they would transfer to a different base model. The aggressive visual compression (16×), text-only LLM generation, and Whisper-based audio encoding are design choices that may interact with Omni-Flow in ways that don't generalize. For example, a model that generates speech tokens directly in the LLM backbone (like Qwen3-Omni) might require different serialization strategies. A model with a different visual encoder might produce tokens at a different rate, affecting the optimal chunk size. More concerningly, the paper does not discuss whether any of the evaluation benchmarks' test data may overlap with the training data of MiniCPM-V 4.5 or MiniCPM 4.1. This is a standard concern for models trained on large-scale web data: benchmarks like MMLU, MMBench, and OCRBench have publicly available test sets whose questions may appear in web-crawled training corpora. Since MiniCPM-o 4.5 is initialized from MiniCPM-V 4.5 weights and further trained, any contamination in the initialization checkpoint propagates to the final model. The paper's strong OCR and document parsing results (OmniDocBench error rate 0.109 vs. 0.214 for Gemini 2.5 Flash) could partly reflect training data overlap rather than genuine capability.

What evidence exists in the paper. None. The paper does not discuss data contamination, does not provide a decontamination analysis, and does not ablate the contribution of MiniCPM family initialization versus omni-modal training to final performance. The results are from a single model family with no replication on alternative architectures.

Mitigation status. Not addressed. The paper does not discuss generalizability or data contamination. This is not unique to MiniCPM-o 4.5—most MLLM papers face the same limitation—but it is particularly relevant here because the paper claims state-of-the-art performance based on small margins (often 1–3 points on individual benchmarks), where training data overlap could plausibly account for the difference. Future work would need to replicate Omni-Flow on a different base model family and conduct standard decontamination analyses to establish that the framework's benefits are genuine and generalizable.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual shift that is subtle but foundational: it recasts interaction architecture—the temporal logic governing when perception and response occur relative to each other—as a first-class design problem for multimodal AI, separate from modality coverage, model scale, or inference latency. Before MiniCPM-o 4.5, the field's working assumption was that progress toward natural multimodal interaction would come from expanding modality support and reducing response time. Models like GPT-4o, Gemini Live, and Qwen3-Omni all operate within this paradigm: they process input, then generate output, then wait for more input. The paper's diagnosis (Section 1, Figure 3) is that this alternating-phase structure creates a qualitative ceiling—no amount of speed or modality expansion can produce behaviors that require parallel perception and response, such as mid-utterance adjustment, proactive interruption, or continuous scene-aware commentary.

The magnitude of this shift is best characterized as a reframing with practical consequences, not a paradigm shift in the Kuhnian sense. The paper does not overturn the transformer architecture or the pretraining-finetuning paradigm. Rather, it identifies that the field has been optimizing within a design space (turn-based interaction) whose fundamental constraints have been invisible because they were conflated with latency and modality limitations. Once those limitations were addressed by prior work, the structural constraint of serialized perception and response became the bottleneck—but only if you view interaction as a temporal architecture problem rather than a capability problem. The paper's contribution is making that constraint visible and providing one concrete framework (Omni-Flow) for operating outside it. The shift is analogous to how the networking community moved from improving modem speeds to designing packet-switching protocols: the bottleneck wasn't the hardware, it was the logic of when bits get sent.

The specific conceptual moves that constitute this shift:

1. Endogenizing turn-taking as a learned behavior. By making the listen/speak decision an explicit prediction within the autoregressive generation loop (the LS formulation, Section 3.3), the paper eliminates the external Voice Activity Detection (VAD) module that has been a universal component of conversational AI systems. This is more than an engineering convenience—it means turn-taking norms can be learned from data rather than hard-coded, enabling culturally-specific, context-dependent, and speaker-adaptive interaction patterns. The ablation in Table 1 provides empirical grounding: the LS formulation substantially outperforms the LT formulation (MMLU 0.65 vs. 0.56 at 1.0s chunks), supporting the claim that separating the decision to speak from the content of speech is architecturally beneficial. This finding suggests a research agenda where turn-taking is studied as a model capability rather than an external systems problem.

2. Temporal alignment as a distinct quality axis. The Time-Aligned Interleaving (TAIL) mechanism (Section 3.4) formalizes a problem that the streaming speech literature had not previously identified: the growing temporal gap between a model's current perceptual state and the speech content being vocalized. Standard speech evaluation metrics (CER, WER, speaker similarity) are blind to this gap. TAIL introduces a tradeoff the paper makes explicit: fixed-text interleaving achieves better acoustic quality (0.86 CER on Chinese), while TAIL sacrifices some quality (1.04 CER) to maintain temporal alignment. This is not a cost-reduction tradeoff—it is a capability-quality tradeoff where the capability (producing temporally relevant speech in a dynamic environment) cannot be measured by existing benchmarks. The paper thus implicitly argues that the field needs new evaluation methodologies for streaming interaction systems, ones that measure whether spoken content is temporally appropriate, not just acoustically accurate.

3. Efficiency as a capability enabler, not a downstream optimization. The paper's architectural decisions—16× visual compression, 5× audio compression, text-only LLM generation, 0.3B speech decoder—are presented not as deployment optimizations but as design constraints that make full-duplex interaction possible on consumer hardware. The paper argues, through its architecture rather than through explicit claims, that some design choices (specifically, having the LLM generate speech tokens directly) create computational burdens that no amount of post-hoc quantization can fully recover for real-time operation. The RTF of 0.21 on a consumer GPU (Table 12) is evidence that the architecture was designed with real-time constraints as a primary consideration, not an afterthought. This shifts the conversation from "how do we deploy large models efficiently" to "how should we design interaction architectures so that real-time operation is feasible on target hardware."

What becomes less attractive as a result of this work:

Direct speech token generation in the LLM backbone. The paper explicitly argues that having the LLM generate speech tokens (~25 tokens/second) impedes efficiency and degrades language capabilities (Section 2). Given that MiniCPM-o 4.5 achieves competitive or superior speech generation quality (Table 5) while generating only text tokens in the LLM, the burden of proof shifts to direct-generation approaches to demonstrate a compelling advantage that justifies the 6–8× increase in decoding steps.
External VAD and hard-coded turn-taking rules. The LS formulation's success (Table 1) and the argument that proactive behavior emerges naturally from learned listen/speak decisions (Section 3.2) suggest that external turn-detection modules are a legacy design pattern that will be superseded by models that learn turn-taking end-to-end. This does not mean VAD is obsolete—it may still be useful as a feature or initialization—but the architectural assumption that turn-taking is an external systems problem rather than a learned model behavior is challenged.
Evaluating streaming systems solely on static accuracy metrics. The TAIL tradeoff (Table 10) demonstrates that optimizing for acoustic quality alone can produce systems that are temporally misaligned and therefore ineffective for dynamic interaction. This implies that benchmark suites like SeedTTS, LibriSpeech, and standard ASR evaluations are insufficient for characterizing full-duplex systems—new metrics that capture temporal coherence, response appropriateness, and proactive behavior quality are needed.

Reconciling prior contradictions. The paper indirectly reconciles a tension in the omni-modal literature between two design philosophies: (1) models like Qwen3-Omni and Mini-Omni that have the LLM backbone directly generate speech tokens, trading off language capability for architectural simplicity, and (2) models like MiniCPM-o 4.5 that separate speech generation into a lightweight decoder, trading off architectural complexity for efficiency and capability preservation. Prior work provided conflicting evidence about which approach was superior—Qwen3-Omni achieved strong results with direct generation, while other work (Hsiao et al., 2025; Xu et al., 2025) showed language capability degradation. MiniCPM-o 4.5's results suggest that the tradeoff depends on the interaction paradigm: for turn-based interaction where the model speaks after processing input, direct generation may be acceptable. For full-duplex interaction where the model must continuously process new input while speaking, the decoding efficiency of text-only LLM generation becomes critical. The conflict in prior results may reflect different implicit assumptions about the interaction setting, and this paper makes that setting explicit—full-duplex interaction imposes stricter efficiency constraints that favor separated architectures.

Follow-Up Research This Work Enables

1. Developing a benchmark for end-to-end full-duplex omni-modal interaction quality. The paper's central limitation is the absence of quantitative evaluation for the see-listen-speak full-duplex scenario (the single full-duplex benchmark, LiveSports-3K-CC, is audio-free). A direct follow-up would construct a benchmark with the following properties: (a) synchronized video and audio streams that require simultaneous perception, (b) temporal annotation of when the model should speak vs. remain silent, (c) ground-truth references for what the model should say at intervention points (to measure content accuracy separately from timing accuracy), and (d) coverage of diverse scenarios (conversation with interruption, proactive reminders, continuous commentary, ambient assistance). A strong benchmark would measure not just accuracy but three orthogonal metrics: temporal appropriateness (did the model speak at the right times?), content accuracy (was what it said correct?), and alignment latency (how long after an environmental event does the spoken response occur?). MiniCPM-o 4.5 and Qwen3-Omni could be evaluated head-to-head to determine whether Omni-Flow's time-aligned architecture provides measurable benefits over fast turn-based streaming for full-duplex interaction quality. A negative result—finding that a fast turn-based model performs as well as MiniCPM-o 4.5 on full-duplex tasks—would substantially narrow the claimed benefit of Omni-Flow.

2. Ablation study isolating the contribution of LS control to proactive behavior. The paper claims that proactive behavior (issuing reminders, commenting without being prompted) emerges from the LS formulation where the model decides autonomously when to speak. This claim is not evaluated. A direct experiment would train two versions of MiniCPM-o 4.5—one with LS control, one with external VAD-based turn detection (mimicking the standard approach)—and measure proactive behavior frequency, precision, and recall on a constructed test set. The test set would consist of video scenarios with annotated "intervention points" where a proactive comment is appropriate (e.g., a timer reaches zero, a safety hazard appears, a visually salient event occurs). The key measurement is whether the LS model initiates speech at these points without explicit user prompts, and whether the VAD-based model fails to do so because it only responds to audio triggers. A finding that LS enables proactive behavior would validate a core design claim; a null result (the VAD model also shows proactive behavior through other mechanisms) would suggest that proactive behavior comes from training data, not architecture, and that Omni-Flow's complexity isn't necessary for this capability.

3. Stress-testing TAIL's temporal alignment in high-variance speech scenarios. Table 10 shows that TAIL sacrifices speech quality (3.93 WER vs. 2.38 WER on English) for claimed temporal alignment, but alignment itself is not measured. A stress-test would construct scenarios with deliberately mismatched text-vocalization ratios: sentences with many short words ("I am at the bus stop and it is on time") versus sentences with long, multisyllabic words ("The extraordinarily sophisticated architectural embellishments characterize the cathedral's distinctive aesthetic"). In scenes where the visual environment changes rapidly (e.g., a sports play developing over 2–3 seconds), the test would measure whether TAIL-annotated speech content at time t actually references visual events at approximately time t, versus lagging by several seconds. The comparison would be against fixed-text interleaving on the same scenarios. A strong follow-up would also measure user-perceived temporal alignment through a human study, since automated metrics for temporal coherence don't exist. If TAIL shows no measurable alignment improvement despite the accuracy cost, the mechanism's design rationale is undermined, and alternative approaches (e.g., faster speech rate adaptation, explicit timestamp annotation in generated text) should be explored.

4. Replicating Omni-Flow on a different base architecture to establish generality. The paper demonstrates Omni-Flow only with the MiniCPM family's specific design choices: 16× visual compression, 5× audio compression, text-only LLM generation, Llama-based speech decoder. An important follow-up would replicate the Omni-Flow framework on a different model architecture—for example, taking Qwen3-Omni's architecture (which generates speech tokens directly in the LLM) and adapting it to use Omni-Flow serialization with TAIL, or taking a different vision-language backbone (e.g., InternVL3.5-8B) and adding speech modules with Omni-Flow. The key measurements: does Omni-Flow improve full-duplex interaction quality regardless of the underlying architecture? Do the optimal chunk size, boundary explicitness, and control formulation (Table 1) generalize to different visual encoding rates and LLM capacities? A null result—Omni-Flow benefits are specific to MiniCPM's architecture—would narrow the framework's claimed generality. A positive result would establish Omni-Flow as a modular interaction layer that can be added to diverse multimodal architectures.

5. Dynamic difficulty-adaptive chunk sizing for Omni-Flow. The paper fixes chunk size at 1.0 second after ablating alternatives (Table 1), but acknowledges the latency-capacity tradeoff: shorter chunks improve responsiveness but reduce per-chunk modeling budget. A dynamic extension would vary chunk size based on estimated scene complexity: use shorter chunks (0.2–0.5s) during rapidly changing scenes (sports, conversation) and longer chunks (1.0–2.0s) during static scenes (waiting, monitoring). The model would need to predict chunk duration as part of the LS control decision—analogous to how TAIL adaptively varies text generation quantity based on playback progress. This would be evaluated on a benchmark with mixed-dynamic scenes, measuring whether adaptive chunking improves responsiveness without sacrificing stability. The ablation in Table 1 provides a starting point: we know the performance of fixed chunk sizes on static benchmarks; adaptive chunking would need to demonstrate that it can match or exceed the 1.0s baseline on stable scenes while improving over 1.0s on dynamic scenes. A failure mode—adaptive chunking destabilizes generation because the model receives inconsistent per-chunk context—would reveal a fundamental tension in the Omni-Flow framework between adaptability and stability.

6. Investigating the cause of MATH-500 and MMLU degradation after omni-modal training. Table 6 shows MiniCPM-o 4.5 losing 7.0 points on MATH-500 and 4.7 points on MMLU relative to its Qwen3-8B backbone. The paper does not analyze this degradation. A targeted investigation would use intermediate checkpoints from the staged training pipeline (speech pretraining → joint pretraining → SFT → RL) to identify which stage causes the loss. Possible hypotheses: (a) the speech pretraining stage (freezing the LLM backbone) introduces a distribution shift when the backbone is later unfrozen; (b) the joint pretraining data mixture underserves mathematical reasoning and knowledge tasks; (c) the RL stage's length reward (Equation 1) penalizes the kind of extended reasoning that MATH problems require; (d) capacity competition—the LLM's limited representational capacity is partially diverted to encoding prosodic features for speech generation, reducing capacity for mathematical reasoning. Each hypothesis suggests a different mitigation: data rebalancing, capacity expansion, or adjusted RL rewards. The experiment would measure MATH-500 and MMLU at each training stage and ablate the data mixture ratios. Understanding this degradation is practically important because future omni-modal models will need to preserve reasoning capabilities, and theoretically important because it reveals the nature of interference between modalities in shared representational spaces.

Practical Applications and Downstream Use Cases

1. Always-on accessibility assistants on consumer devices. The paper's demonstration that full-duplex omni-modal interaction runs on hardware with 11 GB RAM in INT4 (Table 12, RTF 0.21 on RTX 4090, cross-platform support via llama.cpp-omni) means that a continuously-running AI assistant for users with visual or hearing impairments is technically feasible on a high-end laptop or desktop. A system using MiniCPM-o 4.5 could continuously monitor the user's environment through a webcam and microphone, provide spoken descriptions of visual events ("a person is approaching from your left"), read documents aloud when held up to the camera, transcribe ambient speech in real-time, and offer proactive reminders based on visual cues ("your medication is on the table to your right"). The key enabler is that the model does not require explicit user queries to provide assistance—it observes continuously and initiates speech when appropriate (the proactive behavior capability, albeit unevaluated in the paper). The privacy benefit of edge deployment is significant for accessibility: users with disabilities may be particularly sensitive to having continuous video and audio streams sent to cloud servers. The 12 GB RAM requirement is achievable on a MacBook Pro or mid-range Windows desktop, making this a plausible near-term deployment rather than a research concept.

2. Live sports and event commentary generation at low cost. The LiveSports-3K-CC result (Table 8, 54.4 win rate) and the full-duplex streaming capability suggest that MiniCPM-o 4.5 could serve as a basis for automated live commentary systems. Unlike current approaches that require separate vision, language, and speech synthesis modules connected in a serial pipeline, MiniCPM-o 4.5 provides an end-to-end system that takes in a video feed and produces temporally aligned spoken commentary. The efficiency is the practical differentiator: on a single consumer GPU, the model can process and commentate at 4.8× real-time speed (RTF 0.21), meaning a single machine could handle multiple simultaneous streams. This enables applications like automated commentary for amateur sports streams, live event description for visually impaired audiences, or real-time video summarization for surveillance and monitoring. The critical advantage over turn-based approaches is temporal coherence: TAIL's alignment mechanism (Section 3.4) is designed to ensure that spoken comments reference the current state of the event, not events that occurred several seconds ago. A practical deployment would need to measure and optimize this alignment in the target domain, since the paper does not quantitatively evaluate temporal alignment.

3. Real-time collaborative AI for remote work and education. In scenarios like remote pair programming, virtual tutoring, or collaborative document editing, a full-duplex AI could observe the shared screen and conversation continuously, offering suggestions without being explicitly queried, and adjusting its speech mid-utterance if the user takes an unexpected action. For example, during a coding session, the AI could watch the user's editor and terminal, listen to their verbal reasoning, and interject if it detects an impending error—all without the user needing to stop and ask. The efficiency results (Table 11, 212.3 tokens/second throughput in INT4) mean the model can keep up with real-time screen sharing and conversation on the same consumer GPU that might be driving the user's display. The omni-modal understanding benchmarks (Table 7, e.g., JointAVBench 60.0, AVUT-Human 78.6) suggest the model can integrate visual and audio information effectively for reasoning. The key open question—not addressed in the paper—is whether the proactive behavior precision is high enough for this use case: an AI that interrupts too frequently or at inappropriate moments would be a distraction rather than an assistant. Deployment would likely require a confidence threshold on the LS control token to trade off proactivity against false-positive interruptions.

4. Offline-first voice assistants for privacy-sensitive or connectivity-limited environments. The cross-platform llama.cpp-omni implementation (Table 12) and the 11 GB memory footprint mean that a full-duplex voice assistant can run entirely on-device without internet connectivity. This is relevant for: military and field operations where cloud connectivity is unavailable or insecure, healthcare settings where patient data cannot leave the device, industrial environments with unreliable connectivity, and consumer applications where users prefer not to stream continuous audio/video to corporate servers. The model's strong speech understanding (Table 4, e.g., 8.5 WER on GigaSpeech, 75.5 on Speech TriviaQA) and speech generation (Table 5, e.g., 0.86 CER on SeedTTS Test-ZH) suggest it can handle the full interaction loop locally. The proactive behavior capability—if it proves robust in practice—would enable use cases like an on-device assistant that monitors equipment status through a camera and alerts the user to anomalies without being prompted. The practical barrier is that the proactive behavior has not been quantitatively evaluated (Section 6.3), so a deployer would need to independently validate that the model's self-initiated speech is appropriate and reliable for their domain.