Kimi K2.5: Visual Agentic Intelligence

🎯 Pitch

Training an AI on vision tasks actually made its text reasoning better—a reversal of the usual interference between modalities. Kimi K2.5 exploits this bidirectional enhancement alongside a novel 'Agent Swarm' that spins off frozen sub-agents to slash task latency by up to 4.5×.

1. Executive Summary

This technical report introduces Kimi K2.5, an open-source multimodal agentic model that jointly optimizes text and vision through a series of training techniques—joint text-vision pre-training (early fusion at a constant, moderate vision-to-text ratio rather than late-stage injection), zero-vision SFT (activating visual reasoning and tool use from text-only supervised data, without human-designed visual trajectories), and joint text-vision reinforcement learning—which collectively establish bidirectional cross-modal enhancement where visual RL improves textual benchmarks. The model also introduces Agent Swarm, a self-directed parallel agent orchestration framework with Parallel-Agent Reinforcement Learning (PARL) (a decoupled architecture where a trainable orchestrator dynamically decomposes complex tasks into heterogeneous sub-problems executed concurrently by frozen, domain-specialized subagents), achieving state-of-the-art results across coding, vision, reasoning, and agentic benchmarks—including 96.1% on AIME 2025 and 78.4% on BrowseComp—while Agent Swarm reduces inference latency by up to 4.5× over single-agent baselines and improves WideSearch item-level F1 from 72.8% to 79.0%, establishing that parallel orchestration provides both qualitative and efficiency gains only when tasks demand broad exploration or simultaneous handling of independent sub-problems.

2. Context and Motivation

The Core Problem: How Do You Build a General-Purpose Agent That Efficiently Combines Vision, Language, and Action?

The fundamental problem Kimi K2.5 addresses is deceptively simple to state but extraordinarily difficult to solve: how do you build a single model that can simultaneously perceive the visual world, reason deeply about it, take real-world actions through tools, and orchestrate multiple parallel sub-agents to handle complex, long-horizon tasks efficiently?

This is not three separate problems. It is one interconnected problem. A model that excels at visual perception but cannot reason about what it sees cannot act. A model that reasons well but cannot orchestrate parallel execution will be too slow for practical use. A model that handles tool use but degrades in linguistic capability when trained on vision is a net loss. The Kimi team recognized that the dominant paradigm in multimodal AI—building vision-capable models by bolting vision components onto pre-trained language models—creates a cascade of subtle pathologies that compound as you push toward general-purpose agentic deployment. K2.5 is fundamentally a response to this realization, proposing that joint optimization of text and vision from the earliest stages of training, combined with a new paradigm for parallel agent orchestration, is the path forward.

To understand why this matters, we need to appreciate the three interconnected gaps the paper identifies—gaps that exist not because prior work was careless, but because they reflect genuine technical tensions.

Gap 1: Multimodal Models Trade Off Language and Vision Capability—But This Is a Design Choice, Not a Law of Nature

The standard approach to building vision-language models (VLMs) follows a well-established template. You take a powerful, pre-trained LLM—a model that already possesses strong linguistic reasoning capabilities—and you add visual processing to it, typically by training a vision encoder and a projector module that maps visual features into the LLM's token space. This is the approach taken by models like LLaVA, Qwen-VL, and many others. The intuition is reasonable: linguistic competence is the foundation, and visual understanding can be layered on top as a post-hoc addition.

The problem, as the Kimi team's experiments reveal (Section 2.1, Table 1), is that when and how you introduce vision data fundamentally shapes the tradeoff landscape. The conventional wisdom—which the paper explicitly calls out by citing Seed1.5-VL [21] and Qwen3-VL [8] as exemplars—is that vision data should be introduced late in pre-training and at high ratios (e.g., 50% or higher). The rationale is efficiency: since the model already understands language, you can aggressively train it on vision data in a concentrated burst, minimizing the compute spent on vision and preserving linguistic quality.

However, the paper's ablation studies tell a different story. By fixing the total vision-text token budget and varying both the vision-to-text ratio (10:90, 20:80, 50:50) and the timing of vision injection (early, mid, late), the authors find a surprising result: early fusion with a lower vision ratio yields better results across both vision and text benchmarks. Table 1 shows that the "Early, 10:90" configuration achieves 25.8 on Vision Knowledge (vs. 24.2 for Late, 50:50), 43.8 on Vision Reasoning (vs. 39.0), and crucially, 58.5 on Text Reasoning (vs. 57.8). The conventional aggressive, late-stage approach actually produces worse performance on both modality axes.

But Table 1 alone doesn't capture the full dynamic. Appendix B.1 and Figure 9 reveal a critical phenomenon: when vision data is introduced at mid or late stages, text performance exhibits a "dip-and-recover" pattern. As soon as vision tokens enter the training distribution, linguistic capability initially degrades before gradually recovering. The paper attributes this to "modality domain shift—the sudden introduction of vision tokens disrupts the established linguistic representation space, forcing the model to temporarily sacrifice text-specific competence for cross-modal alignment." In contrast, early fusion avoids this representation collapse entirely, producing smooth learning curves across all capabilities.

This is a significant finding because it challenges a deeply held assumption in the VLM community: that text-first training with late vision injection is the safe, conservative choice that preserves language quality. The paper's evidence suggests the opposite—that late injection is actually the riskier approach because it forces the model to undergo a disruptive domain shift that permanently compromises the unified representations that could have been built from the beginning. Early fusion, with a moderate, constant vision-to-text ratio, allows both modalities to co-evolve, producing representations where "text bootstraps vision, [and] vision refines text" (Section 1).

This gap was not obvious before K2.5 because prior work often studied vision and text capabilities in isolation, or accepted the late-fusion paradigm without systematically testing alternatives under controlled token budgets. The paper's controlled experiment—varying timing and ratio while fixing total tokens—provides the first clear evidence that the late-fusion approach is suboptimal.

Gap 2: Supervised Fine-Tuning on Visual Trajectories Is Scarce, Expensive, and Potentially Counterproductive for Generalization

Once a model has been pre-trained on vision and text jointly, a new problem emerges: how do you teach the model to use its visual understanding for practical tasks—like calling tools, manipulating images, or performing visual reasoning? The standard approach is supervised fine-tuning (SFT) on manually annotated or prompt-engineered visual trajectories: human-created examples of the model looking at an image, reasoning about it, and producing a correct tool call or answer.

This approach has a fundamental limitation: high-quality visual SFT data is scarce and narrow in diversity. The paper notes (Section 2.2) that conventional methods "are limited in diversity, often restricting visual reasoning to simple diagrams and primitive tool manipulations (crop, rotate, flip)." If you want your model to handle the vast space of possible visual tasks—pixel-level operations like object counting via binarization, spatial reasoning about object locations, fine-grained OCR on complex documents—manually annotating sufficient training examples is economically infeasible.

But the problem is worse than scarcity alone. The paper reports a counterintuitive empirical finding: when they compared text-only SFT ("zero-vision SFT") against SFT that includes human-designed visual trajectories, the visually-augmented SFT actually performed worse on visual, agentic tasks. The authors hypothesize that this is because joint pre-training (as described in Section 2.1) already establishes strong vision-text alignment. The model already "knows" how to connect visual inputs to linguistic concepts and reasoning patterns. Adding narrow, hand-designed visual trajectories during SFT effectively introduces a distribution shift in the style of visual reasoning, constraining the model to a subset of behaviors that happen to be represented in the SFT data and preventing the natural generalization that joint pre-training could otherwise support.

This finding—that zero-vision SFT is not just an acceptable shortcut but actually superior for generalization—represents a fundamental insight about the division of labor between pre-training and post-training in multimodal models. If joint pre-training has already established robust cross-modal representations, then SFT's role should be to teach the model reasoning patterns and tool-use behaviors in a modality-agnostic way. Vision-specific SFT, counterintuitively, narrows the model's generalization. The model can learn to process specific visual inputs (like a maze or a pie chart) through programmatic operations in IPython during the RL stage, without needing explicit visual SFT to bootstrap that capability.

This is important for practitioners because it fundamentally changes the data strategy for building multimodal agents: invest heavily in joint pre-training, use abundant and diverse text-only SFT for post-training, and let RL handle visual task specificity. The alternative—expensive, narrow visual SFT—is not just costly but actively harmful to generalization.

Gap 3: Sequential Agent Execution Is a Performance and Latency Bottleneck That Cannot Be Fixed by Better Single-Agent Reasoning Alone

The third gap is the most practically consequential for real-world deployment. Existing agentic models, including state-of-the-art systems like Kimi K2-Thinking, operate under a sequential execution paradigm: the model reasons for some number of steps, calls a tool, observes the result, reasons again, calls another tool, and so on. This sequential structure imposes a hard physical constraint: inference latency scales linearly with the number of tool calls. If a task requires 100 tool calls, the user waits for 100 sequential round-trips, regardless of how fast each individual call is.

The paper frames this problem starkly in Section 3:

"Even systems capable of hundreds of reasoning steps, such as Kimi K2-Thinking [1], suffer from linear scaling of inference time, leading to unacceptable latency and limiting task complexity."

This isn't just an engineering inconvenience. It fundamentally limits what kinds of tasks are feasible. As agentic workloads grow in scope—"building a complex project that involves massive-scale research, design, and development"—the sequential paradigm becomes a bottleneck not just for speed but for capability. Many real-world tasks are inherently parallelizable: they require gathering information from dozens of independent sources, analyzing multiple documents simultaneously, or exploring several design alternatives concurrently. A sequential agent must do all of these one at a time, exhausting its reasoning-step budget and the user's patience.

Prior approaches to this problem fall into two categories, both of which the paper identifies as insufficient. Static decomposition approaches pre-define a fixed set of sub-agents with fixed roles and hand-crafted coordination logic. These systems (which the paper references through Anthropic's multi-agent work [5, 7]) require explicit engineering for each task domain and cannot dynamically adapt to novel task structures. Heuristic parallelization pre-specifies when and how the model should spawn parallel sub-agents—for example, "if you see a list of URLs, spawn one agent per URL." This approach is brittle; it cannot learn when parallelism is actually beneficial versus when it adds coordination overhead without meaningful gains.

The paper explicitly argues that parallelism is not inherently good. Spawning many sub-agents that perform useless work or that create coordination complexity without reducing end-to-end latency is harmful. The decision of whether, when, and how to parallelize must be learned from task outcomes, not pre-specified by engineers.

This is where the paper positions its Agent Swarm and PARL framework. Unlike prior multi-agent systems, Agent Swarm does not pre-define sub-agents or pre-specify parallelization heuristics. The orchestrator learns, through RL with task-level rewards, to dynamically create sub-agents with custom system prompts, assign them tasks, and schedule their execution. Crucially, the framework measures and optimizes critical steps—the cumulative length of the longest path through the task execution graph, analogous to the critical path in project management—rather than total work performed. This incentivizes effective parallelism (sub-agents that reduce the longest dependency chain) rather than mere concurrency.

Why These Gaps Matter: Theoretical Significance and Real-World Impact

Theoretical significance. K2.5's findings challenge two foundational assumptions in the multimodal AI community: (1) that text-first, vision-later training is optimal, and (2) that visual SFT is necessary to activate visual capabilities. If early fusion and zero-vision SFT are superior, this implies a fundamentally different theoretical picture of how multimodal representations should be learned—one where cross-modal alignment is built continuously from the earliest stages, and where modality-specific fine-tuning can actually damage the generalization that joint training provides. This has implications for architectural design (when and how to fuse modalities), training curriculum design (ratios and timing), and data pipeline design (what data to annotate and when).

Real-world impact. The latency problem is not academic. As LLMs are deployed in production settings—customer support, financial research, software engineering, medical diagnosis—the wall-clock time to complete a task directly determines user satisfaction and economic viability. A system that takes 4.5× longer than necessary to complete a research task because it operates sequentially is not just slow; it is economically uncompetitive. Agent Swarm's demonstrated 3×–4.5× latency reduction on WideSearch (Figure 8) represents not just a performance improvement but a shift in the feasible task envelope—tasks that were previously impractical due to latency constraints become deployable.

Furthermore, the practical economics of training multimodal agents change substantially based on this paper's findings. If zero-vision SFT works, organizations can avoid the enormous cost of annotating diverse visual reasoning trajectories. If early fusion with moderate vision ratios is optimal, the pretraining data mix and training schedule can be designed accordingly from the start, avoiding expensive late-stage corrections. These are not marginal optimizations; they change the fundamental cost structure of building capable multimodal agents.

How This Paper Positions Itself Relative to Existing Work

Kimi K2.5 explicitly positions itself as a unified architecture for general-purpose agentic intelligence that integrates vision and language, thinking (long chain-of-thought) and instant (non-thinking) modes, chats and agents, within a single model. This positioning is important because it contrasts with several alternative approaches in the literature:

Versus modular vision-language systems (e.g., Qwen3-VL [8], Seed1.5-VL [21]): These models treat visual capability as an add-on to a language backbone, typically with late-stage vision injection. K2.5 argues for native multimodality—co-training from the earliest stages—and provides controlled experimental evidence that this yields better cross-modal performance.

Versus single-agent reasoning specialists (e.g., Kimi K2-Thinking [1], DeepSeek-V3.2 [14]): These models push the frontier of sequential reasoning depth but are inherently limited by the linear scaling of inference time. K2.5's Agent Swarm introduces parallelism as a new scaling axis, orthogonal to reasoning depth, that enables faster completion of complex tasks without sacrificing quality.

Versus pre-defined multi-agent systems (e.g., Anthropic's multi-agent research system [7]): These systems achieve parallelism through human-engineered agent roles and coordination logic. K2.5's PARL framework learns parallelization policies from task rewards, enabling dynamic adaptation to novel task structures without per-domain engineering.

Versus test-time context management approaches (e.g., DeepSeek's Discard-all [14], ReSum [71]): These methods address the context-length bottleneck by compressing or discarding accumulated history reactively. Agent Swarm addresses the same bottleneck proactively through context sharding—each sub-agent maintains an independent, bounded context, and only task-relevant outputs are routed back to the orchestrator—preserving structural information that reactive truncation would lose. Importantly, the paper shows (Figure 7) that Agent Swarm outperforms Discard-all context management on BrowseComp, suggesting that structured parallelization is a more effective solution to the long-context problem than post-hoc truncation heuristics.

Versus prior RL for agents (e.g., Kimi-Researcher [2]): Traditional agentic RL optimizes a single agent's tool-use policy. PARL extends this by giving the orchestrator interfaces for sub-agent creation and task delegation, while keeping sub-agents frozen to avoid credit assignment ambiguity and training instability. This decoupled architecture is a deliberate design choice motivated by the observation that end-to-end co-optimization of orchestrator and sub-agents would suffer from sparse, noisy reward signals where "a correct final answer does not guarantee flawless subagent execution, just as a failure does not imply universal subagent error" (Section 3).

The paper also connects to the broader literature on compute-optimal training, though in a different sense than the Chinchilla scaling laws. Rather than optimizing the pretraining compute budget, K2.5 optimizes the inference-time compute structure—shifting work from sequential to parallel execution to minimize critical path length. This is a complementary form of efficiency optimization that applies at deployment time rather than training time.

In summary, K2.5's position is that true general agentic intelligence requires a model that is natively multimodal (not vision-as-add-on), jointly optimized across modalities (not text-first-then-vision), and capable of dynamic parallel orchestration (not just sequential reasoning). The paper provides principled evidence for each of these claims while acknowledging the boundaries of its approach—for instance, the orchestrator is trained, but sub-agents are frozen, and parallelism is learned through outcome rewards rather than assumed to be inherently beneficial.

3. Technical Approach

3.1 Reader Orientation

Kimi K2.5 is a single, unified model—a trillion-parameter mixture-of-experts transformer—that has been trained to simultaneously perceive images and videos, reason deeply about them using text, execute real-world actions through tools (web search, code, computer control), and dynamically orchestrate multiple parallel sub-agents to handle complex, long-horizon tasks. The problem it solves is the fragmentation and inefficiency in current multimodal agentic systems: existing approaches either sacrifice linguistic capability when adding vision (by bolting vision onto language models late in training), require expensive and narrow manually-annotated visual training data to activate visual tool use, or operate sequentially, causing inference latency to scale linearly with task complexity. The shape of K2.5's solution is a three-stage training pipeline—joint text-vision pre-training from an early stage at a moderate vision ratio, followed by text-only supervised fine-tuning that activates visual reasoning without visual examples, followed by joint text-vision reinforcement learning with a novel parallel-agent RL framework—that produces a model where text and vision mutually enhance each other and where complex tasks can be decomposed into parallel sub-tasks executed concurrently.

3.2 Big-Picture Architecture (Diagram in Words)

The K2.5 system has five major components, organized in a pipeline that flows from raw data to a deployable agentic model:

Kimi K2 Base Model — A pre-trained trillion-parameter mixture-of-experts (MoE) language model with 1.04T total parameters and 32B activated per token (384 experts, 8 activated, sparsity 48). This is the "brain" that all subsequent training builds upon.
MoonViT-3D Vision Encoder + MLP Projector — A native-resolution vision encoder (initialized from SigLIP-SO-400M) that processes images at their original resolutions using the NaViT patch-packing strategy, and extends to video by treating groups of four consecutive frames as spatiotemporal volumes with lightweight temporal pooling for 4× compression. The MLP projector bridges the ViT's output into the LLM's token embedding space.
Joint Pre-training Pipeline — Three stages: (a) standalone ViT training on image/video-text pairs (~1T tokens), (b) joint vision-text pre-training on 15T tokens at 4K context length with early fusion at a constant vision-to-text ratio, and (c) long-context mid-training extending context to 262K tokens via YaRN interpolation with high-quality data.
Post-Training Pipeline — Two phases: (a) zero-vision SFT using only text data to activate visual reasoning and tool use, and (b) joint text-vision reinforcement learning with a unified agentic RL environment, optimizing a policy using a token-level clipped objective with outcome-based, generative, and task-specific visual rewards.
Agent Swarm Orchestration Framework — A dynamic parallel execution system where a trainable orchestrator (the K2.5 model) creates frozen sub-agents (from intermediate policy checkpoints) with custom system prompts, decomposes complex tasks into subtasks, and schedules parallel execution, all optimized via Parallel-Agent Reinforcement Learning (PARL) using critical-step constraints and auxiliary rewards.

Information flow: Raw text and vision data enter the pre-training pipeline → the MoonViT-3D encodes images/video into patch sequences → the MLP projector maps these to token embeddings → the Kimi K2 MoE backbone processes mixed vision-text token sequences with early fusion → post-training applies text-only SFT (activating visual reasoning via programmatic IPython operations) → joint RL refines both text and visual task performance using domain-organized experts and a unified RL environment → for agentic tasks, the trained model serves as an orchestrator that dynamically creates sub-agents and delegates parallel subtasks.

3.3 Roadmap for the Deep Dive

First, the Kimi K2 base model architecture and training (Section 4.1 in the paper)—because K2.5 is built by extending K2, and understanding the base model's scale, MoE structure, and optimizer is prerequisite for the multimodal extensions.
Second, the MoonViT-3D vision encoder and MLP projector (Section 4.2)—because the specific architecture choices (NaViT packing, 3D temporal compression, shared image/video parameters) determine how visual information enters the system and what video capabilities are possible.
Third, the joint pre-training strategy and the early fusion decision (Sections 2.1 and 4.3)—because the paper's central claim about cross-modal enhancement rests on how and when vision data is introduced during pre-training, and the controlled experiments that justify early fusion over late fusion need detailed examination.
Fourth, the zero-vision SFT methodology (Section 2.2)—because this counterintuitive finding (that text-only SFT activates vision capabilities better than vision-inclusive SFT) is a key design choice that shapes the post-training data strategy and the division of labor between SFT and RL.
Fifth, the joint multimodal RL system (Sections 2.3 and 4.4)—because this is where the model learns to reliably use visual inputs, where cross-modal transfer is empirically demonstrated, and where the specific policy optimization algorithm, reward functions, and token-efficiency techniques (Toggle) are defined.
Sixth, the Agent Swarm and PARL framework (Section 3)—because this is the novel contribution for parallel agent orchestration, requiring explanation of the decoupled architecture, the critical-step metric, the PARL reward decomposition, and the prompt construction strategy.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that general agentic intelligence emerges from jointly optimizing text and vision across all training stages (pre-training, SFT, RL) combined with a learned parallel orchestration framework that transforms sequential agent execution into dynamically parallelized sub-task execution.

3.4.1 Kimi K2 Base Model Architecture

The foundation of Kimi K2.5 is Kimi K2, a trillion-parameter mixture-of-experts (MoE) transformer model. Understanding K2's architecture is essential because K2.5's multimodal components are added onto this base, and the paper makes specific claims about how visual training affects the base model's existing linguistic capabilities.

Parameter scale and sparsity. Kimi K2 contains 1.04 trillion total parameters with 32 billion activated parameters per token. The MoE architecture uses 384 experts with 8 experts activated per token, yielding a sparsity factor of 48 (384/8). This means for every token processed, only 8 of the 384 expert feed-forward networks are active, dramatically reducing the per-token compute cost compared to a dense model of equivalent total parameter count. The model was pre-trained on 15 trillion high-quality text tokens.

Optimizer: MuonClip with QK-Clip. The paper states that K2 employs the "token-efficient MuonClip optimizer with QK-Clip for training stability." MuonClip is referenced from Jordan et al. (2024) [30] and Liu et al. (2025) [34]. Muon is an optimizer that operates on the hidden layers of neural networks using a Newton-Schulz iteration to compute matrix square roots, providing an alternative to Adam that has been shown to scale efficiently. MuonClip extends this with clipping mechanisms. QK-Clip specifically refers to clipping applied to the Query-Key attention weights for stability. This optimizer choice matters for K2.5 because the same optimizer is used during multimodal joint pre-training and RL, and the paper later describes a token-level clipping mechanism in the RL objective that borrows conceptually from this design philosophy.

Why MoE for a multimodal agentic model? The paper does not provide an explicit ablation justifying MoE over dense architectures, but the practical rationale is clear from context: supporting 256K context windows while processing high-resolution images and video frames (which generate many vision tokens per input) requires enormous model capacity. A dense 1T-parameter model would be prohibitively expensive for both training and inference. The MoE architecture allows the model to have 1T parameters of capacity (storing diverse knowledge across experts) while only activating 32B parameters per token, making deployment feasible. This architecture also naturally supports the agent swarm paradigm, where multiple sub-agents (themselves frozen MoE models) run concurrently—if each sub-agent were a dense 1T-parameter model, the total memory and compute requirements would be impractical.

Relationship to K2.5. The paper describes K2.5 as built "upon Kimi K2 through large-scale joint pre-training on approximately 15 trillion mixed visual and text tokens." The architectural components of K2 (MoE layers, attention mechanism, tokenizer) are preserved; what changes is the addition of the vision encoder and the training data distribution. The base model provides the linguistic and reasoning foundation that joint training then extends into the multimodal domain.

3.4.2 MoonViT-3D: Native-Resolution Vision Encoder with Temporal Extension

The vision encoder is where visual information first enters the K2.5 system. The paper makes several non-obvious architectural choices that collectively enable efficient, high-resolution image processing and seamless video understanding within a single shared parameter space.

Initialization from SigLIP. MoonViT is initialized from SigLIP-SO-400M, a 400-million-parameter vision transformer trained with a sigmoid loss for image-text contrastive pre-training (Zhai et al., 2023 [77]). SigLIP-SO-400M is a standard strong initialization for vision encoders in VLMs. The "SO" likely refers to a shape-optimized variant. This gives MoonViT a strong starting point for visual feature extraction before it is further trained on the specific data mixtures described in Section 4.3.

Native resolution processing via NaViT packing. The key architectural innovation for images is the adoption of the NaViT (Native Resolution Vision Transformer) patch packing strategy from Dehghani et al. (2023) [15]. In a standard ViT, all input images are resized to a fixed resolution (e.g., 224×224 or 336×336), producing a fixed number of patches regardless of the original image's aspect ratio or detail density. This is wasteful: a wide panorama or a long document screenshot loses information when squashed to a square, and a small icon wastes compute when padded to the standard resolution.

NaViT solves this by: (1) dividing each image into patches at its native resolution (or a user-specified resolution), (2) flattening the 2D grid of patches into a 1D sequence, and (3) concatenating multiple such 1D sequences (from different images or image regions) into a single batch with appropriate attention masking. The attention mask prevents patches from different images from attending to each other while allowing patches within the same image to attend freely. This packing strategy enables the model to train on images at arbitrary resolutions simultaneously, learning to extract features from both high-resolution detailed documents and low-resolution thumbnails within the same batch.

The paper states that MoonViT "eliminates the need for complex sub-image splitting and splicing operations." In previous approaches (such as some versions of Qwen-VL or LLaVA-NeXT), high-resolution images are often split into multiple overlapping sub-images, each processed at a fixed resolution, and the outputs are spliced back together. NaViT's patch packing is a cleaner solution: the image is processed as a single sequence of patches at its native resolution, with the transformer handling the full spatial context directly.

Temporal extension for video: MoonViT-3D. The paper's novel contribution to the vision encoder is the extension from 2D image processing to 3D spatiotemporal processing for video understanding. The key design constraint is complete parameter sharing between image and video processing—there is no separate video encoder module. The motivation is explicit:

"To maximize the transfer of image understanding capabilities to video, we introduce MoonViT-3D with a unified architecture, fully shared parameters, and a consistent embedding space."

The mechanism works as follows:

Temporal grouping: Up to four consecutive video frames are grouped into a spatiotemporal volume.
Joint flattening and packing: The 2D patches from all four frames within the group are "jointly flattened and packed into a single 1D sequence." This means the patches from frame_t × patch_row × patch_col are concatenated into one long sequence, mixing spatial and temporal dimensions.
Unified attention: The "identical attention mechanism operates seamlessly across both space and time." Because the patches are simply concatenated into a 1D sequence with appropriate positional encodings (which must encode both spatial position within each frame and temporal position across frames), the standard self-attention mechanism can attend across both spatial neighbors and temporal neighbors without architectural modification.
Temporal pooling for compression: Before the patch sequence is passed through the MLP projector into the language model, "lightweight temporal pooling aggregates patches within each temporal chunk, yielding 4× temporal compression to significantly extend feasible video length." The paper specifies that this pooling happens "at the patch level"—patches at the same spatial position across the four frames in a group are averaged. This reduces the vision token count for a video by a factor of 4, meaning a video that would otherwise produce 4N tokens produces only N tokens, enabling the model to process videos up to 4× longer within the same context window budget.

What the temporal extension enables. The paper claims this design achieves "strong video understanding performance without requiring specialized video modules or architectural bifurcation." The key benefit is transfer learning: "knowledge and ability obtained from image pretraining transfers holistically to videos through one shared parameter space and feature representation." If the model learns to recognize objects, text, or spatial relationships from millions of static images, that knowledge applies directly to video frames because the encoder processing each frame is literally the same network with the same weights. The only addition is the ability to attend across frames, which the unified attention mechanism handles naturally.

However, the paper acknowledges a limitation: "While the extra temporal attention improves understanding on high-speed motions and visual effects," implying that the simple joint-packing approach (where cross-frame attention is purely learned through positional encodings and self-attention) may not capture fast, complex motion dynamics as effectively as architectures with explicit temporal convolutions or dedicated motion streams. This is a deliberate tradeoff: simplified architecture with shared parameters versus specialized motion processing. The video benchmark results (Table 4: 70.4% on MotionBench, which specifically tests fine-grained motion understanding) suggest this tradeoff is reasonable but leaves room for specialized video architectures to potentially outperform on motion-specific tasks.

Design choice: why not a separate video encoder? The alternative approach—training a separate 3D vision transformer for video with temporal convolutions or factorized attention—would create two problems: (1) it would require duplicating the vision encoder parameters, increasing model size, and (2) it would prevent knowledge transfer from image pre-training to video understanding. By sharing all parameters, the 1T tokens of image pre-training (described in Section 4.3) directly benefit video tasks. This is a bet that the efficiency and transfer benefits of parameter sharing outweigh the accuracy benefits of a specialized video architecture, and the video benchmark results in Table 4 (e.g., 87.4% on Video-MME, 86.6% on VideoMMMU, state-of-the-art on LVBench and LongVideoBench) provide empirical support for this bet in the regime the paper evaluates.

MLP projector. Between the vision encoder and the language model sits an MLP (multi-layer perceptron) projector. The paper does not specify the exact architecture (number of layers, hidden dimensions), but its function is standard: map the vision encoder's output embeddings (which live in the ViT's embedding space, dimension likely 1152 or 1280 based on SigLIP-400M variants) to the language model's token embedding space (which, for a 1T-parameter MoE, is likely in the range of 4096–8192 dimensions). This projector is trained from scratch during the ViT training stage (described in Section 4.3) and then jointly updated with the full model during joint pre-training.

3.4.3 Joint Pre-Training Strategy: The Early Fusion Decision

This section of the paper is where K2.5 makes its most consequential architectural claim: early fusion of vision and text tokens, at a moderate constant ratio, outperforms the conventional approach of late-stage, high-ratio vision injection given a fixed total vision-text token budget. The evidence for this claim comes from a controlled ablation study, and the implications reshape the recommended approach for building multimodal LLMs.

The optimization problem. The paper frames the key design question explicitly:

"Given a fixed vision-text token budget, what is the optimal vision-text joint-training strategy?"

This is a resource allocation problem. Pre-training compute is expensive, and the vision and text token budgets are finite. How should you allocate your vision tokens across the course of pre-training? The three degrees of freedom are: (1) when to first introduce vision data (timing), (2) what fraction of tokens in each training step should be vision tokens (ratio), and (3) how these choices interact.

The conventional approach and why it exists. The paper identifies the conventional wisdom as: "introducing vision tokens predominantly in the later stages of LLM training at high ratios (e.g., 50% or higher) should accelerate multimodal capability acquisition, treating multimodal capability as a post-hoc add-on to linguistic competence." This approach is cited as being used by Seed1.5-VL [21] and Qwen3-VL [8]. The rationale is efficiency: you spend most of your compute budget training a pure language model (which is cheaper per token because vision tokens require running the vision encoder forward pass), and only at the end do you inject a concentrated burst of vision data to "teach" the model to process images. The hope is that linguistic capabilities, already established through text-only pre-training, will be preserved while vision capabilities are rapidly acquired.

The ablation experiment (Table 1, Figure 9). The paper tests three vision injection timings (Early, Mid, Late) crossed with three vision-to-text ratios (10:90, 20:80, 50:50), while fixing the total vision and text token budgets. The configurations are described as:

Early fusion, low ratio (10:90): Vision data is introduced from the beginning at a constant 10% vision-to-text ratio. This means that for every 100 tokens the model sees during pre-training, approximately 10 are vision tokens (image patches) and 90 are text tokens.
Mid fusion, medium ratio (20:80): Vision data is introduced at the midpoint of pre-training at a 20:80 ratio.
Late fusion, high ratio (50:50): Vision data is introduced only in the final 20% of pre-training at a 50:50 ratio. This is the conventional approach.

The paper states that "to strictly meet the targets for different ratios, we pre-trained the model with text-only tokens for a specifically calculated number of tokens before introducing vision data." This ensures that total vision tokens consumed is identical across configurations, isolating the effect of timing and ratio rather than total vision compute.

Results (Table 1). The early fusion, 10:90 configuration achieves the best or near-best performance across all six evaluation categories:

Vision Knowledge: 25.8 (vs. 24.2 for Late/50:50)
Vision Reasoning: 43.8 (vs. 39.0 for Late/50:50, a 4.8-point gap)
OCR: 65.7 (vs. 61.5 for Late/50:50, a 4.2-point gap)
Text Knowledge: 45.5 (vs. 43.1 for Late/50:50)
Text Reasoning: 58.5 (vs. 57.8 for Late/50:50)
Code: 24.8 (vs. 24.0 for Late/50:50)

The pattern is striking: early fusion with a lower vision ratio wins on every metric, including text-only capabilities. The conventional approach of late, aggressive vision injection produces the worst performance on both vision and text tasks. The paper's summary of this finding is direct:

"Surprisingly, we found that the vision ratio has minimal impact on final multimodal performance. In fact, early fusion with a lower vision ratio yields better results given a fixed total vision-text token budget."

Note the phrasing "vision ratio has minimal impact"—this refers to the finding that among the early fusion configurations, varying the ratio from 10% to 50% doesn't dramatically change performance (all early configurations outperform all late configurations). The timing, not the ratio, is the dominant factor.

The "dip-and-recover" phenomenon (Figure 9, Appendix B.1). The paper provides a mechanistic explanation for why late fusion underperforms. In Appendix B.1 and Figure 9, the full learning curves are shown for all configurations across six evaluation axes. The critical observation:

"We observe a 'dip-and-recover' pattern in text performance during mid-fusion and late-fusion stages: when vision data is first introduced, text capability initially degrades before gradually recovering. We attribute this to the modality domain shift—the sudden introduction of vision tokens disrupts the established linguistic representation space, forcing the model to temporarily sacrifice text-specific competence for cross-modal alignment."

In other words, when you take a model that has been trained exclusively on text for trillions of tokens, its internal representations—the way neurons encode concepts, the attention patterns, the feed-forward network specializations—are optimized for processing text. When you suddenly introduce vision tokens (image patches encoded into vectors), those same neurons and attention heads receive inputs from a completely different distribution. The model must reorganize its representations to accommodate both modalities, and during this reorganization, its text-processing ability temporarily degrades. With enough additional training, it recovers, but the paper's evidence suggests this recovery is never complete—the model settles into a representation space that is less optimal for text than the one it started from.

Why early fusion avoids this. The paper explains:

"In contrast, early fusion maintains a healthier and more stable text performance curve throughout training. By co-optimizing vision and language from the outset, the model naturally evolves unified multimodal representations without the shock of late-stage domain migration."

The key insight is that if vision tokens are present from the beginning, the model learns its fundamental representation space to accommodate both modalities jointly. There is no "before vision" representation that gets disrupted; the representations co-evolve from the start. This is why the early fusion configuration achieves better text reasoning (58.5) than the late fusion configuration (57.8) despite having the same total text tokens—the representation space it learned was better structured from the beginning because it was trained on a multimodal distribution the entire time.

What this means for the broader VLM community. This finding challenges the dominant paradigm. The paper explicitly describes this as a "different story" from conventional wisdom [8, 21]. The implication for practitioners is:

"This motivates our native multimodal pre-training strategy: rather than aggressive vision-heavy training concentrated at the end, we adopt a moderate vision ratio integrated early in the training process, allowing the model to naturally develop balanced multimodal representations while benefiting from extended co-optimization of both modalities."

The phrase "native multimodal pre-training" is important—it positions K2.5 not as a language model with vision added, but as a model that was always intended to process both modalities. This is a philosophical shift as much as a technical one: the goal is not to preserve language capability while adding vision; the goal is to build representations that are inherently cross-modal.

Practical implementation in K2.5. Based on this ablation, K2.5's actual pre-training (Section 4.3) uses early fusion with a constant vision-to-text ratio throughout the 15T-token joint pre-training phase. The paper does not explicitly state the exact ratio used in the final model (the ablation tested 10:90, 20:80, 50:50; the final model likely uses something close to 10:90 based on the results), but the principle is clear: constant, moderate ratio, from the beginning.

Nuances and caveats. The paper is careful not to claim that early fusion is universally optimal. The ablation is run at a specific scale (the token budgets and model size are not fully specified, but they're clearly at the scale of a large foundation model given the 15T token budget). The finding might interact with model scale, data quality, or architecture in ways not explored. Additionally, the paper uses a specific evaluation suite—the six categories in Table 1—that emphasizes knowledge, reasoning, OCR, and code. Tasks requiring pure linguistic fluency (like story generation or translation) are not separately reported, and it's possible that late fusion preserves some text-specific capabilities that this evaluation doesn't capture. However, the consistency of the advantage across all six reported categories is compelling.

3.4.4 ViT Training Stage (Standalone Vision Encoder Training)

Before joint pre-training begins, the vision encoder must be capable of extracting useful features from images and videos. The paper describes a two-stage process for establishing this capability (Section 4.3, "ViT Training Stage").

Stage 1: Continual pre-training of MoonViT-3D with a small language model. The MoonViT-3D is "continual pre-trained from SigLIP on image-text and video-text pairs." The term "continual pre-trained" means that rather than training from scratch, the weights are initialized from the SigLIP-SO-400M checkpoint (which was trained on web-scale image-text pairs with contrastive learning) and then further trained on K2.5's specific data mixture. This is important because SigLIP already provides strong visual representations; the continual pre-training adapts these representations to K2.5's data distribution and task requirements.

The training targets are diverse: "image alt texts, synthetic captions of images and videos, grounding bboxes, and OCR texts." "Grounding bboxes" refers to bounding box coordinates for object detection/localization tasks. "Synthetic captions" means captions generated by another model (likely an earlier version of Kimi or a captioning-specific model). The diversity of targets is designed to ensure the vision encoder captures both semantic understanding (captions), fine-grained localization (bounding boxes), and text reading (OCR).

Loss function. Unlike the implementation in Kimi-VL [54] (which used contrastive loss), this continual pre-training "does not include a contrastive loss, but incorporates solely cross-entropy loss $\mathcal{L}_{\text{caption}}$ for caption generation conditioned on input images and videos." This means the training objective is pure next-token prediction: given the visual input (image or video), generate the associated text (caption, bounding box coordinates, OCR text). This is simpler than contrastive approaches and aligns with how the encoder will be used downstream (as a feature extractor whose outputs are fed into an autoregressive language model).

Scale and efficiency. The first stage consumes "about 1T tokens with very few training FLOPs." This is achieved by using a small language model as the text decoder: "we update the MoonViT-3D to align it with Moonlight-16B-A3B via the caption loss." Moonlight-16B-A3B is a small MoE model (16B total, 3B activated per token). By pairing the large vision encoder (400M parameters) with a small language model, the forward and backward passes through the language model are cheap, and most of the compute goes to updating the vision encoder. The output of this stage is a vision encoder that "can primarily understand high-resolution images and videos."

Stage 2: Projector bridging. After the ViT is trained, a "very short second stage follows, updating only the MLP projector to bridge the ViT with the 1T LLM for smoother joint pre-training." This stage is described as brief, and its purpose is to align the vision encoder's output space with the language model's input embedding space before full joint training begins. During joint pre-training, both the ViT and LLM are updated together, but starting with a pre-aligned projector prevents large initial gradients that could destabilize training.

Design choice: why not use contrastive loss? The original Kimi-VL [54] used contrastive loss for vision-language alignment, inspired by CLIP and SigLIP. K2.5's decision to drop contrastive loss in favor of pure captioning loss represents a simplification. The rationale (implied rather than stated) is that contrastive loss is designed to align global image representations with text representations—useful for retrieval tasks—but for a model that will process images through autoregressive generation, the alignment is better achieved through the same next-token prediction objective that the LLM uses. The captioning loss forces the vision encoder to produce features that are directly useful for text generation, which is exactly what the downstream LLM needs. Contrastive loss optimizes for a different objective (discriminative alignment) that may not transfer as directly to generative tasks.

3.4.5 Joint Pre-Training and Long-Context Mid-Training Stages

With the vision encoder trained and the projector aligned, the main joint pre-training phase begins (Section 4.3).

Joint pre-training stage. This stage "continues from a near-end Kimi K2 checkpoint over additional 15T vision-text tokens at 4K sequence length." "Near-end" means the Kimi K2 base model was trained on nearly all of its 15T text tokens before vision data was introduced—the checkpoint is taken from late in K2's pre-training, not from the final fully-trained model. The 4K sequence length (4096 tokens) is relatively short by modern standards but is typical for the main pre-training phase before context extension.

The data recipe "extends Kimi K2's pre-training distribution by introducing unique tokens, adjusting data proportions with increased weight on coding-related content, and controlling maximum epochs per data source." The "unique tokens" likely refer to special tokens added for vision (image start/end markers, patch separators). The increased weight on coding is motivated by the agentic coding tasks K2.5 targets (SWE-Bench, LiveCodeBench). Controlling maximum epochs prevents the model from overfitting to small, high-quality datasets by limiting how many times they are repeated.

Long-context mid-training stage. After the main joint pre-training, a third stage "performs long-context activation with integrated higher-quality mid-training data, sequentially extending context length via YaRN interpolation, from 32K to 262K tokens." The sequence length extension proceeds in stages: starting at 32K, moving to longer contexts (the paper mentions 32K → 262K as the range), with the final model supporting 256K tokens at evaluation time.

YaRN interpolation. YaRN (Yet another RoPE extensioN, Peng et al., 2023 [44]) is a technique for extending the context window of models that use Rotary Position Embeddings (RoPE). In RoPE, attention scores between tokens depend on their relative positions modulated by sinusoidal functions. When you increase the context length beyond what the model was trained on, the positional encodings for distant tokens correspond to frequencies the model has never seen, causing attention patterns to degenerate. YaRN addresses this by interpolating the RoPE frequencies in a specific way (different frequency bands are interpolated differently based on their wavelength), allowing the model to generalize to longer sequences without full re-training. The paper states this "yields significant generalization improvements in long-context text understanding and long video comprehension."

The data during this stage is described as containing "Long Text, Long Video, Reasoning, Long-CoT [Chain-of-Thought]." The token volume decreases as sequence length increases (500B → 200B tokens), reflecting the fact that each training example is now much longer, so fewer total examples are needed.

Practical implications of the context window. The 256K context window is critical for the model's capabilities. For video understanding, with the 4× temporal compression, the model can process over 2000 frames (as stated in the evaluation of video benchmarks). For agentic tasks, the long context enables multi-step tool calling trajectories where tool outputs and reasoning traces accumulate—without aggressive context management, these trajectories can easily exceed 100K tokens. The large context window is also essential for Agent Swarm, where the orchestrator must maintain awareness of multiple parallel sub-agent outputs simultaneously.

3.4.6 Zero-Vision SFT: Activating Visual Capabilities Without Visual Training Data

This is perhaps the most counterintuitive finding in the paper: you can teach a multimodal model to use vision for tool calling and reasoning without showing it a single visual example during the SFT phase. The mechanism, constraints, and implications deserve careful examination.

The cold-start problem for multimodal RL. The paper identifies a specific technical problem: "Pretrained VLMs do not naturally perform vision-based tool-calling, which poses a cold-start problem for multimodal RL." After joint pre-training, the model can process images and generate text conditioned on images, but it does not spontaneously generate tool calls (Python code, web searches, file operations) in response to visual inputs. If you drop this model directly into RL training, the initial policy is so poor that it rarely produces reward-bearing trajectories, and the RL optimization signal is too sparse to improve the policy effectively. This is the "cold-start" problem: RL needs a somewhat reasonable initial policy to bootstrap from.

The conventional solution and its problems. The standard fix is to create SFT data with visual trajectories—human-annotated or model-generated examples of "here's an image, here's the chain of thought reasoning about it, here's the tool call to process it, here's the answer." The paper notes two problems with this approach:

Limited diversity: "Manually annotated or prompt-engineered chain-of-thought (CoT) data ... are limited in diversity, often restricting visual reasoning to simple diagrams and primitive tool manipulations (crop, rotate, flip)."
Counterproductive for generalization (the surprising finding): "Compared to zero-vision SFT, our preliminary experiments show that text-vision SFT yields much worse performance on visual, agentic tasks, possibly because of the lack of high-quality vision data."

The second point is the critical one. Not only is visual SFT data scarce—using what's available actually harms performance relative to using no visual SFT at all. The hypothesized mechanism is that narrow visual SFT trajectories constrain the model to a particular style of visual reasoning (e.g., always describing the image first, then reasoning, then calling a tool), and this prevents the model from developing the flexible, task-adaptive visual reasoning behaviors that joint pre-training made possible.

How zero-vision SFT works. The paper's insight is that tool-use behaviors can be transferred from text to vision domains through a proxy: IPython code execution. The paper states:

"We propose a novel approach, zero-vision SFT, that uses only text SFT data to activate the visual, agentic capabilities during post-training. In this approach, all image manipulations are proxied through programmatic operations in IPython, effectively serving as a generalization of traditional vision tool-use."

The mechanism: during text-only SFT, the model is trained on examples where a user asks a text-based question, and the model responds by writing Python code in an IPython environment, executing it, and providing the answer. For example, a text SFT example might be: "What is the sum of the first 100 prime numbers?" → The model writes a Python script to compute this, executes it, and returns the answer. Importantly, when the model encounters a visual question during subsequent RL or inference, the structure of the response—write code to process the input, execute it, return the result—transfers, even though the SFT examples never involved images. The model has learned the pattern of tool use, and because joint pre-training established a shared representation space for vision and text, the pattern generalizes across modalities.

What zero-vision SFT enables. The paper claims this approach activates "diverse reasoning behaviors, including pixel-level operations such as object size estimation via binarization and counting, and generalizes to visually grounded tasks such as object localization, counting, and OCR." The qualitative examples in Figure 12 support this: the model binarizes a maze image to find paths, applies color segmentation to a pie chart, and computes pixel-level difference maps for spot-the-difference tasks—all behaviors that were never explicitly demonstrated in SFT data but emerged because the model learned to use Python for analysis and transferred that skill to visual inputs.

The RL training curves (Figure 2) as evidence. Figure 2 shows RL training curves for four vision benchmarks (MMMU-Pro, MathVision, CharXiv, OCRBench), starting from the zero-vision SFT checkpoint. The x-axis is "RL flops" (compute spent on RL training), and the y-axis is accuracy. All four curves start at non-trivial accuracy levels (the starting point from zero-vision SFT) and improve monotonically with more RL training. This demonstrates that: (1) zero-vision SFT alone provides a reasonable initial policy (the cold-start problem is solved), and (2) RL further refines visual capabilities from this starting point.

Why this finding matters beyond K2.5. The zero-vision SFT finding implies that for multimodal models with strong joint pre-training, the expensive, labor-intensive process of creating visual SFT data may be unnecessary or even counterproductive. The division of labor should instead be: joint pre-training handles cross-modal alignment, text-only SFT teaches reasoning and tool-use patterns, and RL handles task-specific visual refinement. This is a significant practical insight for teams building multimodal agents: invest in pre-training quality and text SFT diversity rather than in visual annotation.

A note on limitations. The paper acknowledges that text-only SFT "alone activates visual reasoning and tool use" but that it "sometimes ignored" visual inputs and that "images may not be attended to when necessary." This is why RL is still needed: zero-vision SFT gives you a model that can use vision, but RL teaches it when and how reliably to use vision. The cold-start problem is solved by zero-vision SFT, but the reliability problem requires RL.

3.4.7 Joint Multimodal Reinforcement Learning: Policy Optimization

The RL phase is where K2.5's capabilities are refined and where the cross-modal transfer effects are observed. This section describes the policy optimization algorithm, the reward functions, and the token-efficiency technique (Toggle).

Policy optimization objective. The paper defines the RL objective in Equation 1:

$L_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \frac{1}{N} \sum_{j=1}^{K} \sum_{i=1}^{|y_j|} \text{Clip}\left( \frac{\pi_\theta(y_j^i | x, y_j^{0:i})}{\pi_{\text{old}}(y_j^i | x, y_j^{0:i})}, \alpha, \beta \right) \left( r(x, y_j) - \bar{r}(x) \right) - \tau \left( \log \frac{\pi_\theta(y_j^i | x, y_j^{0:i})}{\pi_{\text{old}}(y_j^i | x, y_j^{0:i})} \right)^2 \right]$

where:

$x$ is a problem sampled from dataset $\mathcal{D}$ ,
$K$ responses $\{y_1, ..., y_K\}$ are generated using the previous (old) policy $\pi_{\text{old}}$ ,
$\pi_\theta$ is the current policy being optimized (parameterized by $\theta$ ),
$|y_j|$ is the length (number of tokens) of the $j$ -th response,
$y_j^{0:i}$ is the prefix of the $j$ -th response up to (but excluding) the $i$ -th token,
$r(x, y_j)$ is the reward for response $y_j$ given problem $x$ ,
$\bar{r}(x) = \frac{1}{K} \sum_{j=1}^K r(x, y_j)$ is the mean reward across all $K$ responses for problem $x$ (the baseline),
$N = \sum_{j=1}^K |y_j|$ is the total number of generated tokens in the batch (the normalization factor),
$\alpha, \beta > 0$ are the clipping bounds for the token-level importance ratio,
$\tau > 0$ is the weight of the KL penalty term,
$\text{Clip}(r, \alpha, \beta)$ is a clipping function that returns $r$ if $\alpha \leq r \leq \beta$ , and otherwise returns $r$ but with the gradient zeroed out (i.e., it acts as a gradient mask rather than a value clip).

What this equation computes, operationally. For each token in each generated response, the algorithm: (1) computes the ratio of the current policy's probability of that token to the old policy's probability (the "importance ratio" or "log-ratio" after exponentiation), (2) clips this ratio to the interval $[\alpha, \beta]$ (specifically, allows gradients only for tokens within this range), (3) multiplies the clipped ratio by the advantage $(r(x, y_j) - \bar{r}(x))$ —the excess of the response's reward over the mean reward for that problem, and (4) subtracts a squared-log-ratio penalty weighted by $\tau$ that penalizes the policy for deviating too far from the old policy. The result is averaged over all tokens and all responses in the batch.

Token-level clipping versus PPO. The paper explicitly contrasts this with standard PPO clipping (Schulman et al., 2017 [50]):

"A key distinction from standard PPO clipping [50] is that our method relies strictly on the log-ratio to explicitly bound off-policy drift, regardless of the sign of the advantages."

In standard PPO, the clipping depends on both the importance ratio and the sign of the advantage—if the advantage is positive, the ratio is clipped to a maximum of $1+\epsilon$ ; if negative, to a minimum of $1-\epsilon$ . The K2.5 version decouples this: the ratio is always clipped to $[\alpha, \beta]$ regardless of the advantage sign, and the gradient is simply masked (zeroed out) for tokens with ratios outside this range. This is described as acting "as a simple gradient masking scheme: policy gradients are computed normally for tokens with log-ratios within the interval $[\alpha, \beta]$ , while gradients for tokens falling outside this range are zeroed out."

Why this form? The paper states that this mechanism "is designed to mitigate the off-policy divergence amplified by discrepancies between training and inference frameworks." In large-scale RL training for LLMs, the policy used to generate training data (the "old policy" or "behavior policy") is often served by a different inference system than the one used for training, leading to slight discrepancies in log-probabilities. Standard PPO can amplify these discrepancies because it uses the ratio in both the advantage term and the clipping decision. The K2.5 approach bounds the ratio directly, preventing any single token from exerting outsized influence on the gradient. The paper references Yao et al. (2025) [74] and Zhao et al. (2025) [78] as aligned with this strategy, noting that they "empirically find this mechanism essential for maintaining training stability in complex domains requiring long-horizon, multi-step tool-use reasoning."

The KL penalty term. The $-\tau (\log \frac{\pi_\theta}{\pi_{\text{old}}})^2$ term is a KL divergence penalty (specifically, the squared log-ratio approximates the KL divergence for small deviations). This encourages the policy to stay close to the old policy, preventing catastrophic forgetting of capabilities learned during SFT. The hyperparameter $\tau$ controls the strength of this penalty.

The optimizer. The paper states: "We employ the MuonClip optimizer [30, 34] to minimize this objective"—the same optimizer used for K2's pre-training.

The baseline $\bar{r}(x)$ . Using the mean reward across $K$ responses as a baseline is standard in policy gradient methods (it's a form of REINFORCE with baseline). It reduces variance by centering the rewards: responses that are better than average get a positive advantage and are up-weighted; responses worse than average get a negative advantage and are down-weighted. $K$ (the number of rollouts per problem) is a hyperparameter not specified in this section.

Relationship to Kimi K1.5. The paper notes that this loss function "departs from the policy optimization algorithm used in K1.5 [31]" specifically through the token-level clipping mechanism, suggesting this was a lesson learned from K1.5's training stability challenges.

3.4.8 Reward Functions: Outcome-Based, Generative, and Task-Specific Visual Rewards

The RL objective requires a reward function $r(x, y)$ that evaluates the quality of a response $y$ to a problem $x$ . K2.5 uses a heterogeneous set of reward functions tailored to different task types (Section 4.4.2).

Rule-based outcome rewards for verifiable tasks. For "tasks with verifiable solutions, such as reasoning and agentic tasks," the paper applies "a rule-based outcome reward." This means the reward is 1 if the model's final answer matches the ground truth (for math problems, coding problems with unit tests, etc.) and 0 otherwise. This is the simplest and most reliable reward when ground truth is available.

Budget-control reward for token efficiency. To "optimize resource consumption," the paper "also incorporates a budget-control reward aimed at enhancing token efficiency." The exact form of this reward is not specified in detail, but it penalizes responses that are unnecessarily long (i.e., that consume more tokens than needed to reach the correct answer). This is connected to the Toggle mechanism described later.

Generative Reward Models (GRMs) for open-ended tasks. For tasks where ground-truth is not available (e.g., chat quality, summarization, creative writing), the paper deploys Generative Reward Models:

"Kimi K2 leverages a self-critique rubric reward for open-ended generation [53], and K2.5 extends this line of work by systematically deploying Generative Reward Models (GRMs) across a broad range of agentic behaviors and multimodal trajectories."

GRMs are language models (likely fine-tuned versions of K2 or K2.5 itself) that evaluate a response according to a rubric. The paper emphasizes that GRMs "function not as binary adjudicators, but as fine-grained evaluators aligned with Kimi's values that are critical to user experiences, such as helpfulness, response readiness, contextual relevance, appropriate level of detail, aesthetic quality of generated artifacts, and strict instruction following."

The key advantage of GRMs over rule-based rewards is that they can capture nuanced preferences—a response that is technically correct but unhelpful or poorly formatted would receive a low GRM score, whereas a rule-based verifier would give it full credit. The disadvantage is that GRMs can be gamed (reward hacking) and may have their own biases. To mitigate this, the paper "employ[s] multiple alternative GRM rubrics tailored to different task contexts," essentially ensembling multiple reward models to reduce the risk of overfitting to any single one.

Task-specific visual rewards. For visual tasks, K2.5 uses specialized reward functions that provide more fine-grained supervision than binary correctness. The paper describes five types:

Visual grounding and point localization: "F1-based reward with soft matching." For grounding (bounding box prediction), the soft match is computed from Intersection over Union (IoU) between predicted and ground-truth boxes. For point localization, soft matches are derived from Gaussian-weighted distances under optimal matching (likely Hungarian matching to assign each predicted point to the closest ground-truth point, then computing a Gaussian-weighted similarity score). Using F1 (the harmonic mean of precision and recall) rather than raw accuracy ensures that both false positives (predicting objects that don't exist) and false negatives (missing objects) are penalized.
Polygon segmentation: "Rasterize the predicted polygon into a binary mask and compute the segmentation IoU against the ground-truth mask to assign the reward." This converts the polygon (a list of vertex coordinates) into a pixel-level binary image (1 inside the polygon, 0 outside) and computes the overlap with the ground-truth mask. The IoU is a continuous value between 0 and 1, providing a smoother reward signal than binary correctness.
OCR: "Normalized edit distance to quantify character-level alignment between predictions and ground-truth." Edit distance (Levenshtein distance) counts the minimum number of character insertions, deletions, and substitutions needed to transform the prediction into the ground truth. Normalizing by the ground-truth length gives a value in $[0, 1]$ where 1 is perfect and 0 is complete mismatch.
Counting: "Rewards are assigned based on the absolute difference between predictions and ground-truth." If the model predicts $k$ objects and the true count is $k^*$ , the reward might be $e^{-|k - k^*|}$ or $1/(1+|k-k^*|)$ —the paper doesn't specify the exact function, but it's a decreasing function of the error.
Complex visual puzzles: "Synthesize complex visual puzzle problems and utilize an LLM verifier (Kimi K2) to provide feedback." For open-ended visual reasoning tasks where rule-based scoring is difficult, the paper uses K2 as a judge model, similar to how GRMs are used for text tasks.

Why task-specific visual rewards? Binary correctness rewards are too sparse for visual tasks that require intermediate precision. A bounding box that is slightly too large is still partially correct, and a binary 0/1 reward would treat it the same as a completely wrong prediction. The soft rewards provide gradient signal that helps the model learn how to improve—making the bounding box tighter, adjusting the polygon boundary, correcting specific OCR characters—rather than just learning that it was wrong.

3.4.9 Toggle: Token-Efficient Reinforcement Learning

The paper identifies a fundamental tension in RL training for reasoning models: you want the model to be token-efficient (not generating unnecessarily long chains of thought), but if you constrain token usage too aggressively during training, the model fails to develop the ability to "think longer" on hard problems at test time.

The problem: length overfitting. The paper states:

"We observe a length-overfitting phenomenon: models trained under rigid budget constraints often fail to generalize to higher compute scales. Consequently, they cannot effectively leverage additional inference-time tokens to solve complex problems, instead defaulting to truncated reasoning patterns."

This is a critical insight for test-time compute scaling. If you train a model with a strict "no more than N tokens per problem" constraint, it learns patterns that fit within N tokens. At inference time, even if you allow 2N or 4N tokens, the model doesn't know how to use them productively—it has been optimized to solve problems within the budget, not to strategically deploy extra computation for harder cases.

The solution: Toggle. The paper proposes a training heuristic called Toggle that alternates between two phases: