Kimi-VL Technical Report

🎯 Pitch

A 2.8B-parameter vision-language model matches GPT-4o on agent tasks and smashes long-video benchmarks, but the real surprise is how its 'thinking' variant scales: test-time compute alone boosts math reasoning scores from 18.7% to 36.8% without any change to the base model weights.

1. Executive Summary

This technical report introduces Kimi-VL, an efficient open-source Mixture-of-Experts vision-language model that activates only 2.8B parameters in its language decoder yet competes with much larger models across multimodal reasoning, long-context understanding, and agent tasks. The paper also presents Kimi-VL-Thinking, a long-thinking variant developed through long chain-of-thought supervised fine-tuning and reinforcement learning—operationalizing test-time compute scaling via extended reasoning chains that improve with longer thinking token budgets (e.g., rising from 18.7% to 36.8% on MathVision as max thinking length scales from 1K to 16K tokens). The model achieves 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc within a 128K context window, while its native-resolution vision encoder MoonViT enables strong high-resolution perception—scoring 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro—establishing that a compact MoE architecture with only ~3B activated parameters can match or exceed flagship dense models like GPT-4o on agent tasks (8.22 vs. 5.03 on OSWorld) and surpass GPT-4o-mini across OCR and document understanding, though the largest gains from long-thinking emerge primarily on reasoning-heavy benchmarks rather than general perception tasks where performance saturates early (MathVista plateaus at 4K thinking tokens with no further improvement to 16K).

2. Context and Motivation

The Core Problem: Open-Source VLMs Lag Behind in Efficiency, Scalability, and Reasoning

The fundamental gap this paper addresses is a structural imbalance in the vision-language model landscape: while the open-source community has made rapid progress on pure language models—particularly with Mixture-of-Experts architectures (e.g., DeepSeek-V3, DeepSeek-R1) that achieve strong performance with sparse activation—the VLM space remains dominated by dense architectures that lack both the computational efficiency of MoE designs and the advanced reasoning capabilities that long chain-of-thought training enables.

This matters for several practical reasons the paper identifies (Section 1):

Deployment feasibility: A model that activates only 2.8B parameters during inference can run on hardware that would struggle with dense 7B+ parameter models, making advanced multimodal capabilities accessible to resource-constrained environments—edge devices, smaller organizations, and real-time applications where latency budgets are tight.
Research accessibility: The open-source community needs reference implementations that demonstrate how to combine architectural innovation (MoE, native-resolution vision encoding) with training methodology (long-CoT SFT, RL) in the multimodal domain, so that subsequent work can build on tested recipes rather than starting from scratch.
Reasoning depth: Complex multimodal tasks—college-level science problems, multi-step visual reasoning, video understanding requiring temporal inference—demand models that can engage in extended deliberation. The paper argues that open-source VLMs have not yet delivered this capability, leaving a gap between what proprietary systems (OpenAI o1, Kimi k1.5) can do and what the community can use and study.

The Specific Gap: No Open VLM Combines MoE Efficiency, Long Context, High-Resolution Perception, and Long-Thinking

Prior to this work, existing open-source VLMs exhibited a pattern of partial coverage—each addressing some dimensions while falling short on others. The paper explicitly identifies these shortcomings (Section 1):

Dense architectures dominate, sacrificing efficiency. Recent strong open-source VLMs—Qwen2.5-VL (Bai et al., 2025) and Gemma-3 (Gemma Team et al., 2025)—use dense transformer decoders. A dense 7B model activates all 7B parameters on every token, whereas an MoE model with 16B total parameters but only 2.8B activated per token achieves comparable or better performance while using roughly 2.6× fewer FLOPs per forward pass. The paper notes that this efficiency gap widens during training: the reported training throughput of Kimi-VL is approximately 60% higher than a comparable 7B dense VLM (Section 2.5, Parallelism subsection), meaning MoE architectures can be trained faster and cheaper on the same hardware budget.

Existing MoE VLMs have critical limitations. The two prior MoE-based VLMs the paper cites—DeepSeek-VL2 (Wu et al., 2024) and Aria (Li et al., 2024)—each have specific architectural or capability shortcomings:

DeepSeek-VL2 uses a traditional fixed-size vision encoder, which limits its ability to handle images at varying native resolutions. When an image exceeds the encoder's training resolution, it must be resized or split into sub-images with complex stitching logic (as in LLaVA-OneVision)—operations that can distort fine-grained visual information or break spatial coherence. More critically, DeepSeek-VL2 supports only a 4K context length, severely restricting its ability to process long videos, multi-page documents, or multi-image sequences. And it does not support long-thinking—no CoT training or RL-based reasoning enhancement.
Aria activates 3.9B parameters (slightly more than Kimi-VL's 2.8B) but the paper notes it "falls short in fine-grained visual tasks," suggesting its vision encoder or training recipe does not adequately preserve spatial detail for OCR, grounding, and high-resolution perception. Like DeepSeek-VL2, Aria does not support long-thinking.

Long-context multimodal understanding is underdeveloped. While text-only LLMs have pushed context windows to 128K+ tokens (and beyond, with techniques like RoPE extension), the paper observes that applying these techniques to multimodal inputs—where the content alternates between text tokens and visual feature embeddings—introduces additional challenges. Long videos and multi-page documents require the model to attend across long sequences where the signal is sparse (e.g., a key visual cue appears in one frame among thousands). The paper notes that DeepSeek-VL2's 4K context window makes it "insufficient for certain advanced applications that involve extremely long sequences or high-volume contextual information" and that prior work had not demonstrated robust needle-in-a-haystack retrieval for multimodal haystacks (video NIAH at full 128K length).

Long-thinking for multimodal reasoning is essentially absent from open-source. At the time of this work, the paper identifies only proprietary systems—OpenAI's o1 series and Kimi k1.5—as having demonstrated long chain-of-thought reasoning on multimodal inputs. Open-source VLMs like Qwen2.5-VL and Gemma-3, despite their strong standard benchmarks, do not support extended CoT generation or RL-based reasoning optimization. The paper frames this as a critical missing piece: just as DeepSeek-R1 showed that RL-driven long-thinking dramatically improves text-based reasoning, the multimodal domain needs an equivalent demonstration that these techniques transfer to visual reasoning tasks.

Fixed-resolution vision encoders create a tradeoff between detail and computational cost. Traditional vision encoders process images at a fixed resolution (e.g., 224×224 for early ViTs, or 384×384 for SigLIP-SO-400M). To handle high-resolution inputs, models must either resize (losing fine detail needed for OCR and small-object recognition) or split the image into crops (increasing sequence length quadratically and breaking spatial relationships between crops). The paper argues that neither approach is satisfactory for tasks like ScreenSpot-Pro, where the model must localize UI elements on 4K-resolution screenshots, or InfoVQA, where tiny chart labels and axis tick marks carry essential information. The native-resolution approach—processing images at their original aspect ratio and resolution without cropping or resizing—is identified as a key architectural innovation that prior open-source VLMs had not adequately implemented.

Prior Approaches and Where They Fall Short

The paper implicitly organizes prior work into several categories, each with identifiable limitations:

Dense VLMs with moderate context windows. Qwen2.5-VL (8.3B parameters, dense) and Gemma-3-12B-IT (12B parameters, dense) represent the state of open-source dense VLMs. While they achieve strong results on standard benchmarks, the paper's evaluation (Table 3) shows they are consistently outperformed or matched by Kimi-VL on key tasks: Qwen2.5-VL-7B loses on 19 out of 24 benchmarks despite having 2.59× more activated parameters; Gemma-3-12B-IT is beaten on OCR (InfoVQA: 83.2 vs. 43.8), agent tasks (ScreenSpot-Pro: 34.5 vs. unreported but implied weak), and long video understanding (Video-MME w/o subtitle: 67.8 vs. 58.2). More fundamentally, these dense models lack long-thinking capabilities entirely.

MoE VLMs with architectural compromises. DeepSeek-VL2 (4.1B activated, 28B total) uses an MoE architecture similar to Kimi-VL's but with a fixed-resolution vision encoder and 4K context limit. Table 3 shows Kimi-VL outperforming it on nearly every benchmark, with particularly large gaps on Video-MME (67.8 vs. not reported—presumably infeasible at 4K context), MLVU (74.2 vs. not reported), and OCRBench (867 vs. 811). This demonstrates that the MoE language decoder alone is insufficient—the vision encoder and context length are equally critical design choices.

Proprietary systems with limited accessibility. GPT-4o and GPT-4o-mini serve as strong baselines but are closed-source, have undisclosed architectures, and cannot be fine-tuned or deployed privately. The paper uses them primarily as benchmarks to demonstrate that an open model with ~3B activated parameters can compete with or exceed proprietary systems costing orders of magnitude more to serve. For example, Kimi-VL achieves 8.22 on OSWorld vs. GPT-4o's 5.03, and 35.1 on MMLongBench-Doc vs. GPT-4o-mini's 29.0.

Long-thinking proprietary models. OpenAI o1 and Kimi k1.5 demonstrated that RL-trained long-CoT reasoning substantially improves multimodal reasoning (Table 4: o1 achieves 74.8 on MMMU and 77.3 on MMMU-Pro; Kimi k1.5 achieves 74.9 on MathVista). However, neither is open-source, and their training recipes are described only at a high level. The paper positions Kimi-VL-Thinking as the first open-source model to replicate this paradigm, providing sufficient detail (Section 2.4: the RL objective, the warmup dataset construction, the length penalty and curriculum sampling strategies) that other researchers can build on the approach.

Test-time compute scaling in the text domain but not multimodal. The paper's long-thinking variant operationalizes a form of test-time compute scaling: the model generates longer reasoning chains at inference time, and performance improves with increased thinking token budgets (Figure 13). While this mirrors findings from the text domain (e.g., DeepSeek-R1's scaling behavior), the paper notes that no prior work had demonstrated such scaling curves for multimodal reasoning or explored the interaction between thinking length and benchmark characteristics (e.g., MathVista saturating at 4K tokens while MathVision continues improving to 16K).

How This Paper Positions Itself

The paper frames Kimi-VL not as a single-point breakthrough but as a systematic integration of architectural and training innovations that were previously demonstrated only in isolation or in the text domain:

From text to multimodal: The paper takes the MoE architecture proven effective in text-only models (Moonlight, similar to DeepSeek-V3) and shows how to successfully train it jointly on text and multimodal data while preserving language capabilities—a non-trivial challenge because the vision encoder's features must be integrated without disrupting the language model's pretrained representations.
From fixed to native resolution: The paper adopts the NaViT patch-and-pack paradigm for vision encoding but extends it with 2D RoPE to handle high-resolution spatial relationships, arguing that this combination is what enables strong performance on fine-grained visual tasks without the computational cost of multi-crop processing.
From text-only long context to multimodal long context: The paper demonstrates that RoPE extension from 50,000 to 800,000 base frequency—a technique well-established for text—works for multimodal sequences when applied with a carefully designed data mixture (25% long data, 75% replay of shorter data) that preserves short-context abilities while extending the context window to 128K.
From proprietary long-thinking to open-source long-thinking: The paper adapts the RL recipe from Kimi k1.5 to the multimodal domain, providing concrete details on the warmup dataset (constructed via prompt engineering to teach planning, evaluation, reflection, and exploration), the RL objective (online policy mirror descent with KL regularization), and training strategies (length penalty, curriculum sampling, prioritized sampling). This is positioned as a contribution to research reproducibility—other groups can now attempt to replicate and improve on multimodal long-thinking without needing to guess at the recipe.
An updated reasoning variant integrates capabilities: Kimi-VL-Thinking-2506 demonstrates that long-thinking can be combined with continued vision encoder training (MoonViT extended to 3.2M pixels) to simultaneously improve reasoning (MathVision: 36.8 → 56.9) and general perception (MMBench: 76.0 → 84.4, ScreenSpot-Pro: from unreported to 52.8), addressing the potential concern that long-thinking training might degrade non-reasoning abilities—a concern the paper acknowledges by showing that the 2506 version actually improves on both axes simultaneously.

The paper's position is essentially: the building blocks for an efficient, capable, open-source VLM exist, but no prior work has assembled them correctly. Dense architectures waste compute, existing MoE VLMs compromise on vision or context, and long-thinking has been locked behind proprietary APIs. Kimi-VL is the first model to demonstrate that these pieces can work together—and that when they do, a model with only 2.8B activated parameters can compete with systems 5–10× larger while offering capabilities (long-thinking, 128K multimodal context, native high-resolution perception) that even some larger models lack.

3. Technical Approach

This is primarily an empirical systems design paper whose core contribution is a careful integration of architectural and training innovations—MoE language decoder, native-resolution vision encoder, long-context activation, and long-thinking post-training—into a single open-source VLM that demonstrates these components can work together effectively, with the long-thinking variant operationalizing test-time compute scaling through extended chain-of-thought generation where accuracy improves monotonically with increased thinking token budgets.

3.1 Reader Orientation

Kimi-VL is a vision-language model that takes images, videos, and text as input and produces text responses—it can answer questions about visual content, solve math problems with diagrams, navigate computer interfaces, transcribe handwritten documents, and engage in extended step-by-step reasoning. The problem it solves is the tension between efficiency, capability, and openness in VLMs: prior open-source models were either dense (wasting compute), lacked long-context understanding (unable to process long videos or multi-page documents), used fixed-resolution vision encoders (losing fine visual detail), or could not perform extended reasoning (no long-thinking). The "shape" of the solution is a three-component architecture—a native-resolution vision encoder that preserves detail without cropping, an MLP projector that bridges vision to language, and an MoE language decoder that activates only a small fraction of its parameters per token—trained through a multi-stage pipeline that progressively builds capabilities (language, then joint multimodal, then long-context, then instruction-following, then long-thinking via RL), with the long-thinking variant demonstrating that inference-time reasoning budget can be scaled to improve performance on hard problems.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, each with a distinct responsibility:

MoonViT Vision Encoder — processes images at their native resolution (without cropping or resizing) by flattening patches into a 1D sequence, combining SigLIP's absolute positional embeddings with 2D rotary positional embeddings to preserve fine-grained spatial information. Output: continuous image feature vectors.
MLP Projector — compresses the spatial dimensions of MoonViT's output (2×2 pixel shuffle downsampling) and projects features into the dimensionality of the language model's embedding space. Acts as the bridge between vision and language modalities.
MoE Language Decoder (Moonlight) — a 16B total parameter transformer where only 2.8B parameters are activated per token (shared experts plus a subset of routed experts). Processes the interleaved sequence of text tokens and projected visual features to generate autoregressive text output, including chain-of-thought reasoning traces.
Training Pipeline — a four-stage pre-training sequence (ViT training, joint pre-training, cooldown, long-context activation) followed by post-training stages (joint SFT, long-CoT SFT, RL) that progressively build capabilities while preserving earlier ones.
Inference-Time Reasoning Mechanism (Kimi-VL-Thinking only) — at test time, the model generates extended chain-of-thought traces (up to 16K tokens), with performance scaling as a function of the thinking token budget. The RL training objective includes a length penalty to prevent excessive verbosity.

Information flows as follows: visual input (image, video frame, or screenshot) enters MoonViT → patches flattened and packed into 1D sequence with positional encodings → MLP projector compresses and projects → projected visual features are interleaved with text tokens in the language model's input sequence → MoE decoder processes the sequence, activating only relevant experts per token → autoregressive text output generated, which for the thinking variant includes an internal reasoning trace before the final answer.

3.3 Roadmap for the Deep Dive

First, the RL objective and training strategies (Section 2.4), since it is the core mechanism that enables long-thinking behavior and the one the paper describes with the most mathematical specificity—understanding it provides the foundation for interpreting all the thinking model's scaling behavior.
Second, the vision encoder design (MoonViT), because native-resolution processing is the key architectural differentiator from prior work and the design choices (patch packing, dual positional embeddings, progressive resolution training) are non-obvious.
Third, the MoE language decoder and MLP projector, which together determine the model's computational efficiency and modality integration—explaining why the MoE architecture matters and how the projection layer bridges the vision-language gap.
Fourth, the pre-training stages (ViT training, joint pre-training, cooldown, long-context activation), walking through each stage's data composition, training objectives, and the design rationale for the progressive approach.
Fifth, the post-training stages (joint SFT, long-CoT SFT, RL), explaining how instruction-following and reasoning capabilities are layered on top of the pre-trained model.
Sixth, the data construction pipeline (Section 3), covering pre-training data categories, instruction data generation, and reasoning data synthesis—since the paper argues that data strategy is as important as architecture.
Seventh, the infrastructure and parallelism strategy (Section 2.5), which makes training a 16B MoE VLM feasible and explains the reported 60% throughput advantage over dense architectures.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper. The core idea is that an effective open-source VLM requires simultaneous attention to architecture (MoE for efficiency, native-resolution vision encoding for fine-grained perception), training strategy (multi-stage with careful data composition), and reasoning capability (long-CoT SFT and RL for test-time scaling), and that none of these alone is sufficient.

Reinforcement Learning Objective and Training Strategies

The reinforcement learning stage is what transforms the base instruction-tuned model into the long-thinking variant, and the paper provides a concrete mathematical formulation for how this works.

The RL objective. The paper adopts a variant of online policy mirror descent, a policy-gradient method that iteratively refines the model's policy (its distribution over reasoning chains and answers) while regularizing against the previous iteration's policy to prevent catastrophic forgetting or mode collapse. The objective for the i-th training iteration is:

$\max_{\theta} \mathbb{E}_{(x,y^*) \sim \mathcal{D}} \left[ \mathbb{E}_{(y,z) \sim \pi_\theta} [r(x, y, y^*)] - \tau \text{KL}(\pi_\theta(x) \parallel \pi_{\theta_i}(x)) \right]$

where $\theta$ represents the model parameters being optimized, $\pi_\theta$ is the current policy model that generates both reasoning traces $z$ and answers $y$ , $\pi_{\theta_i}$ is the reference policy from the previous iteration (treated as fixed during this update), $(x, y^*)$ are problem-answer pairs sampled from the training dataset $\mathcal{D}$ , $r(x, y, y^*) \in \{0, 1\}$ is a binary reward that checks whether the model's proposed answer $y$ matches the ground truth $y^*$ , and $\tau > 0$ is a hyperparameter controlling the strength of KL regularization.

What it computes: the objective maximizes two terms simultaneously. The first term is the expected reward under the current policy—the model is rewarded for generating reasoning chains that lead to correct answers. Since the reward is binary (1 for correct, 0 for incorrect), this term directly optimizes for task accuracy. The second term is a KL divergence penalty that measures how far the current policy has moved from the reference policy $\pi_{\theta_i}$ . The outer expectation over the data distribution $\mathcal{D}$ means both terms are averaged across all training problems. The parameter $\tau$ trades off between exploitation (getting high reward now) and stability (not moving too far from the previous policy too quickly).

Why this form: online policy mirror descent with KL regularization is chosen because it addresses two failure modes of naive policy gradient. First, without regularization, policy gradient can cause the model to collapse to a narrow distribution where it generates only a few memorized reasoning patterns, reducing generalization. The KL penalty keeps the policy close to the previous iteration, which acts as a trust region—the model can improve but cannot completely abandon previously learned behaviors. Second, using the previous iteration's policy as reference (rather than the base model) means the regularization adapts as the model improves, allowing sustained progress over many iterations. This is the same algorithmic family used in Kimi k1.5 (K. Team et al., 2025) and conceptually similar to the approach in OpenAI's o1 series, though the paper does not disclose specific RL hyperparameters such as the value of $\tau$ .

Training iteration mechanics. Each RL iteration proceeds as follows: a batch of problems is sampled from the dataset $\mathcal{D}$ , the current model $\pi_{\theta_i}$ generates reasoning chains and answers for each problem, rewards are computed by comparing answers to ground truth, the policy gradient is computed from the objective in Equation 1, and model parameters are updated to $\theta_{i+1}$ . The updated model then becomes the reference policy for the next iteration. This means the model is always learning from on-policy data—the trajectories used for gradient computation are generated by the current (or immediately previous) model, not from a fixed dataset, which the paper argues is essential for long-thinking because it allows the model to explore and refine its own reasoning strategies.

Length penalty to mitigate overthinking. A known problem with RL-trained reasoning models is "overthinking"—the model generates excessively long reasoning chains that consume computation without improving accuracy. The paper implements a length-based reward penalty (exact formulation not specified) that penalizes responses exceeding a certain length, incentivizing the model to be concise when additional reasoning does not actually help. This is why the Kimi-VL-Thinking-2506 variant achieves a 20% reduction in average output token length while simultaneously improving accuracy on reasoning benchmarks: the RL training with length penalty shapes the model to allocate reasoning tokens more efficiently, spending computation only when it genuinely contributes to problem-solving.

Curriculum sampling and prioritized sampling. To make RL training more efficient, the paper employs two sampling strategies beyond uniform random selection from the training dataset. Curriculum sampling uses difficulty labels (presumably derived from base model performance or human annotation) to control the distribution of problems seen during training—the model might start with easier problems where correct reasoning chains are easier to discover, then progressively incorporate harder problems as its capability improves. Prioritized sampling uses per-instance success rates to focus training on the most "pedagogically valuable" examples, which the paper describes as those where the model currently has intermediate success rates (not trivially easy where it already gets near-perfect reward, and not impossibly hard where reward is near-zero, but in the regime where additional training can shift the outcome). This is conceptually similar to the idea of "zone of proximal development" in human learning and to techniques used in prior RLHF work to improve sample efficiency. The paper does not disclose the exact difficulty labelling methodology or the specific prioritization formula.

Why RL rather than only SFT. The paper could have stopped at long-CoT SFT (described below), using a warmup dataset of good reasoning traces to teach the model to think step by step. The addition of RL is motivated by the observation that SFT alone teaches the model to imitate reasoning patterns but does not optimize for the end goal of getting correct answers. SFT trains the model to produce reasoning chains that look like the training examples, but there is no direct pressure to improve answer accuracy—the model can learn to produce eloquent but incorrect reasoning. RL, by contrast, only rewards correct final answers (the binary reward $r(x, y, y^*) \in \{0, 1\}$ depends solely on correctness, not on the quality of the reasoning trace). However, because the RL training is initialized from the SFT model that already knows how to generate structured reasoning, the policy gradient primarily refines the reasoning strategy rather than having to discover it from scratch. This two-stage approach—SFT to teach the form of reasoning, RL to optimize for its effectiveness—is a central design pattern in the paper.

Test-time scaling mechanism. The RL training enables what the paper calls "test-time scaling" because the model learns to generate variable-length reasoning chains, and at inference time, the user can control the maximum thinking token length. Figure 13 demonstrates the effect: on MathVision, accuracy rises from 18.7% at 1K thinking tokens to 36.8% at 16K tokens; on MMMU, from 49.2% to 61.7%; on MathVista, from 66.7% to 71.3% (though saturating at 4K). The model has learned that longer reasoning chains are beneficial for harder problems, and during RL training, it receives positive reward for producing correct answers after extended reasoning on difficult problems, shaping its inference-time behavior to allocate more tokens when the problem warrants it. The paper notes that MathVista saturates early because the required reasoning depth is already captured within 4K tokens—an important boundary condition showing that longer thinking does not help uniformly across all tasks.

MoonViT: Native-Resolution Vision Encoder

The vision encoder is the component that converts raw pixel inputs into feature vectors the language model can process, and the paper's design choices here are central to its strong performance on fine-grained visual tasks.

The core problem: fixed-resolution vs. native-resolution encoding. Traditional vision encoders like SigLIP-SO-400M (which MoonViT is initialized from) process images at a fixed resolution—the image is resized to a predetermined size (e.g., 384×384 pixels) before being divided into patches and fed through the transformer. This creates a fundamental tradeoff: if the original image is high resolution (e.g., a 4K screenshot with small UI elements), resizing to 384×384 destroys fine detail; if the image has an unusual aspect ratio (e.g., a very tall screenshot or a wide panoramic video frame), resizing to a square distorts the content. The alternative—splitting the image into multiple fixed-size crops, processing each independently, and then stitching the features back together—preserves some detail but breaks spatial relationships between crops and increases computational cost quadratically (each crop must go through the full encoder).

Native-resolution processing via patch packing. MoonViT adopts the approach from NaViT (Dehghani et al., 2023): instead of resizing the image, the model divides it into patches at its original resolution, flattens the 2D grid of patches into a 1D sequence, and concatenates sequences from multiple images (which may have different numbers of patches) into a single batch. This is possible because the core computations in a vision transformer—the self-attention layers and feed-forward networks—operate on 1D sequences and do not inherently require fixed-length inputs. The key insight is that the attention mechanism does not care about the 2D layout of the patches; it attends over all patches equally regardless of their original spatial positions. What makes this work in practice is that FlashAttention (Dao et al., 2022) natively supports variable-length sequences, so the computational cost scales with the actual number of patches rather than a fixed maximum. The paper reports that this enables "non-compromised training throughput for images of varying resolutions"—meaning that processing a 1000×1000 image (many patches) and a 100×100 image (few patches) in the same batch does not waste computation on padding.

Dual positional embeddings: absolute + 2D RoPE. The positional encoding is where the design becomes non-obvious. MoonViT is initialized from SigLIP-SO-400M, which uses learnable absolute positional embeddings—a fixed-size lookup table where each spatial position (row, column) in the original fixed-resolution grid has a learned embedding vector. When images are processed at higher resolutions than the original 384×384, these absolute embeddings must be interpolated to cover the larger grid. The paper notes that simple interpolation works initially but "these interpolated embeddings become increasingly inadequate as image resolution increases"—the model has never seen fine-grained positional distinctions during training, so interpolated positions are essentially out-of-distribution.

To address this, the paper incorporates 2D rotary positional embeddings (RoPE) applied independently across the height and width dimensions. For a patch at spatial position (h, w), the 2D RoPE encoding applies standard RoPE rotation to the feature vector based on h (encoding vertical position) and also based on w (encoding horizontal position). Unlike absolute embeddings, RoPE encodes relative position—the attention score between two patches depends on their relative spatial offset, not their absolute coordinates—which generalizes naturally to resolutions and aspect ratios unseen during training. The two positional embedding approaches "work together": the interpolated absolute embeddings provide a coarse spatial prior inherited from SigLIP's pretraining, while the 2D RoPE provides fine-grained relative position information that scales to arbitrary resolutions.

Progressive resolution training. The vision encoder is not trained at full resolution from the start. The paper mentions implementing "a progressive resolution sampling strategy to gradually allow larger size" during the standalone ViT training stage. This means the training images are initially limited to a moderate resolution, and the maximum resolution is increased over the course of training. This prevents the model from being overwhelmed by very long patch sequences early in training when its representations are still noisy, and it allows the positional embeddings (both absolute and RoPE) to gradually adapt to higher resolutions rather than being thrust into an extremely out-of-distribution regime. The final MoonViT in Kimi-VL-Thinking-2506 can process images up to 3.2 million pixels (4× the original limit of approximately 800K pixels from SigLIP-SO-400M's 384×384 resolution, after accounting for patch size), achieved through continued training of the vision encoder in the later model variant.

Why not use multi-crop processing like LLaVA-OneVision? The paper explicitly contrasts its approach with models that use "complex sub-image splitting and splicing operations." In multi-crop approaches, a high-resolution image is divided into overlapping or non-overlapping crops, each crop is processed independently by a fixed-resolution encoder, and the resulting features are somehow fused (e.g., concatenated or pooled). The problems with this approach are: first, it multiplicatively increases the number of vision tokens (each crop produces a full set of features, so 9 crops = 9× more tokens), which directly increases the language model's computation quadratically (attention scales as O(n²) in sequence length); second, each crop is processed in isolation, so the model cannot attend between visual features in different crops during the encoding stage, potentially missing cross-region relationships (e.g., an object spanning two crops). The native-resolution approach avoids both problems: there is only one set of vision tokens per image (proportional to the actual pixel count, not the number of crops), and all patches attend to all other patches within the image during encoding.

ViT training objective: CoCa-style combined loss. During the standalone ViT training stage (before alignment with the LLM), MoonViT is trained with a combined objective that includes both contrastive and generative components. The specific loss function is:

$\mathcal{L} = \mathcal{L}_{\text{siglip}} + \lambda \mathcal{L}_{\text{caption}}$

where $\mathcal{L}_{\text{siglip}}$ is the SigLIP contrastive loss (a variant of the standard contrastive loss that uses sigmoid rather than softmax over pairs), $\mathcal{L}_{\text{caption}}$ is a cross-entropy loss for generating image captions conditioned on the image features, and $\lambda = 2$ is a weighting hyperparameter.

What it computes: the SigLIP term encourages the image encoder and text encoder to produce similar representations for matched image-text pairs and dissimilar representations for unmatched pairs—this is the standard contrastive pre-training objective that teaches the model to align visual and textual semantics. The caption term goes further: it feeds the image features into a text decoder and trains it to autoregressively generate the corresponding caption, which teaches the model to extract more detailed information from the image (enough to produce a descriptive sentence) rather than just matching global semantics. The weighting $\lambda = 2$ means the caption loss contributes twice as much as the contrastive loss to the total gradient, prioritizing the generative objective.

Why this form: the CoCa approach (Yu et al., 2022) is chosen because it produces vision encoders with stronger fine-grained understanding than contrastive-only training. Contrastive training teaches the model to recognize whether an image and text match but does not require extracting specific details (e.g., "a cat on a chair" and "a dog on a chair" might both be close to an image of a cat on a chair because they share many features). The caption loss forces the model to encode sufficient detail to reproduce the exact caption, which requires discriminating between "cat" and "dog," "on" and "under," etc. The paper reports an interesting emergent behavior: "during training, we observed an emergence in the caption loss while scaling up OCR data, indicating that the text decoder had developed some OCR capabilities"—the captioning objective, when trained on sufficient data that includes text in images, causes the vision encoder to learn to read.

Initialization and alignment. MoonViT is initialized from SigLIP-SO-400M (Zhai et al., 2023), a pre-trained 400M-parameter vision transformer. The text decoder used in the CoCa-style training is "a tiny decoder-only language model" (size unspecified). After training the ViT with 2T tokens of image-text data using the combined objective, an alignment stage with 0.1T tokens is performed where only MoonViT and the MLP projector are updated (the language model is frozen). The paper states that "this alignment stage significantly reduces the initial perplexity of MoonViT embeddings in the language model, allowing a smoother joint pre-training stage." What this means operationally: before alignment, feeding MoonViT features through the projector into the language model produces high perplexity—the language model finds these features "surprising" because they come from a distribution very different from the text embeddings it was trained on. The alignment stage adjusts the projector and fine-tunes MoonViT to produce features that are more compatible with the language model's expected input distribution, so when joint training begins and the language model starts being updated on these features, the gradient updates are smaller and less disruptive to the language model's pre-existing knowledge.

MLP Projector and MoE Language Decoder

The MLP projector and MoE language decoder together form the bridge between vision and language and the computational core of the model, respectively.

Pixel shuffle compression. Before projection, the visual features from MoonViT undergo a pixel shuffle operation that performs 2×2 spatial downsampling. In a standard pixel shuffle (also known as sub-pixel convolution or depth-to-space rearrangement), the spatial dimensions are reduced by a factor of r in each direction, and the channel dimension is correspondingly expanded by a factor of r². Here, with 2×2 downsampling (r = 2), the height and width of the feature map are each halved, reducing the total number of visual tokens by a factor of 4, while the channel dimension is quadrupled. This is critical for efficiency: vision transformers typically produce one feature vector per image patch, and high-resolution images produce many patches (e.g., a 1024×1024 image with 16×16 patches yields 4096 tokens—a very long sequence for the language model). The 4× compression reduces the vision token count substantially, making it feasible to process high-resolution images without the language model's attention cost becoming prohibitive.

Two-layer MLP projection. The pixel-shuffled features are then fed through a two-layer MLP (multi-layer perceptron, i.e., two fully-connected layers with a nonlinearity between them) that projects them from MoonViT's feature dimension to the language model's embedding dimension. The paper does not specify the hidden dimension or activation function of this MLP, but the standard approach (two linear transformations with a GELU or similar activation in between) maps the vision features into the same vector space as the language model's token embeddings, so that the language model can treat visual features as a special kind of "token" and interleave them with text tokens in its input sequence.

Why a simple MLP rather than a more complex connector? Some VLM architectures use cross-attention (where the language model attends to visual features through separate key-value projections) or Q-former modules (where a learned set of queries extracts relevant information from visual features). The paper's choice of a simple MLP is motivated by simplicity and training stability: a two-layer MLP with pixel shuffle compression adds very few parameters and can be trained efficiently, and the joint training stages (where both the projector and language model are updated) allow the language model to adapt its representations to the visual features, making a more complex connector unnecessary. The pixel shuffle compression is the essential ingredient that makes this simple approach work: without it, the raw visual token count would be too high for efficient processing.

Moonlight MoE language decoder. The language model is Moonlight (J. Liu et al., 2025a), a 16B total parameter MoE transformer with an architecture similar to DeepSeek-V3 (DeepSeek-AI, A. Liu, et al., 2025). The key architectural feature is that not all 16B parameters are used for every token. Instead, the feed-forward network (FFN) layers in the transformer are replaced with MoE layers consisting of:

Shared experts: a small number of FFN sub-networks that process every token, providing a baseline of common knowledge and capabilities.
Routed experts: a larger number of FFN sub-networks (e.g., 64 or 128 experts in total), where for each token, only a small subset (e.g., 2–4) is activated based on a learned routing function.
Router: a lightweight gating mechanism that takes each token's hidden state as input and outputs a probability distribution over experts, selecting the top-K experts for that token.

The total parameters across all experts sum to 16B, but because only the shared experts and 2–4 routed experts are active per token, the effective computational cost per token is equivalent to approximately 2.8B parameters. This is the fundamental efficiency advantage of MoE: the model has the representational capacity of 16B parameters (diverse knowledge stored across different experts) but the inference cost of a much smaller model. The paper notes that the training throughput of this MoE VLM is "around 60% higher than a 7B dense VLM" (Section 2.5), which directly translates to faster experimentation and lower training cost.

Why MoE for a VLM specifically? The paper argues that MoE is particularly well-suited to multimodal models because different types of inputs (text vs. images) and different types of tasks (math reasoning vs. visual perception vs. OCR) can potentially be handled by different experts, allowing the model to develop specialized sub-networks without interference. However, the paper does not provide ablation studies demonstrating this specialization empirically—it states the efficiency argument (fewer activated FLOPs per token) and relies on the broader MoE literature for evidence of effective specialization. The router is trained end-to-end with the rest of the model through standard backpropagation, so any specialization emerges from the data distribution rather than being explicitly programmed.

Initialization and context length progression. The language model is initialized from "an intermediate checkpoint in Moonlight's pre-training stage—one that has processed 5.2T tokens of pure text data and activated an 8192-token (8K) context length." This means the model had already undergone extensive text-only training before any multimodal data was introduced, giving it strong language capabilities as a foundation. The 8K context length at initialization is significant because the final model extends this to 128K—a 16× increase performed during the joint long-context activation stage. The paper uses standard RoPE base frequency extension (from 50,000 to 800,000) to achieve this, a technique where the rotary position embeddings are rescaled to cover longer sequences without retraining positional representations from scratch.

Pre-Training Stages

The pre-training pipeline consists of four sequential stages, each with specific data composition, training objectives, and capabilities being developed.

Stage 1: ViT Training (2T + 0.1T tokens). This stage trains MoonViT as a standalone vision encoder before integrating it with the language model. The 2T token portion uses the CoCa-style combined objective ( $\mathcal{L} = \mathcal{L}_{\text{siglip}} + 2\mathcal{L}_{\text{caption}}$ ) with a tiny language decoder, training on image-text pairs where text targets include "image alt texts, synthetic captions, grounding bboxes, and OCR texts." The image and text encoders compute the SigLIP contrastive loss (matching images to their text descriptions), while the text decoder performs next-token prediction conditioned on image features from the encoder (generating captions, OCR text, or bounding box coordinates). Both encoders are initialized from SigLIP-SO-400M weights. The 0.1T alignment portion updates only MoonViT and the MLP projector (the LLM is frozen) to make visual features compatible with the language model.

Stage 2: Joint Pre-training (1.4T tokens). This is where the language model first learns to process visual information. The training combines pure text data (sampled from the same distribution used for Moonlight's original text training) with multimodal data. The key design choice is the progressive multimodal ratio: "the initial steps utilize solely language data, after which the proportion of multimodal data gradually increases." This is critical because introducing visual features too abruptly could disrupt the language model's pre-existing representations—the gradual ramp-up gives the model time to adapt its internal feature spaces to accommodate the new modality without catastrophically forgetting language capabilities. The paper states that multimodal data reaches up to 40% of the training mixture but does not specify the exact schedule. The language model's original learning rate scheduler is resumed from the loaded checkpoint, meaning the training continues as if it were a single continuous run rather than starting from a new schedule. The total consumption for this stage is 1.4T tokens.

Stage 3: Joint Cooldown (0.6T tokens). After the main pre-training, a cooldown phase continues training on higher-quality data to "ensure superior performance." The data strategy is more sophisticated than simple filtering: for text data, the paper uses synthetic question-answer (QA) pairs generated from existing math, knowledge, and code corpora by "a proprietary language model, implementing rejection sampling techniques to maintain quality standards." These synthesized QA pairs are validated before inclusion. For multimodal data, the paper additionally filters and rewrites "academic visual or vision-language data sources to QA pairs," creating question-answer formats from sources like research papers, textbooks, and educational diagrams. The paper explicitly notes that these QA pairs are "kept at a low portion to avoid overfitting these QA patterns"—if too much of the cooldown data is in QA format, the model might learn to generate QA-style responses rather than developing general understanding. This stage also includes a "re-warmup to higher LR" (learning rate), meaning the learning rate, which had decayed during joint pre-training, is briefly increased before decaying again during cooldown. The total consumption is 0.6T tokens.

Stage 4: Joint Long-Context Activation (0.3T tokens). The final pre-training stage extends the context window from 8K to 128K tokens. This is done in two sub-stages, each extending the context length by 4× (8K → 32K → 128K). The RoPE base frequency is reset from 50,000 to 800,000, which rescales the rotary position embeddings to cover the longer sequence lengths without changing the model architecture. The data composition is critical: in each sub-stage, 25% of tokens come from long data (long text, long interleaved documents, long videos, long multi-page documents) while 75% comes from replaying shorter data from previous stages. This 25/75 split is the result of empirical exploration that "confirmed this composition allows the model to effectively learn long-context understanding while maintaining short-context ability." The intuition is that if too much of the data is long-context, the model might overfit to the long-context format and degrade on standard-length inputs; the 75% replay acts as a regularizer preserving existing capabilities. A small portion of QA pairs is also synthesized to "augment the learning efficiency of long-context activation," similar to the cooldown stage.

Verification of long-context capability. The paper evaluates the long-context model using needle-in-a-haystack (NIAH) tests, where a specific fact ("needle") is inserted at a random position within a long sequence of irrelevant content ("haystack"), and the model must retrieve it when asked. Table 2 reports recall accuracy for both text and video haystacks across context length ranges. For haystacks up to 65,536 tokens (64K), recall is 100.0% for both text and video. At the 65,536–131,072 token range (64K–128K), recall drops to 87.0% for text and 91.7% for video. The video NIAH result is particularly notable because it demonstrates that the model can localize specific visual information within a very long video sequence—a substantially harder task than text NIAH because visual features are higher-dimensional and less discretely separable than text tokens.

Post-Training Stages

After pre-training, the model undergoes post-training to develop instruction-following, dialogue, and reasoning capabilities.

Stage 5: Joint Supervised Fine-Tuning (SFT). This stage fine-tunes the base model on instruction-response pairs to create the interactive Kimi-VL model. Training uses the ChatML format (OpenAI, 2024), which structures conversations with explicit role tags (system, user, assistant) that allow the model to distinguish between instructions and responses. The loss is applied only to "answers and special tokens, with system and user prompts being masked" — meaning the model is trained to generate the assistant's response but not to reproduce the user's query or system prompt. The data mixture includes both pure-text dialogue data (from Moonlight's training) and multimodal instruction data. This ensures the model maintains general conversational ability while learning to handle visual inputs.

The training proceeds in two sub-stages, each for 1 epoch:

32K context stage: learning rate decays from $2 \times 10^{-5}$ to $2 \times 10^{-6}$ . This teaches instruction-following at a moderate context length.
128K context stage: learning rate re-warmups to $1 \times 10^{-5}$ and decays to $1 \times 10^{-6}$ . This extends instruction-following to the full 128K context window.

The re-warmup strategy—raising the learning rate at the start of the 128K stage rather than continuing the decay—is designed to allow the model to adapt to the longer context format with sufficient gradient magnitude. The paper uses example packing (concatenating multiple training examples into each sequence) to improve training efficiency, a standard technique for SFT that reduces padding waste when examples have varying lengths.

Stage 6: Long-CoT Supervised Fine-Tuning (Warmup). Before RL training, the model is fine-tuned on a "small yet high-quality long-CoT warmup dataset" designed to teach it the structure of extended reasoning. The dataset construction uses prompt engineering: a powerful long-CoT model (Kimi k1.5) is prompted with "curated reasoning prompts" to generate multiple detailed reasoning trajectories for each question, and correct trajectories are identified through verification (likely comparing the final answer to ground truth). The resulting dataset encodes four cognitive processes:

Planning: the model systematically outlines steps before execution, showing how to decompose a problem.
Evaluation: critical assessment of intermediate steps, teaching the model to check its own work.
Reflection: reconsidering and refining the approach when an initial attempt is flawed, demonstrating backtracking.
Exploration: considering alternative solutions, showing that multiple paths can lead to the correct answer.

The SFT on this warmup dataset "primes the model to internalize these multimodal reasoning strategies" — it learns to produce reasoning traces that exhibit planning, self-checking, and revision, which provides a strong initialization for the RL stage. Without this warmup, the RL agent would need to discover these reasoning patterns from scratch, which would be very sample-inefficient because the reward signal (binary correctness) provides no guidance on reasoning quality, only on the final answer.

Stage 7: Reinforcement Learning (described in detail above). This stage takes the long-CoT SFT model and optimizes it for answer correctness using the policy mirror descent objective, with the length penalty and sampling strategies described earlier. The output is Kimi-VL-Thinking, the long-thinking variant.

Module updatability during post-training. The paper specifies that during SFT, "we optimize the language model, MLP projector, and vision encoder" — all three components are trainable. This is notable because some VLM approaches freeze the vision encoder during instruction tuning to avoid disrupting visual representations. The paper's approach of jointly updating all components suggests that the SFT data is carefully curated to avoid degrading visual perception while teaching instruction-following, though the paper does not provide ablations on this design choice.

Data Construction Pipeline

The paper devotes substantial detail to describing the data used for pre-training and instruction tuning, organized by modality and purpose.

Pre-training data categories (Section 3.1). The multimodal pre-training corpus is organized into six categories, each serving a specific purpose:

Caption data: provides "fundamental modality alignment and a broad range of world knowledge." Sources include open-source datasets (LAION-5B from Schuhmann et al., 2022; DataComp from Gadre et al., 2024) and in-house data. The paper strictly limits the proportion of synthetic captions "to mitigate the risk of hallucination stemming from insufficient real-world knowledge" — synthetic captions can describe what's in an image incorrectly, teaching the model false associations. Image resolution is varied during pre-training to ensure the vision tower works at both high and low resolution.
Image-text interleaving data: documents where images appear within text (like webpages with illustrations or textbooks with figures). The paper argues this data type is important for multiple reasons: it develops multi-image comprehension, provides detailed knowledge context for images, builds longer multimodal context understanding, and even "contributes positively to maintaining the model's language abilities." Sources include open interleaving datasets (OBELICS from Laurençon et al., 2024; Multimodal-C4 from Zhu et al., 2024) and in-house data from textbooks, webpages, and tutorials. A "data reordering procedure" ensures images and text are in the correct sequential order—a non-trivial quality control step for web-scraped interleaving data where images and their captions might be misaligned.
OCR data: enables the model to read text from images. Sources include open-source and in-house datasets covering "multilingual text, dense text layouts, web-based content, and handwritten samples." The paper also incorporates non-text optical recognition following OCR 2.0 principles (Wei et al., 2024): figures, tables, geometry diagrams, mermaid plots, and natural scene text. Data augmentation—rotation, distortion, color adjustments, noise addition—is applied to improve robustness. Multi-page OCR data is specifically collected "to activate the model's understanding of long documents," enabling comprehension of entire academic papers or scanned books rather than just single-page documents.
Knowledge data: analogous to text pre-training data but in multimodal form—textbooks, research papers, academic materials processed through both a layout parser and OCR model. A careful taxonomy balances content across categories. The paper notes a specific challenge: infographics can cause the model to "focus solely on OCR-based information," so an "additional pipeline" captures the purely textual information embedded within images to prevent the model from treating infographics as just text-extraction tasks.
Agent data: screenshots paired with actions for GUI interaction tasks. The data is collected on virtual machine environments where heuristic methods capture screenshots and corresponding actions. Action spaces are categorized by environment (Desktop, Mobile, Web). Icon data teaches the model the meaning of software GUI icons. Multi-step trajectories collected from human annotators include synthesized Chain-of-Thought reasoning (following Aguvis methodology from Xu et al., 2024) that demonstrates the planning process behind each action. These demonstrations cover both Ubuntu and Windows environments, giving the model cross-platform agent capabilities.
Video data: included during pre-training, cooldown, and long-context stages to develop two abilities: understanding long sequences dominated by images (hour-long videos) and perceiving fine-grained spatio-temporal correspondence in short clips. Sources span open-source datasets and in-house web-scale data, covering varying durations and diverse scenes. For long videos, a pipeline produces "dense captions" (detailed descriptions at many time points), but the proportion of synthetic dense descriptions is "strictly limited" to reduce hallucination risk—the same principle as for caption data.

Text data strategy. The text component of pre-training uses the same data as Moonlight, covering English, Chinese, Code, Mathematics & Reasoning, and Knowledge domains. Each data source undergoes "rigorous individual validation" to assess its contribution to training, with high-value subsets upsampled. The paper emphasizes that the sampling strategy is "empirically determined through extensive experimentation" and that "isolated evaluations" identify which subsets contribute most to knowledge acquisition.

Instruction data construction (Section 3.2). For non-reasoning tasks (chart interpretation, agent grounding, OCR, image-grounded conversation, QA, writing, text processing), the data pipeline follows a generate-rank-refine pattern: human annotators create a seed dataset to train a seed model, the seed model generates multiple responses to diverse prompts, annotators rank these responses, and the top-ranked response is refined to produce the final version. For reasoning tasks (visual coding, visual reasoning, math/science problems), the paper uses rejection sampling: the model generates multiple responses, and those with correct final answers (verified through rule-based or model-based checks) are kept. The complete SFT dataset comprises "approximately a 1:1 ratio of text tokens to image tokens."

Reasoning data construction (Section 3.3). For the long-CoT SFT and RL stages, data is generated by prompting Kimi k1.5 (K. Team et al., 2025) with "curated reasoning prompts" to generate multiple detailed reasoning trajectories per question. A reward model (probably also proprietary) judges correctness, and wrong trajectories are filtered out based on both model evaluation and rule-based rewards. The paper notes that this "resembles rejection sampling but focuses on generating long-CoT reasoning paths through prompt engineering" — the key difference being that prompt engineering shapes the style and structure of the reasoning (encouraging planning, evaluation, reflection, exploration) while rejection sampling filters for correctness.

Infrastructure and Parallelism Strategy

The paper's training infrastructure is designed to handle the unique challenges of training a large MoE VLM with variable-length visual inputs and long sequences.

Data loading system. The system stores visual data in original format (not pre-processed), with on-the-fly shuffling, mixing, tokenization, loss masking, and packing during training. This supports adjusting data proportions without re-processing the corpus. A key feature is "random augmentation of both visual and text data, while preserving the correctness of 2D coordinate and orientation information during transformations" — for example, if an image with bounding box annotations is rotated for augmentation, the bounding box coordinates must be correspondingly transformed. The system also ensures strict reproducibility: "random states and other states across different data loader workers" are controlled so that interrupted training can be resumed with identical data sequences. Multiple caching strategies maintain high throughput when reading from S3-compatible object storage.

4D parallelism. Training uses four parallel strategies simultaneously:

Data Parallelism (DP): replicates the model across devices, each processing different micro-batches, enabling larger effective batch sizes.
Expert Parallelism (EP): distributes expert modules in MoE layers across devices. When combined with DP, experts on a given device can handle tokens from different DP groups, improving computational efficiency by overlapping computation.
Pipeline Parallelism (PP): splits the model into layer-based stages. The paper describes specific placement: the Vision Tower and several decoder layers in the first stage, the output layer and additional decoder layers in the last stage, and remaining decoder layers evenly across intermediate stages "based on their time overhead" — this asymmetric placement minimizes pipeline bubbles where one stage waits for another.
Context Parallelism (CP): splits long sequences across devices for attention computation, reducing peak memory. This is essential for the 128K context length, where the attention matrix would not fit in a single device's memory.

Memory optimization. ZeRO-1 reduces optimizer state overhead through distributed optimizer state without extra communication costs. Selective Checkpointing Activation recomputes only layers "that have low time overhead but high memory consumption," trading some additional computation for substantial memory savings. For extremely long sequences, recomputation is expanded to additional layers to prevent out-of-memory errors.

Training throughput comparison. The paper reports that "the resulting training throughput of our model is around 60% higher than a 7B dense VLM." This throughput advantage comes from the combination of MoE architecture (fewer FLOPs per token) and the absence of multi-crop processing (which would generate proportionally more vision tokens for the language model to process). The 60% figure is a concrete efficiency metric that matters for research iteration speed and training cost.

Summary of Design Choices and Their Justifications

MoE over dense: fewer activated parameters per token (2.8B vs. 7–12B) reduces inference and training FLOPs while maintaining representational capacity through total parameters (16B).
Native-resolution vision encoding over fixed-size with multi-crop: avoids increasing vision token count multiplicatively, preserves spatial relationships across the full image, and scales naturally to varying resolutions through patch packing plus 2D RoPE.
Dual positional embeddings (absolute + 2D RoPE): absolute embeddings provide a coarse spatial prior from SigLIP initialization, while 2D RoPE provides relative position information that generalizes to unseen resolutions.
CoCa-style ViT training over contrastive-only: the captioning loss forces the encoder to extract fine-grained details, which is essential for OCR and detailed visual understanding.
Progressive multimodal ratio in joint pre-training over abrupt introduction: prevents catastrophic disruption of language capabilities by gradually introducing visual features.
Cooldown with synthetic QA pairs over uniform data: synthetic QA pairs activate specific capabilities (math, knowledge, code) more efficiently than random web data, but are kept at low proportion to avoid format overfitting.
25/75 long-to-short data split for context extension: empirically found to balance long-context learning with preservation of short-context performance.
Two-stage thinking development (long-CoT SFT warmup then RL): SFT teaches the form of reasoning (planning, evaluation, reflection); RL optimizes for correctness via the binary reward, which provides no intermediate guidance on reasoning quality.
Online policy mirror descent with KL regularization over standard policy gradient: prevents policy collapse and enables sustained improvement over many iterations.
Length penalty in RL reward over pure accuracy reward: mitigates overthinking, resulting in 20% token reduction while maintaining or improving accuracy in the 2506 variant.
All components trainable during SFT over freezing the vision encoder: allows the vision representations to adapt to instruction-following tasks, but requires careful data curation to avoid degrading visual perception.

4. Key Insights and Innovations

Innovation 1: Test-Time Compute Scaling for Multimodal Reasoning via Controllable Thinking Token Budgets

The most conceptually distinctive contribution of this paper is not the architecture or the training recipe in isolation, but the demonstration that multimodal reasoning performance can be systematically scaled at test time by adjusting the token budget allocated to chain-of-thought generation—and that this scaling behaves differently across task types, with some benchmarks saturating early while others continue improving with longer reasoning chains. This operationalizes an idea that had been demonstrated in the text-only domain (through models like OpenAI o1 and DeepSeek-R1) but had not been shown for multimodal inputs in an open-source model: that the model's "thinking time" is a tunable inference parameter, and increasing it produces monotonic accuracy gains on hard reasoning tasks without architectural changes.

Before this work, the dominant paradigm for VLMs was fixed-computation inference: a model processes the input and produces an answer in a single forward pass, with the amount of computation determined entirely by the input length and model size. The idea of spending variable compute at inference time—letting the model "think harder" on difficult problems—existed in the proprietary domain (OpenAI o1's "reasoning tokens," Kimi k1.5's long-CoT) but had not been demonstrated as a controllable, empirically characterized mechanism in an open-source VLM. The paper's Figure 13 provides the key evidence: accuracy on MathVision rises steadily from 18.7% at 1K thinking tokens to 36.8% at 16K tokens, and MMMU improves from 49.2% to 61.7% across the same range. These are not marginal gains—they represent the model nearly doubling its accuracy on MathVision through additional inference computation alone.

What makes this a conceptual advance rather than merely an empirical observation is the asymmetric scaling behavior the paper documents. MathVista saturates at 4K thinking tokens (70.9%), with no further improvement to 16K, while MathVision and MMMU continue improving across the full range. This reveals that the relationship between reasoning depth and task difficulty is not uniform: some tasks require only a modest amount of deliberation, and additional computation yields no return. The paper does not provide a predictive theory for which tasks will saturate when, but the very fact that this asymmetry is documented—and that the model can be deployed with a tunable thinking budget—changes how practitioners should think about inference-time resource allocation. Rather than treating the model as a fixed-cost function, one can dial the thinking budget up or down based on estimated task difficulty, analogous to how the compute-optimal scaling framework (in the earlier reference paper) allocates test-time compute across search strategies by difficulty.

The significance of this finding extends beyond raw performance. It establishes that the text-only finding that "longer reasoning helps" transfers to the multimodal domain, but with domain-specific saturation characteristics that matter for deployment. A practitioner deploying Kimi-VL-Thinking for a math-intensive application can set a high thinking budget knowing it will help; a practitioner using it for chart interpretation or general VQA can use a lower budget to reduce latency and cost without sacrificing accuracy. This is a fundamental shift in how to think about VLM inference—from a fixed-capability model to one whose effective capability is a function of the compute budget the user chooses to allocate.

Innovation 2: Native-Resolution Multimodal Processing as a Unified Alternative to Multi-Crop and Fixed-Resolution Paradigms

The paper's architectural contribution centers on MoonViT—a vision encoder that processes images at their native resolution and aspect ratio without cropping, resizing, or splitting into sub-images—but the intellectual contribution goes deeper than the mechanism itself. What the paper demonstrates, through systematic comparison with models that use alternative approaches, is that native-resolution encoding with dual positional embeddings (absolute + 2D RoPE) can simultaneously improve fine-grained visual perception, reduce computational cost, and eliminate the engineering complexity of multi-crop pipelines—solving three problems that prior work had treated as a tradeoff rather than a unified design challenge.

The dominant approaches before this work represented a fundamental tension. Fixed-resolution encoders (used in DeepSeek-VL2 and most VLMs built on CLIP/SigLIP backbones) forced a choice: resize high-resolution images and lose fine detail (making OCR and small-object recognition difficult), or split images into multiple crops (as in LLaVA-OneVision) and pay a quadratic cost in vision tokens while breaking spatial coherence across crop boundaries. The field had largely accepted this as an unavoidable engineering tradeoff—you either sacrificed detail or paid the computational price. Kimi-VL's design rejects this framing entirely. By combining NaViT's patch-and-pack paradigm (which allows variable-length sequences in the vision encoder) with 2D RoPE (which provides relative position information that generalizes to unseen resolutions), the model achieves a third option: process images at whatever resolution they arrive, with computational cost proportional to actual pixel count rather than to a fixed crop budget.

The evidence for why this matters is distributed across the paper's evaluation. On InfoVQA—a benchmark requiring reading small text in infographics—Kimi-VL achieves 83.2%, surpassing GPT-4o (80.7%) and substantially outperforming models with fixed-resolution encoders like Gemma-3-12B-IT (43.8%) and Llama-3.2-11B-Inst. (34.6%). On ScreenSpot-Pro—which requires localizing UI elements on 4K-resolution screenshots—Kimi-VL scores 34.5%, while GPT-4o (which presumably uses some form of resolution adaptation, though its architecture is undisclosed) scores only 0.8%. These are not marginal improvements; they represent order-of-magnitude gaps on tasks where fine-grained visual perception is the primary bottleneck.

The innovation is fundamental rather than incremental because it changes the architectural design space for VLMs. Prior to this work, a VLM designer choosing a vision encoder faced a menu of unsatisfying options: fixed resolution with detail loss, multi-crop with computational bloat, or cross-attention connectors that add complexity and parameters. Kimi-VL demonstrates that none of these compromises are necessary—that a carefully designed native-resolution encoder can match or exceed multi-crop approaches on detail-intensive tasks while maintaining the simplicity and efficiency of single-pass encoding. The key enablers (patch packing, 2D RoPE, progressive resolution training) are individually known techniques, but their integration into a working VLM that demonstrates the claimed benefits across a comprehensive benchmark suite is what makes the contribution significant.

Moreover, the paper demonstrates through Kimi-VL-Thinking-2506 that the native-resolution capability can be extended post-hoc: by continuing to train MoonViT to handle up to 3.2 million pixels per image (4× the original limit), the model achieves dramatic improvements on high-resolution perception and OS grounding (ScreenSpot-Pro: 52.8%, up from 34.5% in the instruct model; OSWorld-G: 52.5%). This shows that the native-resolution design is not a fixed ceiling but a scalable capability that can be pushed further with additional training—an important property for a research platform that others will build on.

Innovation 3: Demonstrating That an MoE Architecture Can Be the Sole Efficiency Driver in a VLM—Not a Secondary Optimization

MoE architectures had been applied to VLMs before (DeepSeek-VL2, Aria), so the mere use of MoE is not novel. What distinguishes Kimi-VL's contribution is the comprehensive demonstration that MoE efficiency does not come at the cost of capability, and in fact, that a model with only 2.8B activated parameters can match or exceed the performance of dense models activating 3–4× more parameters per token while being 60% faster to train. This reframes MoE from a "nice-to-have" efficiency optimization to a first-class architectural decision around which the entire training pipeline can be organized.

Prior MoE VLMs had shown the architecture was viable but left open questions about whether the efficiency gains came with hidden costs. DeepSeek-VL2 (4.1B activated, 28B total) achieved strong results but was limited to 4K context and a fixed-resolution encoder, making it unclear whether the architecture itself was the bottleneck or whether the surrounding design choices held it back. Aria (3.9B activated) fell short on fine-grained visual tasks. Both models could be interpreted as evidence that MoE VLMs make tradeoffs—efficiency for capability, or efficiency for context length. Kimi-VL's results directly refute this interpretation: the model achieves best-in-class or competitive performance across 24 benchmarks spanning college-level reasoning, OCR, agent tasks, long video, long documents, and multi-image understanding, all while activating fewer parameters than any competitor.

The key metric is in Table 3. Kimi-VL outperforms Qwen2.5-VL-7B (actually 8.3B parameters, all activated on every token) on 19 out of 24 benchmarks, despite Qwen activating 2.59× more parameters per forward pass. It outperforms Gemma-3-12B-IT on OCR tasks by enormous margins (InfoVQA: 83.2 vs. 43.8; OCRBench: 867 vs. 702) and substantially leads on agent tasks (ScreenSpot-Pro: 34.5 vs. unreported but implied weak) and long video understanding (Video-MME w/o subtitles: 67.8 vs. 58.2). It outperforms DeepSeek-VL2, the closest architectural comparator, on virtually every benchmark where comparison is possible. These are not cherry-picked results—they span the full range of evaluated capabilities.

The significance lies in what this implies for the VLM design space going forward. If a 2.8B-activated MoE can match a 7–12B dense model across the board, then for a fixed inference budget, the MoE architecture allows either (a) serving the same capability at lower cost, or (b) training a model with more total parameters (and thus more representational capacity) within the same activation budget. The paper's reported 60% training throughput advantage over a 7B dense VLM means that research teams with fixed GPU budgets can iterate faster or train larger total models. This is a fundamental shift in the economics of VLM development, not just an incremental efficiency tweak.

However, the paper does not provide ablation studies that isolate the MoE architecture's contribution from other design choices (native-resolution encoder, training data composition, cooldown and long-context strategies). It is possible that some of the performance advantages attributed to the MoE architecture actually stem from better training data or the careful multi-stage training pipeline. The paper's contribution is therefore a existence proof—demonstrating that MoE VLMs can work extremely well when properly designed—rather than a controlled experiment proving that MoE is causal in the gains. Future work that trains a dense version of Kimi-VL with identical data and training recipe would provide the controlled comparison needed to fully isolate the architecture's contribution.

Innovation 4: Systematically Combining Long-Thinking with General Visual Perception Without Degradation—A Negative Result Avoided

A recurring concern in the reasoning-model literature is that optimizing models for extended chain-of-thought can degrade their performance on standard, non-reasoning tasks—the model becomes so accustomed to generating long internal monologues that it either wastes tokens on simple queries or loses the ability to give direct answers. This concern has empirical grounding: prior RL-trained reasoning models in the text domain have sometimes shown regression on general benchmarks. Kimi-VL-Thinking-2506 demonstrates that this tradeoff is not inevitable: the model simultaneously improves on hard reasoning benchmarks (+20.1% on MathVision, +8.4% on MathVista, +2.1% on MMMU over the original thinking variant) while also improving on general perception and understanding (MMBench: 76.0 → 84.4; MMStar: 64.2 → 70.4; MMVet: 69.5 → 78.1) and achieving new state-of-the-art results on high-resolution perception (V*: 83.2; ScreenSpot-Pro: 52.8).

What makes this intellectually significant is not the raw numbers but what it reveals about the relationship between reasoning optimization and general capability. The naive view is that there is a tradeoff: training a model to think longer makes it worse at thinking short. The paper's result suggests that when long-thinking training is combined with continued training on the vision encoder and careful data curation, the reasoning capabilities act more like a superset than an alternative—the model learns to modulate its thinking depth based on the input, generating short responses for simple queries (averaging 180 tokens on MMBench, 1/3 of the previous thinking model) while deploying extended reasoning for hard problems.

Table 5 is the key evidence for this claim. It shows that across a diverse set of benchmarks that do not require extensive reasoning—OCR, general multimodal understanding, video comprehension, agent grounding, and long document understanding—the Kimi-VL-Thinking-2506 model matches or exceeds both the original instruct model and the earlier thinking variant. This is not automatic; the paper's earlier thinking variant (Kimi-VL-Thinking, without the "-2506" suffix) actually showed regression on some of these benchmarks (e.g., MMBench dropped from 82.9 to 76.0; RealWorldQA dropped from 68.1 to 64.0). The fact that the 2506 version recovers and exceeds the instruct model's performance while simultaneously improving reasoning metrics suggests that the earlier regression was a training artifact—likely addressable through continued vision encoder training, refined RL data, or other unspecified improvements—rather than a fundamental limitation.

This finding is significant because it removes a barrier to adoption. If long-thinking models were only useful for reasoning benchmarks and came at the cost of general capability degradation, practitioners would need to maintain two separate models: one for reasoning and one for general use. The 2506 result suggests that a single model can serve both purposes, simplifying deployment and making long-thinking a feature that can be turned on (by setting a higher thinking token budget) when needed rather than a separate model that must be swapped in.

The contribution is incremental in mechanism but fundamental in its implications for the VLM design philosophy. The specific techniques used (continued ViT training, RL refinement, presumably improved data) are not individually novel, but the demonstration that they can collectively eliminate the reasoning-generalization tradeoff changes the default assumption from "reasoning optimization probably degrades general ability" to "with proper training, reasoning optimization can be a net positive across the board." This is a negative result avoided, and such results are important for guiding the field's expectations and research priorities.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on 24 benchmarks spanning college-level reasoning, general visual understanding, multi-image perception, mathematical reasoning, OCR, OS agent grounding, long document understanding, long video comprehension, egocentric and fine-grained video perception, and multimodal reasoning with chain-of-thought (Tables 3, 4, 5). Key benchmarks include MMMU (val split, 11.5K multimodal college-level questions across six disciplines; Yue, Ni, et al. 2024), MMBench-EN-v1.1 (2974 multiple-choice questions; Yuan Liu et al. 2023), MMStar (1500 human-selected vision-indispensable samples; Lin Chen et al. 2024), MMVet (16 integrated VL capability combinations; W. Yu et al. 2024), RealWorldQA (700+ images with spatial understanding questions; x.ai, 2024), AI2D (5000+ grade school science diagrams; Kembhavi et al. 2016), BLINK (multi-image visual cognition benchmark; X. Fu et al. 2024), MathVista (math and visual reasoning; P. Lu et al. 2023), MathVision (3040 competition math problems across 16 disciplines; K. Wang et al. 2024), InfoVQA (infographic VQA; Mathew et al. 2022), OCRBench (scored out of 1000 points across five OCR tasks; Yuliang Liu et al. 2023), ScreenSpot-V2 and ScreenSpot-Pro (GUI grounding across platforms; Zhiyong Wu et al. 2024; K. Li et al. 2025), OSWorld (real computer environment interaction; T. Xie et al. 2024), WindowsAgentArena (Windows OS agent benchmark; Bonatti et al. 2024), MMLongBench-Doc (QA on up to 100+ page documents), Video-MME (900 videos, 2700 QA pairs; C. Fu et al. 2024), MLVU (1730 videos, 3102 QA pairs for long video understanding; J. Zhou et al. 2024), LongVideoBench (3763 videos, 6678 human-annotated MCQs; H. Wu et al. 2024), EgoSchema (5031 MCQs from egocentric video; Mangalam et al. 2023), VSI-Bench (5000+ QA pairs on visual-spatial video understanding; Yang et al. 2024), TOMATO (1484 QA pairs on temporal reasoning; Shangguan et al. 2025), VideoMMMU (300 expert-level videos, 900 questions; K. Hu et al. 2025), MMVU (3000 questions, 1529 videos across 27 subjects; Y. Zhao et al. 2025), and MMMU-Pro (no size specified in the paper). The paper also evaluates on the V* benchmark for high-resolution perception (no citation provided in the main text but referenced in Table 5). Appendix B provides detailed descriptions of each benchmark's composition, task structure, and evaluation protocol. For the Kimi-VL-Thinking variants, additional evaluation on MathVision (full), MathVista (mini), MMMU (val), and MMMU-Pro (average) is presented in Table 4, while Table 5 extends evaluation to general multimodal, video, and agent benchmarks.

Base model(s). The primary model is Kimi-VL-A3B, which uses Moonlight—an MoE language model with 2.8B activated parameters (16B total)—paired with a 400M-parameter MoonViT vision encoder (Section 2.1). The language model is initialized from an intermediate Moonlight checkpoint that processed 5.2T text-only tokens with an 8K context window. Two thinking variants are evaluated: Kimi-VL-Thinking (developed through long-CoT SFT and RL as described in Section 2.4) and Kimi-VL-Thinking-2506 (an updated version with continued MoonViT training to 3.2M pixels and improved reasoning efficiency, Section 4.3). The paper frames the 2.8B activated parameter count as enabling comparison with efficient VLMs—dense models like Qwen2.5-VL-7B (8.3B parameters, all activated) and Gemma-3-12B-IT (12B activated), and the MoE model DeepSeek-VL2 (4.1B activated, 28B total)—while the 16B total parameter capacity provides representational headroom.

Metrics. The primary metric is accuracy (Acc) or Pass@1, defined as the fraction of questions for which the model's single generated answer exactly matches the ground truth. For some benchmarks (InfoVQA), the paper uses the ANLS metric on the test set (standard for that benchmark). OCRBench uses a composite score out of 1000 points. For agent benchmarks, OSWorld and WindowsAgentArena use Pass@1 with execution-based evaluation (the agent's actions are executed in a real or simulated environment, and success is determined by task completion). For the thinking variants, the paper reports Pass@1 at varying maximum thinking token lengths (1K, 2K, 4K, 8K, 16K tokens; Figure 13) to characterize test-time scaling behavior. All metrics are reported as percentages or raw scores as specified by each benchmark's standard protocol.

Baselines. The paper compares against six models, reported in Table 3 (Section 4.1):

GPT-4o (OpenAI et al., 2024): Proprietary flagship multimodal model, architecture undisclosed, included as an upper-bound reference. Results shown in gray in Table 3.
GPT-4o-mini: Smaller proprietary model, included for efficient-model comparison.
Qwen2.5-VL-7B (Bai et al., 2025): Dense architecture, 8.3B total parameters (7.6B LLM + 0.7B vision tower), all activated. The paper notes this model actually has 8.3B parameters despite the "7B" designation.
Llama-3.2-11B-Inst.: Dense architecture, 11B total parameters (8B LLM + 2.6B vision tower).
Gemma-3-12B-IT (Gemma Team et al., 2025): Dense architecture, 12B total parameters (12B LLM + 0.4B vision tower).
DeepSeek-VL2 (Wu et al., 2024): MoE architecture, 4.1B activated (28B total) LLM parameters + 0.4B vision tower.

For the thinking model comparison (Table 4), additional baselines include GPT-4o, GPT-4o-mini, o1 (OpenAI's long-thinking proprietary model, ~12B activated parameters per the paper's listing), Qwen2.5-VL-72B (dense, 72B), Gemma-3-27B-IT (dense, 27B), QVQ-72B-Preview (long-thinking open-source model), Kimi-k1.5 (proprietary long-thinking model; K. Team et al., 2025), and QVQ-Max-Preview. For Table 5 (general capabilities of thinking models), baselines include GPT-4o, Qwen2.5-VL-7B, and Gemma-3-12B-IT.

Generation budget / compute accounting. For the standard Kimi-VL (non-thinking) model, inference follows standard autoregressive generation presumably with a fixed maximum output length—results in Table 3 are single-pass Pass@1 without test-time compute scaling. For the thinking variants, the paper introduces a thinking token budget as the primary compute control: the maximum number of tokens the model can use for its internal chain-of-thought before producing a final answer. Figure 13 varies this budget across {1K, 2K, 4K, 8K, 16K} tokens and reports Pass@1. This is the paper's operationalization of test-time compute scaling—longer thinking budgets allow the model to engage in more extensive reasoning, with computational cost approximately linear in token count. The paper reports average output token lengths for the thinking variants: Kimi-VL-Thinking-2506 averages 2.4K tokens on MMMU-val and 4.4K tokens on MathVision, representing ~20% reduction from the original Kimi-VL-Thinking (2.9K and 5.8K respectively; Section 4.3). On general benchmarks, the 2506 variant averages 180 tokens on MMBench, 1/3 of the previous thinking model. The paper does not report inference FLOP counts, latency measurements, or GPU memory usage for different thinking budgets—compute is measured solely in token counts, which is a reasonable proxy for total computation but does not capture potential differences in per-token cost (e.g., due to KV-cache management at long sequence lengths).

Cross-validation / statistical protocol. The paper does not describe any cross-validation or statistical significance testing. Results are single-point accuracy scores on standard benchmark test or validation sets. For agent benchmarks with execution-based evaluation (OSWorld, WindowsAgentArena), the paper does not specify the number of evaluation episodes or report variance. For the test-time scaling curves (Figure 13), the paper reports Pass@1 at discrete thinking token lengths without error bars or confidence intervals, making it difficult to assess whether the observed trends (e.g., MathVista saturation at 4K tokens) are statistically reliable or within sampling noise. For the thinking variant's test-time scaling across different benchmarks, the paper does not disclose the number of evaluation samples at each thinking token budget or whether the same subset of questions was used across all budget levels.

Main Quantitative Results

Standard Kimi-VL: Comparison Against State-of-the-Art Efficient VLMs

Headline result from Table 3. Kimi-VL-A3B, with only 2.8B activated LLM parameters, outperforms Qwen2.5-VL-7B (8.3B activated) on 19 out of 24 benchmarks, outperforms Gemma-3-12B-IT (12B activated) on most OCR and agent tasks by substantial margins, outperforms DeepSeek-VL2 (4.1B activated) on virtually every comparable benchmark, and achieves results competitive with or exceeding GPT-4o-mini while approaching GPT-4o on several metrics. The paper frames this as evidence that MoE architecture with proper training design can achieve dense-model-competitive performance with substantially fewer activated parameters.

College-level academic problems (Section 4.1.1). On MMMU (val), Kimi-VL achieves 57.0%, behind GPT-4o (69.1%), GPT-4o-mini (60.0%), Gemma-3-12B-IT (59.6%), and Qwen2.5-VL-7B (58.6%), but ahead of DeepSeek-VL2 (51.1%) and Llama-3.2-11B-Inst. (48.0%). On VideoMMMU, Kimi-VL scores 52.6%, outperforming Qwen2.5-VL-7B (47.4%) and DeepSeek-VL2 (44.4%), but trailing GPT-4o (61.2%) and Gemma-3-12B-IT (57.2%). On MMVU (val), Kimi-VL scores 52.2%, comparable to DeepSeek-VL2 (52.1%) and behind Qwen2.5-VL-7B (50.1%)—note that this is a slight underperformance. The pattern here is that Kimi-VL is competitive on college-level multimodal reasoning but not dominant, with performance roughly on par with 7-8B dense models and behind ~12B dense models on MMMU specifically.

General visual ability (Section 4.1.2). On MMBench-EN-v1.1, Kimi-VL achieves 83.1%, matching GPT-4o (83.1%) and outperforming Qwen2.5-VL-7B (82.6%), Gemma-3-12B-IT (74.6%), DeepSeek-VL2 (79.6%), and Llama-3.2-11B-Inst. (65.8%). On AI2D, Kimi-VL achieves 84.9%, surpassing all compared models including GPT-4o (84.6%). On MMVet, Kimi-VL scores 66.7%, closely matching Qwen2.5-VL-7B (67.1%) and GPT-4o-mini (66.9%). On RealWorldQA, Kimi-VL scores 68.1%, outperforming Gemma-3-12B-IT (59.1%) and comparable to Qwen2.5-VL-7B (68.5%). On MMStar, Kimi-VL scores 61.3%, behind Qwen2.5-VL-7B (63.9%) and GPT-4o (64.7%), but ahead of DeepSeek-VL2 (55.5%) and Gemma-3-12B-IT (56.1%). These results demonstrate robust general visual understanding that is competitive with dense models activating 3× more parameters.

Multi-image reasoning. On BLINK, Kimi-VL scores 57.3%, outperforming Qwen2.5-VL-7B (56.4%), GPT-4o-mini (53.6%), Gemma-3-12B-IT (50.3%), and Llama-3.2-11B-Inst. (39.8%). DeepSeek-VL2 and GPT-4o results are not reported. The paper frames this as evidence that Kimi-VL's ability to process multiple images benefits from its architecture, though no ablation isolates whether the MoE design, interleaving training data, or long-context capability drives this advantage.

Mathematical reasoning (Section 4.1.3). On MathVista, Kimi-VL achieves 68.7%, outperforming all compared models: GPT-4o (63.8%), Qwen2.5-VL-7B (68.2%), Gemma-3-12B-IT (56.1%), DeepSeek-VL2 (62.8%), Llama-3.2-11B-Inst. (47.7%), and GPT-4o-mini (52.5%). This is one of Kimi-VL's strongest results—a 2.8B-activated model leading a benchmark typically dominated by larger models. On MathVision, the pattern reverses: Kimi-VL scores 21.4%, behind Qwen2.5-VL-7B (25.1%), Gemma-3-12B-IT (32.1%), and GPT-4o (30.4%), but ahead of DeepSeek-VL2 (17.3%) and Llama-3.2-11B-Inst. (13.6%). The paper attributes the MathVision gap to "limited activated parameters" (Section 4.1.3), noting that MathVision includes competition-level problems that are substantially harder. This asymmetry—Kimi-VL leads on MathVista but trails on MathVision—is important context for understanding the thinking variant's later improvements: the base model's math reasoning is strong but has a clear ceiling on very hard problems.

OCR and document understanding (Section 4.1.4). On InfoVQA, Kimi-VL achieves 83.2%, surpassing all models: GPT-4o (80.7%), Qwen2.5-VL-7B (82.6%), DeepSeek-VL2 (78.1%), and substantially ahead of Gemma-3-12B-IT (43.8%) and Llama-3.2-11B-Inst. (34.6%). On OCRBench, Kimi-VL scores 867/1000, leading Qwen2.5-VL-7B (864), GPT-4o-mini (785), DeepSeek-VL2 (811), and far ahead of Gemma-3-12B-IT (702). These are among the largest absolute gaps in Table 3 and are the paper's strongest evidence for the native-resolution vision encoder's effectiveness—fixed-resolution encoders in dense models like Gemma-3 and Llama-3.2 struggle with fine-grained text, while Kimi-VL's native-resolution MoonViT preserves the detail needed for OCR.

Agent grounding and interaction (Section 4.1.5). On single-step grounding: ScreenSpot-V2 achieves 92.8% (Kimi-VL) vs. 86.8% (Qwen2.5-VL-7B) vs. 18.1% (GPT-4o)—but note the paper's footnote that GPT-4o results use Omniparser, which significantly degrades performance, making the comparison misleading. ScreenSpot-Pro achieves 34.5% (Kimi-VL) vs. 29.0% (Qwen2.5-VL-7B) vs. 0.8% (GPT-4o with Omniparser). On multi-step agent interaction: OSWorld achieves 8.22% (Kimi-VL) vs. 5.03% (GPT-4o) vs. 2.5% (Qwen2.5-VL-7B). WindowsAgentArena achieves 10.4% (Kimi-VL) vs. 9.4% (GPT-4o) vs. 3.4% (Qwen2.5-VL-7B). These results show Kimi-VL leading all compared models on OS agent tasks, including GPT-4o, which the paper highlights as particularly significant given GPT-4o's status as a flagship proprietary model. The ScreenSpot-Pro result is notable because it involves 4K-resolution screenshots where fine-grained visual perception is critical.

Long document and long video understanding (Section 4.1.6). On MMLongBench-Doc, Kimi-VL scores 35.1%, ahead of GPT-4o-mini (29.0%), Qwen2.5-VL-7B (29.6%), and Gemma-3-12B-IT (21.3%), but behind GPT-4o (42.8%). On Video-MME without subtitles, Kimi-VL scores 67.8%, leading Qwen2.5-VL-7B (65.1%), Gemma-3-12B-IT (58.2%), and DeepSeek-VL2 (not reported—presumably infeasible at 4K context). On Video-MME with subtitles, Kimi-VL scores 72.6%, ahead of Qwen2.5-VL-7B (71.6%) and Gemma-3-12B-IT (62.1%), behind GPT-4o (77.2%). On MLVU (MCQ subset), Kimi-VL scores 74.2%, state-of-the-art among all models including GPT-4o (64.6%) and Qwen2.5-VL-7B (70.2%). On LongVideoBench, Kimi-VL scores 64.5%, behind only GPT-4o (66.7%) and ahead of GPT-4o-mini (58.2%), Qwen2.5-VL-7B (56.0%), and Gemma-3-12B-IT (51.5%). The pattern is consistent: Kimi-VL substantially outperforms other efficient VLMs on long-form multimodal understanding and approaches GPT-4o—the one model it does not consistently match in this category.

Egocentric and fine-grained video perception (Section 4.1.7). On EgoSchema (full hidden test set), Kimi-VL achieves 78.5%, significantly outperforming GPT-4o (72.2%) and Qwen2.5-VL-7B (65.0%). On VSI-Bench, Kimi-VL scores 37.4%, ahead of GPT-4o (34.0%) and Qwen2.5-VL-7B (34.2%). On TOMATO, Kimi-VL scores 31.7%, outperforming Qwen2.5-VL-7B (27.6%) and GPT-4o-mini (28.8%) but behind GPT-4o (37.7%). These results demonstrate that Kimi-VL's video understanding extends beyond simple content recognition to nuanced temporal and spatial reasoning about egocentric perspectives and fine-grained object relationships.

Kimi-VL-Thinking: Reasoning Extension Results

Headline result from Table 4. Kimi-VL-Thinking substantially improves over the base Kimi-VL on all reasoning benchmarks: +2.6% on MathVista (68.7% → 71.3%), +4.7% on MMMU (57.0% → 61.7%), and +15.4% on MathVision (21.4% → 36.8%). The thinking variant achieves 71.3% on MathVista (outperforming GPT-4o's 63.8%), 61.7% on MMMU (outperforming GPT-4o-mini's 60.0% and Qwen2.5-VL-7B's 58.6%), and 36.8% on MathVision (exceeding GPT-4o's 30.4%, Gemma-3-27B-IT's 35.5%, and QVQ-72B-Preview's 35.9%). On MMMU-Pro, Kimi-VL-Thinking achieves 43.0%, ahead of GPT-4o-mini (37.6%) and Qwen2.5-VL-7B (38.1%), but behind GPT-4o (51.7%) and o1 (51.1%). On VideoMMMU, Kimi-VL-Thinking scores 55.5%, behind GPT-4o (61.1%), o1 (60.2%), and Gemma-3-12B-IT (61.8%), but ahead of Qwen2.5-VL-7B (47.0%).

Comparison against larger thinking models from Table 4. Despite activating only ~3B parameters, Kimi-VL-Thinking is competitive with much larger models: it exceeds QVQ-72B-Preview (a 72B dense long-thinking model) on MathVision (36.8% vs. 35.9%) and approaches its performance on MathVista (71.3% vs. 71.4%). It trails o1 (74.8% MMMU, 77.3% MMMU-Pro, 71.0% MathVista) and Kimi k1.5 (74.9% MathVista, 70.0% MMMU) by substantial margins on reasoning benchmarks—these proprietary models remain stronger—but the paper frames the comparison as demonstrating that open-source long-thinking at small scale is feasible and competitive.

Test-time scaling behavior (Figure 13). This figure presents the relationship between maximum thinking token length and Pass@1 accuracy on three benchmarks. On MathVision, accuracy rises monotonically from 18.7% at 1K tokens to 36.8% at 16K tokens, a 1.97× improvement. The curve shows no signs of saturation at 16K, suggesting further gains with longer budgets are possible. On MMMU, accuracy rises from 49.2% at 1K to 61.7% at 16K, with consistent gains at each doubling of the token budget. On MathVista, accuracy rises from 66.7% at 1K to 71.3% at 16K, but nearly all of the improvement occurs by 4K tokens (70.9%), after which the curve flattens—the model reaches 70.6% at 8K before a slight uptick to 71.3% at 16K. The paper interprets this as evidence that "the necessary reasoning depth is already captured within a relatively short context" for MathVista, while MathVision and MMMU require longer deliberation.

The practical implication: for MathVista-style tasks, a 4K token thinking budget is sufficient, and additional tokens are wasted computation. For MathVision or MMMU, scaling the thinking budget to 16K (or potentially beyond) continues to yield meaningful improvements. However, the paper does not report whether these saturation points are consistent across different problem difficulties within each benchmark—a finer-grained analysis (by difficulty bin, as in the reference paper) would strengthen the claim that saturation is a function of problem complexity rather than benchmark artifact.

Kimi-VL-Thinking-2506: Integrated Thinking Model Results

Headline result: reasoning improvements (Table 4, rightmost column). Kimi-VL-Thinking-2506 further improves over the original thinking variant: MathVision jumps from 36.8% → 56.9% (+20.1 percentage points), MathVista from 71.3% → 80.1% (+8.8 points), MMMU from 61.7% → 64.0% (+2.3 points), MMMU-Pro from 43.0% → 46.3% (+3.3 points), and VideoMMMU from 55.5% → 65.2% (+9.7 points). On MathVision, the 56.9% score places the 2506 variant above Gemma-3-27B-IT (35.5%), QVQ-72B-Preview (35.9%), and Kimi k1.5 (38.6%), though still behind Qwen2.5-VL-72B (38.1%—but note the paper's listing of "38.1" for Qwen2.5-VL-72B on MathVision appears in the "Non-Thinking Model" section of Table 4). On MathVista, 80.1% exceeds Qwen2.5-VL-72B (74.8%), GPT-4o (63.8%), and QVQ-72B-Preview (71.4%), approaching Kimi k1.5 (74.9%)—the paper claims this as state-of-the-art among open-source VLMs. On VideoMMMU, 65.2% is the highest reported score in Table 4 among open-source models and 4 points above GPT-4o (61.2%).

Headline result: token efficiency (Section 4.3 text). The 2506 variant reduces average output token length by ~20% on reasoning benchmarks compared to the original thinking variant: 2.9K → 2.4K on MMMU-val and 5.8K → 4.4K on MathVision. On general benchmarks, the improvement is more dramatic: averaging 180 tokens on MMBench, which is 1/3 of the previous thinking model's token count while simultaneously improving accuracy from 76.0% → 84.4%. This is the paper's evidence that the length penalty in RL training, combined with unspecified refinements, produces more efficient reasoning—the model learns to allocate thinking tokens where they matter and be concise otherwise.

Headline result: general capability improvements (Table 5). The 2506 variant recovers and exceeds the original Kimi-VL-A3B-Instruct model on non-reasoning benchmarks, addressing the regression observed in the original thinking variant. On MMBench-EN-v1.1: 84.4% (2506) vs. 82.9% (Instruct) vs. 76.0% (original Thinking). On MMStar: 70.4% vs. 61.7% vs. 64.2%. On MMVet: 78.1% vs. 66.7% vs. 69.5%. On RealWorldQA: 70.0% vs. 68.1% vs. 64.0%. On OCRBench: 869 vs. 864 vs. 864. On Video-MME (w/ subtitles): 71.9% vs. 72.7% vs. 66.0%—a slight regression from Instruct. On MMLongBench-Doc: 42.1% vs. 35.1% vs. 32.5%—a substantial 7-point improvement over Instruct. On ScreenSpot-Pro: 52.8% (2506, not reported for Instruct or original Thinking in Table 5). On OSWorld-G: 52.5% (not reported for original Thinking). The MMLongBench-Doc result at 42.1% is notable because it matches GPT-4o (42.8%)—the paper claims this as the first open-source model to achieve parity with GPT-4o on this benchmark.

High-resolution perception results (Table 5). The 2506 variant's continued MoonViT training to 3.2M pixels yields strong improvements on high-resolution tasks: 83.2% on V* and 52.8% on ScreenSpot-Pro. The paper does not report these benchmarks for the original thinking variant, so the improvement is relative to the Instruct model (ScreenSpot-Pro: 34.5% from Table 3). The V* result of 83.2% is presented without comparison to baseline models—the metric's context is unclear without competitor numbers.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of systematically removing or varying individual components and measuring the performance impact. There are no experiments that, for example, replace MoonViT with a fixed-resolution encoder while keeping all else equal, train a dense version of the language model with identical data, remove the cooldown stage and measure degradation, or compare different MoE configurations (number of experts, number activated per token). The paper provides, instead:

Test-time scaling across thinking token budgets (Figure 13). This serves as an ablation on the thinking mechanism itself—varying the inference-time compute budget reveals how performance scales. The key findings are that MathVision and MMMU scale monotonically with budget while MathVista saturates at 4K tokens. This is not an ablation in the classical sense (it tests a deployment parameter, not a design choice), but it characterizes the thinking model's behavior and provides evidence that the RL training successfully produced a model that can use additional computation productively—the model was not merely memorizing reasoning patterns of a fixed depth.

Comparison of thinking model vs. instruct model on non-reasoning tasks (Tables 3, 4, 5). The evolution from Kimi-VL-A3B → Kimi-VL-Thinking → Kimi-VL-Thinking-2506 across Tables 3, 4, and 5 serves as an implicit ablation on the thinking training pipeline. The original Thinking variant showed regression on general benchmarks (MMBench: 82.9% → 76.0%; RealWorldQA: 68.1% → 64.0%), demonstrating that the initial long-CoT SFT + RL training degraded non-reasoning capabilities. The 2506 variant recovered and exceeded the Instruct baseline, demonstrating that with further training (continued MoonViT training, refined RL, unspecified improvements), the regression can be reversed. This is a negative result avoided—the paper shows that thinking training does not inherently degrade general ability, but the earlier regression is acknowledged.

Needle-in-a-haystack (NIAH) evaluation across context lengths (Table 2). This serves as an ablation on the long-context activation stage. By testing recall accuracy at different haystack lengths, the paper demonstrates that the 128K context window is genuinely usable—not just a theoretical maximum that the model cannot effectively utilize. The clean 100% recall up to 64K tokens for both text and video haystacks, with only mild degradation at 64K–128K (87.0% text, 91.7% video), provides evidence that the joint long-context activation stage (with its 25/75 long-to-short data split) was effective. What is missing is a comparison against: (a) a model without the long-context activation stage, tested at 8K and evaluated for degradation at longer lengths, or (b) a model trained with a different long-to-short data ratio to validate the 25/75 choice.

Kimi-VL-Thinking-2506 vs. Kimi-VL-Thinking (Tables 4 and 5). The improvements from the original Thinking variant to the 2506 variant can be viewed as an ablation on continued training and methodological refinement. The 2506 variant incorporates continued MoonViT training (3.2M pixels vs. original limit), presumably refined RL training, and unspecified other improvements. The large jump on MathVision (36.8% → 56.9%) and the recovery of general capabilities (Table 5) demonstrate that the original Thinking model's performance was not a ceiling—further training and refinement could push it substantially higher. However, the paper does not disentangle which specific changes caused each improvement: was the MathVision jump primarily from the extended vision encoder (processing higher-resolution inputs might help read math notation), from better RL training, or from both?

RL length penalty effectiveness. The reduction in average output token length from the original Thinking model to the 2506 variant (2.9K → 2.4K on MMMU, 5.8K → 4.4K on MathVision, and a dramatic reduction to 180 tokens on MMBench) provides implicit evidence that the length penalty in the RL objective (Section 2.4) was effective at reducing overthinking. However, the paper does not report an ablation where the length penalty is removed, so the causal effect of this specific design choice cannot be isolated from other changes in the 2506 training recipe.

Progressive multimodal ratio in joint pre-training. The paper states that "the initial steps utilize solely language data, after which the proportion of multimodal data gradually increases" and that "we observe that joint pre-training preserves the model's language capabilities while successfully integrating visual comprehension abilities" (Section 2.3). No ablation is provided comparing progressive introduction to abrupt introduction of multimodal data. The claim that this progressive approach is necessary for preserving language ability is asserted based on observation but not experimentally validated through a controlled comparison.

Cooldown stage contribution. The paper describes the joint cooldown stage extensively (Section 2.3) and claims it "significantly improves both language and multimodal abilities of the model." No ablation compares performance with and without the cooldown stage, or evaluates how cooldown data composition (synthetic QA pair ratio, high-quality subset selection) affects final performance.

Critical Assessment

Claim 1: "Kimi-VL outperforms Qwen2.5-VL-7B on 19 out of 24 benchmarks despite having 2.59× fewer activated parameters." This claim is quantitatively supported by Table 3—the side-by-side comparisons verify that Kimi-VL achieves higher scores on the majority of benchmarks. However, the claim's framing requires two qualifications. First, the benchmarks where Qwen2.5-VL-7B wins include important categories: MMMU (58.6% vs. 57.0%), MMStar (63.9% vs. 61.3%), MathVision (25.1% vs. 21.4%), and MMVU (50.1% vs. 52.2%—Kimi-VL wins this one marginally). These are not random losses but concentrated in college-level reasoning and fine-grained visual perception benchmarks, suggesting the advantage is not uniform across task types. Second, the paper does not control for differences in pre-training data volume, data sources, or training compute—Qwen2.5-VL-7B may have been trained on less data or with different data mixtures, so the parameter efficiency claim cannot be cleanly attributed to architecture alone. The paper demonstrates that Kimi-VL achieves this level of performance with fewer activated parameters, but does not prove that the MoE architecture causes the efficiency—it is possible that superior training data, the multi-stage training pipeline, or the cooldown strategy contribute substantially.

Claim 2: "Kimi-VL surpasses GPT-4o in several key domains." This claim, featured in the abstract and Section 1, requires careful unpacking. Table 3 shows Kimi-VL beating GPT-4o on MMBench (83.1% tie), AI2D (84.9% vs. 84.6%), MathVista (68.7% vs. 63.8%), InfoVQA (83.2% vs. 80.7%), OCRBench (867 vs. 815), ScreenSpot-V2 (92.8% vs. 18.1%), ScreenSpot-Pro (34.5% vs. 0.8%), OSWorld (8.22% vs. 5.03%), WindowsAgentArena (10.4% vs. 9.4%), MLVU (74.2% vs. 64.6%), and EgoSchema (78.5% vs. 72.2%). However, the GPT-4o agent results are compromised: the paper's footnote states "GPT-4o and GPT-4o-mini results use Omniparser without UIA," meaning the comparison is against a GPT-4o configuration using a specific parsing tool that dramatically degrades performance—the 0.8% on ScreenSpot-Pro is not representative of GPT-4o's capability with better tooling. Excluding agent benchmarks, Kimi-VL genuinely surpasses GPT-4o on OCR (InfoVQA, OCRBench), some video understanding (MLVU, EgoSchema), and math (MathVista). The claim is "supported with agent benchmark qualification"—Kimi-VL does outperform GPT-4o in several domains where the comparison is fair, but the most dramatic leads (ScreenSpot, OSWorld) are against a crippled GPT-4o configuration, weakening the headline claim.

Claim 3: "The model achieves 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc within a 128K context window." Both numbers are directly from Table 3 and are the best among efficient VLMs (only GPT-4o exceeds on LongVideoBench at 66.7%, and only GPT-4o exceeds on MMLongBench-Doc at 42.8%). The claim is factually supported. However, the paper does not provide a baseline comparison showing what performance would be without the 128K context extension—would a version of Kimi-VL with only 8K context perform dramatically worse on these benchmarks? The NIAH results (Table 2) confirm the model can retrieve information across the full context window, but they do not isolate whether the long-context activation stage actually improved benchmark performance or merely enabled it (i.e., the benchmarks may require >8K context, so without extension they would be infeasible, but the paper does not quantify the performance gain from extending from, say, 32K to 128K).

Claim 4: "Long-thinking improves performance, with MathVision rising from 18.7% to 36.8% as thinking budget scales from 1K to 16K tokens." Figure 13 directly supports this claim for the three benchmarks shown. However, the paper only shows test-time scaling for three benchmarks (MathVision, MathVista, MMMU). The claim that long-thinking improves performance is demonstrated where tested, but the scope of demonstrated effectiveness is narrow—readers should not assume that all benchmarks in Table 4 would show similar scaling behavior. The paper's own evidence shows that scaling does not help uniformly (MathVista saturates at 4K), and it does not report scaling curves for MMMU-Pro, VideoMMMU, or any non-reasoning benchmarks. The paper also does not report whether the 1K-token data point in Figure 13 represents the model generating minimal CoT or no CoT—the baseline for what "no thinking" performance would be is unclear. The 18.7% at 1K tokens on MathVision may already include some thinking, meaning the true gap from zero-thinking to full-thinking could be larger than the curve suggests, or conversely, the 1K baseline might be close to the base model's performance (which was 21.4% from Table 3).

Claim 5: "The model sets a new standard for efficient yet capable multimodal thinking models with only 3B activated parameters." This claim is supported by Table 4's comparisons: Kimi-VL-Thinking-2506 at 56.9% on MathVision exceeds all models except Qwen2.5-VL-72B (reported at 38.1%—which is actually lower, but appears to be a different configuration) and O1 (at ~12B activated, reported at 38.1%). The 80.1% on MathVista is state-of-the-art in the comparison table. However, the claim is weakened by the narrowness of the comparator set: the paper compares against Qwen2.5-VL-72B, Gemma-3-27B-IT, and QVQ-72B-Preview as large models, but these are not all long-thinking models (Qwen2.5-VL-72B and Gemma-3-27B-IT are standard non-thinking VLMs, making the comparison somewhat apples-to-oranges). A fairer comparison would exclusively pit Kimi-VL-Thinking-2506 against other long-thinking models (QVQ-72B, O1, Kimi k1.5)—and against these, the 2506 variant trails O1 and Kimi k1.5 substantially on MMMU (64.0% vs. 74.8% vs. 70.0%) and is behind O1 on MathVista (80.1% vs. 71.0%—the paper's Table 4 lists O1 at 71.0% on MathVista, which is lower than Kimi-VL-Thinking-2506's 80.1%). The claim "new standard for efficient multimodal thinking models" holds if the emphasis is on the efficiency adjective—among thinking models, Kimi-VL-Thinking-2506 activates far fewer parameters than any competitor—but it is not the highest-performing thinking model in absolute terms.

Missing experiments that would strengthen the paper:

Ablation on MoE architecture: Train a dense version of Kimi-VL with approximately 2.8B parameters (all activated, not MoE) on identical data to isolate the architecture's contribution vs. the training recipe's contribution. Without this, the paper cannot claim that MoE specifically causes the efficiency—only that Kimi-VL overall is efficient.
Ablation on native-resolution encoding: Replace MoonViT with a fixed-resolution encoder (e.g., standard SigLIP at 384×384 with multi-crop processing) while keeping the rest of the pipeline identical, and measure the performance gap on OCR and high-resolution benchmarks. The paper attributes strong OCR performance to MoonViT but provides no controlled evidence.
Scaling study across model sizes: The paper presents one model size (2.8B activated). A scaling study training Kimi-VL at, say, 1B, 2.8B, and 7B activated parameters (all MoE) would show how performance scales with compute and whether the architectural choices (MoonViT, training stages) interact with scale. This would also address the "too limited to address highly specialized or domain-specific problems" limitation the paper acknowledges (Section 5).
Long-thinking test-time scaling on more benchmarks: Figure 13 shows three benchmarks. Extending this to MMMU-Pro, VideoMMMU, and the non-reasoning benchmarks from Table 5 would clarify the boundary conditions for when longer thinking helps. The paper claims MathVista saturates early, but is this true for all benchmarks below some difficulty threshold? Without broader scaling data, the generality of the saturation finding is unknown.
Training data ablation: The paper describes six data categories and extensive quality control but never ablates their individual contributions. How much does the agent data contribute to OSWorld performance? How much does OCR data contribute to InfoVQA? Does the interleaving data improve multi-image reasoning on BLINK, or is that from other factors?
Cooldown stage ablation: Does removing the cooldown stage (and its synthetic QA pairs) significantly degrade performance, or is it a marginal refinement? The paper claims it "significantly improves" capabilities but provides no evidence.
Long-context ratio ablation: The 25/75 long-to-short data split is described as empirically determined. An ablation showing that, say, 50/50 degrades short-context performance or that 10/90 fails to properly activate long-context would validate this choice.
Statistical significance and variance: No error bars, confidence intervals, or multiple evaluation runs are reported for any result. For benchmarks with small test sets (e.g., MMMU-Pro, whose size is not reported but may be a few hundred questions), a 2–3 percentage point difference might not be statistically significant. The test-time scaling curves (Figure 13) would benefit from showing variance across different subsets of questions or multiple evaluation runs with different random seeds.
Comparison with chain-of-thought prompting of dense models: For the thinking model comparison (Table 4), the paper does not evaluate what happens when baseline models (Qwen2.5-VL-7B, Gemma-3-12B-IT) are prompted with standard CoT (e.g., "Let's think step by step") rather than their default single-pass generation. The long-thinking advantage might partly reflect the comparison between CoT and no-CoT rather than between RL-trained long-thinking and prompted CoT.

Conditional nature of the claims. The paper's central efficiency claims—that a 2.8B-activated MoE VLM can match or exceed much larger dense models—hold most strongly for OCR tasks (InfoVQA, OCRBench), agent tasks (ScreenSpot, OSWorld), and certain video understanding tasks (MLVU, EgoSchema). They hold moderately for general visual understanding (MMBench, AI2D, MMVet) where performance is competitive but not leading. They hold least strongly for college-level academic reasoning (MMMU) where larger dense models (Gemma-3-12B-IT, GPT-4o) maintain clear leads, and for the hardest math problems (MathVision) where the base model trails several larger models. The thinking variant partially closes the MathVision gap dramatically (56.9% vs. competitors' 25–38%) but still trails the absolute best models (O1 at an unspecified MathVision score, though the paper's Table 4 lists Qwen2.5-VL-72B at 38.1% and O1 at 38.1%—both lower than 56.9%, suggesting Kimi-VL-Thinking-2506 may actually lead on this benchmark among reported results). The deployment implication: the model is most suitable for applications emphasizing OCR, agent interaction, fine-grained visual perception, and mathematical reasoning where test-time thinking can be allocated, and less suitable for college-level general knowledge reasoning where raw parameter count (or proprietary training) still provides an edge.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Unaccounted For, and Test-Time Scaling Gains Are Reported Without Amortization

The assumption or constraint. The paper's test-time scaling framework (Kimi-VL-Thinking, Section 4.2) assumes the user can freely choose a thinking token budget at inference time and that accuracy will improve monotonically with budget—at least on hard reasoning tasks. However, the paper does not provide any mechanism for deciding what budget to allocate to a given problem. Without some form of difficulty estimation, a practitioner must either (a) use a fixed budget for all problems (wasting tokens on easy problems or under-allocating to hard ones) or (b) run the model multiple times with different budgets and somehow select the best answer. The reference paper on compute-optimal test-time scaling showed that allocating a fixed budget uniformly across problems is deeply suboptimal (~4× less efficient than difficulty-conditioned allocation), yet Kimi-VL provides no difficulty estimation method and reports test-time scaling gains per-benchmark without accounting for the cost of budget selection.

The consequence. The headline gains from test-time scaling (e.g., MathVision rising from 18.7% to 36.8% as the budget increases from 1K to 16K tokens; Figure 13) represent an upper bound on achievable performance at a given budget—they show what the model can achieve if you spend that many tokens, but not what a deployable system would achieve because the system has no way to know whether a given problem needs 1K, 4K, or 16K tokens. If a practitioner sets a uniform budget of 4K tokens, they lose the gains from 8K–16K scaling on the hardest MathVision problems; if they set 16K for everything, they waste 12K tokens on every MathVista problem (where performance saturates at 4K) and 15.8K tokens on every MMBench problem (where the thinking model already solves it in ~180 tokens). The total inference cost in deployment—amortized across easy, medium, and hard problems—is therefore unknown and likely much higher than the per-benchmark scaling curves suggest.

What evidence exists in the paper. Figure 13 directly reveals the asymmetry: MathVista saturates at 4K thinking tokens with essentially no improvement from 4K to 16K, while MathVision continues improving across the full range. The paper acknowledges this saturation (Section 4.2: "for this task, the necessary reasoning depth is already captured within a relatively short context, and additional computation does not yield further improvements") but draws no implication about deployment cost. The paper also reports that on MMBench, the thinking model uses only 180 tokens on average (Section 4.3)—far below even the 1K minimum shown in Figure 13—yet the paper offers no mechanism for the model or system to know that MMBench problems are easy and can be solved with minimal thinking. The RL length penalty (Section 2.4) appears to teach the model to be concise on simple problems, but this learned conciseness is not equivalent to a system-level budget allocation mechanism—the model still produces varying-length outputs and there is no guarantee it will be optimally brief across all problem types.

Mitigation status. The paper does not address this limitation. It does not propose a difficulty estimation method, a budget allocation policy, or even an analysis of what fraction of the total inference compute is wasted on problems that do not benefit from additional thinking. The authors' suggestions for future work (Section 5) focus on scaling model size, expanding pre-training data, and refining post-training, without mentioning the budget allocation problem. This is a significant gap because the reference literature on test-time compute scaling has established that difficulty-conditioned allocation is the primary driver of practical efficiency—without it, the gains from test-time scaling are substantially diluted in deployment.

2. General Capability Regression During Initial Thinking Training, With Unclear Root Cause

The assumption or constraint. The paper's long-thinking training pipeline (long-CoT SFT followed by RL; Section 2.4) assumes that optimizing for extended chain-of-thought reasoning will not fundamentally conflict with the model's ability to handle standard, non-reasoning tasks. However, the paper's own results violate this assumption for the first thinking variant: Kimi-VL-Thinking shows clear regression on general perception benchmarks compared to the base Kimi-VL-A3B-Instruct model, with MMBench-EN-v1.1 dropping from 82.9% to 76.0% (-6.9 points), RealWorldQA dropping from 68.1% to 64.0% (-4.1 points), and Video-MME dropping from 72.7% to 66.0% (-6.7 points) (Table 5). The 2506 variant recovers and exceeds the instruct baseline, but the paper provides no causal analysis of why the regression occurred or which specific change in the 2506 training fixed it.

The consequence. The regression and subsequent recovery pattern creates two problems for downstream practitioners. First, it demonstrates that the thinking training pipeline as described (long-CoT SFT + RL) is brittle—small differences in training recipe (the unspecified "refinements" in the 2506 variant) can cause substantial performance swings on tasks that are not the direct target of thinking optimization. A practitioner attempting to replicate the thinking training on a different base model or with different data would not know which levers to adjust if they observe similar regression. Second, it undermines the paper's implicit claim that long-thinking training is a generally applicable post-training step. The fact that the initial thinking model degraded on general tasks suggests that the training pipeline as described in Section 2.4 is not sufficient for preserving general capability—the 2506 variant achieved stability through additional work that the paper does not fully specify (the paper mentions continued MoonViT training to 3.2M pixels and that the 2506 version "integrates key abilities," but the precise methodological changes beyond continued ViT training are not enumerated).

What evidence exists in the paper. Table 5 provides the direct comparison: Kimi-VL-A3B-Instruct vs. Kimi-VL-Thinking vs. Kimi-VL-Thinking-2506 across nine non-reasoning benchmarks. The regression pattern is unambiguous: six of the nine benchmarks show lower scores for the original Thinking variant compared to Instruct. The paper acknowledges this implicitly by presenting Table 5 as evidence that the 2506 variant "integrates key abilities" that were apparently lost, but it does not discuss the regression explicitly as a limitation of the thinking training methodology. The paper states that the 2506 variant "reduces the average output token length by around 20%" and "only requires in average 180 tokens per answer when solving MMBench, 1/3 compared to the previous thinking model while improving 8.4% accuracy" (Section 4.3). This framing—that the 2506 variant is more efficient—sidesteps the question of why the original thinking model was using 540 tokens on MMBench (3× more) while getting worse results. The length penalty in the RL objective (Section 2.4) should have penalized excessive token usage, yet the original thinking model apparently overthought even on simple recognition tasks. This suggests the RL training did not adequately teach the model to modulate thinking depth—a failure mode that is acknowledged only through the improvement in the 2506 variant, not analyzed directly.

Mitigation status. Partially addressed through the 2506 variant, which demonstrates that the regression can be reversed. However, the paper does not provide a recipe for avoiding the regression in the first place, does not isolate which training change(s) caused the recovery (continued ViT training? refined RL data? different RL hyperparameters? better length penalty tuning?), and does not discuss whether the instability is inherent to the two-stage SFT-then-RL approach or specific to their implementation. Section 5 acknowledges that "the reasoning capability... has yet to reach its theoretical upper bound" but does not mention the generalization stability issue. Future work directions focus on scaling up rather than on understanding and preventing capability regression during thinking training.

3. Single Benchmark Family, Single Model Scale, and Absence of Controlled Ablations Limit Generality and Causal Attribution

The assumption or constraint. The paper presents results from exactly one model configuration—Kimi-VL-A3B with 2.8B activated LLM parameters, 400M vision encoder parameters, and the specific training pipeline described in Sections 2.3–2.4—evaluated across a fixed set of benchmarks. All performance claims (efficiency vs. dense models, effectiveness of native-resolution encoding, benefits of long-thinking training) are based on this single configuration. The paper assumes that the relative performance ordering observed between Kimi-VL and its competitors would generalize to other model scales, other training data compositions, and other task domains, but provides no scaling study, no architecture ablation, and no evaluation outside the benchmark suite to support this assumption.

The consequence. The paper's central claims are fundamentally correlational rather than causal. When the paper states that Kimi-VL outperforms Qwen2.5-VL-7B on 19/24 benchmarks despite activating 2.59× fewer parameters, this demonstrates that Kimi-VL with its specific design achieves this performance level, but does not demonstrate that the MoE architecture, native-resolution encoder, or multi-stage training pipeline cause the efficiency advantage. Qwen2.5-VL-7B was trained by a different team with different data, different hyperparameters, different compute budgets, and different design priorities. The performance gap could be partially or substantially attributable to Kimi-VL having access to better pre-training data (the paper describes extensive data curation in Section 3.1, including in-house agent data, synthesized QA pairs, and carefully filtered interleaving data), more training tokens (the paper does not disclose Qwen2.5-VL-7B's training data volume), or the cooldown and long-context stages (which Qwen2.5-VL may or may not have equivalents for). None of these confounds are controlled for.

This matters for two reasons. For practitioners, the lack of controlled ablations makes it impossible to know which design choices to replicate. If a team wants to build a VLM with similar efficiency, should they invest in MoE architecture, native-resolution vision encoding, the specific data mixture, or all three? The paper provides no guidance because it never isolates the contribution of any individual component. For researchers, the causal claims are unsubstantiated. The paper's framing—that it demonstrates "an effective open-source VLM requires simultaneous attention to architecture, training strategy, and reasoning capability"—is a reasonable hypothesis but is not experimentally validated. A dense version of Kimi-VL with identical data might perform similarly; a version without the cooldown stage might show only minor degradation; a version with a fixed-resolution encoder plus multi-crop processing might match or exceed the native-resolution approach. Without these controlled comparisons, the paper demonstrates a compelling existence proof but does not advance understanding of why the design works.

What evidence exists in the paper. The absence of evidence is the evidence for this limitation. The paper contains no: architecture ablation (MoE vs. dense at matched total parameters), vision encoder ablation (MoonViT with native resolution vs. fixed resolution with multi-crop), training stage ablation (with vs. without cooldown, with vs. without joint long-context activation, different long-to-short data ratios in context extension), data ablation (contribution of each of the six pre-training data categories), or scaling study (Kimi-VL at 1B, 2.8B, 7B activated parameters). The closest thing to an ablation is the comparison between Kimi-VL-A3B-Instruct, Kimi-VL-Thinking, and Kimi-VL-Thinking-2506 (Tables 3, 4, 5), which can be interpreted as an implicit ablation on the thinking training pipeline. But even this comparison conflates multiple changes (long-CoT SFT, RL training, continued ViT training, unspecified data and algorithmic improvements in the 2506 variant), so the contribution of each component cannot be disentangled.

Additionally, all evaluation is on a fixed benchmark suite. While the 24 benchmarks span diverse capabilities (college-level reasoning, OCR, agent tasks, video, long documents), they are all academic benchmarks with clean evaluation protocols. The paper provides no evaluation on real-world deployment scenarios (latency at different thinking budgets, memory usage during long-context inference, performance on adversarial or out-of-distribution inputs, robustness to input perturbations) that would inform practical deployment decisions. The paper's qualitative examples (Figures 6–12) are impressive but cherry-picked—they demonstrate what the model can do in favorable cases, not how often it fails or what its error modes look like.

Mitigation status. Not addressed. The paper does not frame the absence of ablations as a limitation. Section 5 (Conclusion, Limitation, and Future Work) identifies three limitations: model size is "too limited to address highly specialized or domain-specific problems," reasoning capability "has yet to reach its theoretical upper bound," and long-context ability is "still insufficient for certain advanced applications." These are capability ceilings, not methodological limitations of the evaluation. The future work directions—"scaling up the model size, expanding pre-training data, and enhancing post-training algorithms"—reflect a commitment to larger-scale development rather than to controlled scientific understanding of which design choices matter. The paper releases code and models, which will enable third-party ablations, but the paper itself does not provide them.

4. Long-Context Capability Degrades at the Upper End of the 128K Window, and Performance on Real Long-Context Tasks Is Not Guaranteed

The assumption or constraint. The paper claims a 128K extended context window (Section 1, Section 2.3) and presents the model as capable of "processing diverse long inputs" including hour-long videos and 100+ page documents. This claim is supported by benchmark results (LongVideoBench: 64.5; MMLongBench-Doc: 35.1; Video-MME: 67.8 w/o subtitles; MLVU: 74.2; all from Table 3) and needle-in-a-haystack (NIAH) evaluations (Table 2). However, the NIAH results reveal that retrieval capability is not uniform across the 128K window: while recall is 100.0% for haystacks up to 65,536 tokens (64K), it drops to 87.0% for text haystacks and 91.7% for video haystacks in the 65,536–131,072 token range (64K–128K). This means the model fails to retrieve the needle in approximately 1 in 8 text queries and 1 in 12 video queries when information is placed in the last quarter of the context window.

The consequence. The degraded performance in the 64K–128K range has direct implications for tasks that require attending to information distributed across the full context window. For long document understanding (MMLongBench-Doc, which involves QA on documents up to 100+ pages), questions whose answers are located in the latter portions of a very long document will be answered less reliably than questions drawing from earlier portions. The paper's reported MMLongBench-Doc score of 35.1% (Table 3) averages across all questions but does not stratify by the position of the relevant information within the document—the score may be substantially lower for questions requiring retrieval from the 64K–128K range. Similarly, for long video understanding (LongVideoBench, Video-MME), questions about events occurring late in hour-long videos will be answered less accurately than questions about early events, creating a temporal position bias that the paper does not characterize.

This limitation is particularly consequential because the paper claims long-context understanding as one of Kimi-VL's defining capabilities (Section 1: "Kimi-VL processes long"). The 100% recall at shorter haystack lengths is excellent, but the degradation at 64K–128K means the model does not fully deliver on the "128K context window" promise—the effective window where retrieval is near-perfect is approximately 64K, with the remaining 64K exhibiting measurable reliability loss. For a practitioner deploying the model for long-document QA at scale, the 13% failure rate on text retrieval in the upper half of the window could translate to an unacceptable error rate for applications requiring high recall (e.g., legal document review, contract analysis).

What evidence exists in the paper. Table 2 provides the direct evidence: recall rates stratified by haystack length. The degradation is clear and monotonic—perfect recall at every length range up to 64K, then a discontinuous drop to 87.0% (text) and 91.7% (video). The paper does not show NIAH results at finer granularity within the 64K–128K range, so it is unclear whether the degradation is concentrated at the very end of the window (e.g., 120K–128K) or distributed across the entire upper half. The paper also does not report whether the degradation varies with needle difficulty (single fact vs. complex relationship) or with the density of the haystack (how much irrelevant content is present).

The benchmark results in Table 3 for long-context tasks do not include position-stratified analysis. For MMLongBench-Doc, LongVideoBench, and Video-MME, the paper reports aggregate scores without analyzing whether accuracy varies with the temporal or positional location of the relevant information. The Kimi-VL-Thinking-2506 variant improves MMLongBench-Doc from 35.1% to 42.1% (Table 5), which the paper frames as "matching GPT-4o," but it is unclear whether this improvement comes from better retrieval across the full window or from stronger reasoning that compensates for imperfect retrieval.

Mitigation status. The paper acknowledges this limitation implicitly by publishing the NIAH breakdown in Table 2 rather than only reporting an aggregate NIAH score, which suggests the authors are aware of the upper-window degradation. However, the paper does not discuss the degradation as a limitation in Section 5 or elsewhere, does not propose mitigation strategies (e.g., placing the most relevant information earlier in the context window if the application allows it, or using retrieval-augmented approaches that chunk long documents), and does not set expectations for practitioners about the effective reliable context length (which appears to be ~64K for near-perfect retrieval). The acknowledgment that long-context ability "is still insufficient for certain advanced applications that involve extremely long sequences" (Section 5) is vague and does not reference the specific NIAH evidence or suggest concrete reliability boundaries.

5. The RL Training Objective Rewards Only Answer Correctness, Providing No Guarantee of Faithful or Interpretable Reasoning

The assumption or constraint. The RL objective for Kimi-VL-Thinking (Section 2.4, Equation 1) uses a binary reward r(x, y, y^*) \in \{0, 1\} based solely on whether the model's final answer y matches the ground truth y^*. The reasoning trace z generated by the model is not evaluated for quality, coherence, faithfulness to the input, or logical validity—it is treated as a latent variable that is optimized only insofar as it leads to correct answers. The long-CoT SFT warmup (Stage 6) provides initial examples of "good" reasoning (planning, evaluation, reflection, exploration), but the RL stage that follows optimizes purely for correctness, which could in principle discover reasoning strategies that produce correct answers for reasons unrelated to the actual problem—including exploiting dataset biases, relying on spurious correlations, or generating reasoning traces that sound plausible but do not reflect the model's actual decision process.

The consequence. The paper provides no evaluation of whether Kimi-VL-Thinking's reasoning traces are faithful (do they accurately describe the model's computation?) as opposed to merely plausible (do they look like reasonable reasoning to a human reader?). This distinction matters for several deployment scenarios. In educational applications, a model that produces sound reasoning but arrives at an answer through a different process (or vice versa) could mislead students about problem-solving methodology. In high-stakes applications (medical, legal, financial), a reasoning trace that appears valid but masks unreliable heuristics could create false confidence. In debugging and model improvement, if the reasoning trace does not reflect the model's actual computation, attempts to improve the model by analyzing its reasoning will be misguided.

The paper's qualitative examples (Figures 6–12) showcase impressive reasoning traces—identifying Einstein's manuscripts through handwriting analysis and German terminology (Figure 6), solving geometry problems with explicit theorem application (Figure 8), navigating Chrome settings to enable "Do Not Track" (Figure 10). These examples demonstrate that the model can generate coherent reasoning, but they are cherry-picked successes. The paper provides no systematic evaluation of reasoning quality: no measurement of whether reasoning steps are factually grounded in the input, whether the reasoning is logically consistent, whether the final answer actually follows from the stated reasoning, or how often the model produces reasoning that contradicts its own earlier statements or misrepresents visual content.

This limitation is not unique to Kimi-VL—it is a known challenge in the chain-of-thought and reasoning-model literature more broadly—but the paper's exclusive reliance on answer-correctness reward makes it particularly acute. The RL training could, in principle, produce a model that generates elaborate-looking reasoning traces as a form of "reasoning steganography"—encoding the answer in the reasoning in a way that looks informative but is actually used by the model as a scratchpad for computation unrelated to the surface content of the trace. The length penalty in the RL objective mitigates this by penalizing excessively long reasoning, but a penalty for length does not distinguish between concise reasoning that is faithful and concise reasoning that is merely efficient at exploiting surface patterns.

What evidence exists in the paper. The paper provides extensive evidence that reasoning traces lead to correct answers (Table 4 shows accuracy improvements from thinking; Figure 13 shows scaling behavior). It provides qualitative evidence that reasoning traces look good in selected examples (Figures 6–12). It provides no evidence that reasoning traces are faithful. There is no evaluation of: whether the model's stated reasoning steps are consistent with its actual computation (e.g., through attention analysis or causal intervention), whether the model can correctly identify when its own reasoning contains errors, whether the reasoning is robust to perturbations in the input that should change the reasoning in predictable ways, or whether the model sometimes generates correct answers with reasoning that is factually incorrect about the input (hallucinated visual features, misidentified objects, incorrect theorem applications that happen to lead to the right answer by coincidence).

The paper also does not evaluate whether the "cognitive processes" claimed to be encoded in the warmup data (planning, evaluation, reflection, exploration; Section 2.4) actually persist through RL training. The warmup SFT teaches these patterns, but the RL stage optimizes only for answer correctness, which could cause the model to abandon these structured reasoning patterns in favor of whatever leads most efficiently to correct answers. The 2506 variant's 20% reduction in token length could reflect either more efficient structured reasoning or a shift toward less structured, more heuristic problem-solving—the paper provides no analysis to distinguish these possibilities.

Mitigation status. Not addressed. The paper treats answer correctness as a sufficient evaluation metric for reasoning quality—all thinking model results (Tables 4, Figure 13) are reported as Pass@1 accuracy on the final answer. The paper does not discuss faithfulness as a concern, does not propose any evaluation methodology for reasoning quality, and does not mention faithfulness or interpretability in the limitations section (Section 5). This is a significant omission for a model that is explicitly presented as demonstrating "long-horizon reasoning capabilities" (Section 1) and whose core contribution is enabling extended chain-of-thought generation. A practitioner deploying the thinking variant in a setting where the reasoning trace will be shown to users (e.g., an educational tool, a decision-support system) has no evidence that the displayed reasoning is reliable.

6. The Base Model Has a Hard Performance Ceiling on the Most Difficult Problems, and Test-Time Scaling Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper acknowledges that Kimi-VL's 2.8B activated parameters impose a fundamental capability ceiling: "the current model size... remains too limited to address highly specialized or domain-specific problems, or problems that are strongly dependent on language abilities, restricting Kimi-VL's ability to handle extremely complex scenarios" (Section 5). This limitation is structural—no amount of test-time scaling can enable the model to solve problems that require capabilities (knowledge, reasoning depth, linguistic precision) that exceed what the base parameters can encode. The paper's test-time scaling results (Figure 13) show that longer thinking helps on MathVision, MMMU, and MathVista, but all three benchmarks show performance at very low thinking budgets that is substantially above zero—the model already has some baseline capability that scaling amplifies.

The consequence. For the hardest problems in any domain, Kimi-VL—and by extension the thinking variants—will simply fail, regardless of how many thinking tokens are allocated. This is most directly visible in difficulty bin 5 (hardest problems) in the reference paper's framework, where no amount of test-time compute helped. Kimi-VL does not report difficulty-stratified results, so the paper provides no direct evidence of where the capability ceiling lies or how it manifests, but the acknowledgement in Section 5 makes the existence of such a ceiling explicit. The practical consequence is that Kimi-VL cannot serve as a universal reasoning engine—it is suitable for problems within its capability envelope and unsuitable for problems outside it, and a deployment system needs a mechanism (which the paper does not provide) to detect when a problem exceeds the model's capability so that it can escalate to a larger model or flag for human review rather than spending computation on a guaranteed failure.

Moreover, the test-time scaling curves (Figure 13) may give a misleading impression of unbounded improvement. While MathVision and MMMU show monotonic improvement up to 16K tokens, the curves will eventually saturate—the paper does not test beyond 16K, so the saturation point is unknown, but the existence of a parameter-imposed ceiling means the curves cannot continue improving indefinitely. A practitioner who extrapolates the MathVision trend and allocates 64K or 128K thinking tokens expecting continued gains may be wasting enormous computation for marginal or zero improvement, without the paper providing any guidance on where the practical limit lies.

This limitation also has implications for the model's performance on benchmarks that are not explicitly evaluated. The paper's benchmark suite emphasizes visual perception, OCR, agent interaction, and math reasoning—domains where the native-resolution encoder and MoE architecture provide specific advantages. Benchmarks requiring deep world knowledge, multi-hop reasoning over long text passages, or sophisticated language understanding may expose the model's parameter ceiling more acutely. The paper does not evaluate on text-only reasoning benchmarks (e.g., GSM8K, MATH without visual context) that would isolate the language model's reasoning capability from the vision components, making it difficult to assess whether the parameter ceiling mainly constrains language reasoning, visual understanding, or both.

What evidence exists in the paper. The paper's own results demonstrate the capability ceiling indirectly. On MMMU—a college-level reasoning benchmark—Kimi-VL scores 57.0% (Table 3), behind GPT-4o (69.1%), Gemma-3-12B-IT (59.6%), and Qwen2.5-VL-7B (58.6%). Even with thinking, Kimi-VL-Thinking reaches only 61.7%, and the 2506 variant reaches 64.0% (Table 4)—still substantially behind GPT-4o's non-thinking 69.1% and far behind O1's 74.8%. The thinking variants close some of the gap but do not eliminate it, consistent with a parameter-imposed ceiling on tasks requiring broad college-level knowledge and complex cross-domain reasoning.

On MathVision—the hardest math benchmark—the base model scores only 21.4% (Table 3). The thinking variant dramatically improves this to 36.8% and the 2506 variant to 56.9% (Table 4). While 56.9% is impressive relative to the base model, it still means the model fails on 43.1% of MathVision problems even with extensive thinking. The paper does not analyze what types of problems comprise this 43.1%—are they problems requiring mathematical knowledge the base model lacks? Problems with visual complexity exceeding the encoder's resolution? Problems requiring reasoning chains too long for the 16K token budget? Without this analysis, the nature of the ceiling remains opaque.

The paper's acknowledgement in Section 5 is explicit about the existence of the ceiling but vague about its nature: "too limited to address highly specialized or domain-specific problems." The paper does not characterize which domains are primarily affected, whether the ceiling is primarily in the vision encoder (limited resolution, limited capacity for fine-grained detail), the language model (limited parameters for knowledge storage and complex reasoning), or both.

Mitigation status. The paper proposes scaling up model size as the primary mitigation: "In the future, we will tackle these challenges by scaling up the model size, expanding pre-training data, and enhancing post-training algorithms" (Section 5). This is a reasonable direction—larger models have higher capability ceilings—but it is a deferred solution. The current release provides no mechanism for detecting when a problem exceeds the model's capability, no guidance on the practical upper bound of test-time scaling effectiveness, and no characterization of which problem types are most affected by the parameter ceiling. A practitioner deploying the current model must either accept that some fraction of problems will be unsolvable regardless of compute budget, or implement their own escalation logic without guidance from the paper on how to identify unsolvable problems a priori.

7. Implications and Future Directions

How This Work Changes the Landscape

Kimi-VL shifts the open-source VLM landscape from a world where architectural efficiency, long-context understanding, high-resolution perception, and long-thinking reasoning were treated as separate research threads—each demonstrated in isolation by different models, each with identifiable compromises—to one where all four capabilities can coexist in a single model activating only ~3B parameters. This is not a paradigm shift in the sense of introducing a fundamentally new technique; rather, it is a unifying existence proof that resolves a tension the field had implicitly accepted: that MoE efficiency, native-resolution vision, long multimodal context, and RL-driven chain-of-thought reasoning were somehow in tension, or at least that combining them required tradeoffs so severe as to be impractical. Kimi-VL demonstrates that this tension was an artifact of suboptimal integration, not a fundamental constraint.

The specific reframing is this: MoE architecture is not merely an efficiency optimization bolted onto a VLM—it can be the architectural backbone around which all other capabilities are organized, provided the training pipeline is designed to preserve and extend rather than replace. Prior MoE VLMs (DeepSeek-VL2, Aria) had shown MoE could work but left open whether the architecture imposed hidden costs on context length (DeepSeek-VL2 capped at 4K), fine-grained perception (Aria "fell short"), or reasoning depth (neither supported long-thinking). Kimi-VL closes these gaps simultaneously—128K context with verified retrieval, superior OCR via native-resolution encoding (83.2 on InfoVQA vs. 43.8 for Gemma-3-12B-IT), and long-thinking that scales monotonically with token budget (MathVision: 18.7% → 36.8% from 1K to 16K tokens; Figure 13). The implication is that a properly designed MoE VLM is not just more efficient at matched capability—it can be more capable at matched activation budget.

The paper also reconciles a latent contradiction in the reasoning-model literature that had been emerging but not explicitly articulated. Early long-thinking models (OpenAI o1, Kimi k1.5) demonstrated dramatic reasoning improvements but were proprietary, leaving open whether the approach would transfer to smaller open-source models. Simultaneously, the few open-source attempts at training reasoning models showed patterns of capability regression on non-reasoning tasks—a degradation that threatened to make long-thinking a specialized tool rather than a general capability. Kimi-VL-Thinking-2506 demonstrates that this regression is not inherent: the model simultaneously improves on hard reasoning (MathVision +20.1 points over the original thinking variant) and on general perception (MMBench: 76.0 → 84.4; MMStar: 64.2 → 70.4; ScreenSpot-Pro: 52.8, up from 34.5 in the instruct model), all while reducing token consumption by ~20% on reasoning benchmarks and 3× on general ones. This resolves the apparent contradiction—long-thinking training can degrade general ability if done naively, but with continued vision encoder training and refined RL (the unspecified improvements in the 2506 variant), the degradation is not only reversible but the thinking process can actually strengthen general capabilities.

The research directions this work reorients are substantial. It makes MoE-first VLM design the default hypothesis—the burden of proof now shifts to advocates of dense architectures to demonstrate that their efficiency disadvantages are justified by qualitatively different capabilities, rather than the reverse. It makes native-resolution encoding a baseline expectation rather than an exotic alternative; the 83.2 on InfoVQA vs. competitors' 34–44% on fixed-resolution dense models provides a stark efficiency-capability frontier that future VLMs must address. It makes multimodal long-thinking an open-source priority rather than a proprietary niche—the paper provides sufficient detail (the RL objective, the warmup dataset construction, the length penalty, the curriculum and prioritized sampling strategies) that other groups can replicate and extend the approach. And it shifts the frame on capability regression during reasoning training from an accepted cost to a solvable engineering problem—Kimi-VL-Thinking-2506 establishes that the regression seen in earlier thinking variants is addressable, changing the default expectation from "thinking training will hurt general ability" to "with proper training, thinking should improve general ability."

What becomes less attractive after this work: (1) dense-only VLM architectures without a compelling efficiency narrative, since Kimi-VL demonstrates that a 2.8B-activated MoE can match or exceed 7–12B dense models across 19/24 benchmarks; (2) multi-crop vision processing as the default approach for high-resolution images, since native-resolution encoding achieves superior OCR and grounding without the quadratic token cost of multi-crop pipelines; (3) fixed-context evaluation of VLMs without reporting context-position-stratified metrics, since the NIAH results (Table 2) reveal meaningful degradation at 64K–128K that aggregate benchmark scores conceal.

Follow-Up Research This Work Enables

Difficulty-conditioned thinking budget allocation with position-aware context retrieval. The paper demonstrates test-time scaling curves (Figure 13) showing that MathVista saturates at 4K thinking tokens while MathVision improves monotonically to 16K, and NIAH results (Table 2) showing retrieval degrades from 100% at 64K to 87% at 128K for text. A natural follow-up is to design a meta-policy that, given a prompt, estimates both the required thinking depth and the likely position of relevant information in the context, then jointly allocates the thinking budget and positions the most relevant context segments earlier in the window. A concrete experiment: on MMLongBench-Doc, stratify questions by where the answer-relevant text appears (early vs. middle vs. late in the document), measure accuracy vs. position, then test whether a retrieval-augmented system that re-orders document chunks to place likely-relevant sections before 64K can recover the NIAH-measured retrieval gap while using a modest thinking budget. The paper's open-source release makes this feasible—researchers can instrument the model's attention patterns or probe representations to build a lightweight difficulty-and-position estimator without retraining.

Isolating the causal contribution of native-resolution encoding through controlled swap experiments. The paper attributes strong OCR and grounding performance (InfoVQA: 83.2; ScreenSpot-Pro: 34.5) to MoonViT's native-resolution design but provides no controlled ablation. A decisive experiment would: train a variant of Kimi-VL where MoonViT is replaced with a fixed-resolution SigLIP-SO-400M encoder plus a standard multi-crop pipeline (e.g., 9 crops at 384×384, features concatenated or pooled), keeping the language model, training data, and all training stages identical, then evaluate on the full benchmark suite. The performance delta on InfoVQA, OCRBench, ScreenSpot-Pro, and V* would isolate the native-resolution contribution. A negative result—where the multi-crop version matches or exceeds MoonViT—would redirect attention to training data quality rather than encoder architecture, which is valuable for the field. The paper's public release of training infrastructure details (Section 2.5, Appendix C on data loading) makes this experiment architecturally feasible for groups with sufficient compute.

Faithfulness auditing of long-thinking reasoning traces through causal intervention. The RL objective rewards only answer correctness (Equation 1, binary reward), providing no guarantee that generated reasoning traces reflect the model's actual computation. A critical stress-test would: (1) build a dataset of MathVision problems where the model produces correct answers, (2) perturb the reasoning trace by deleting, reordering, or contradicting specific reasoning steps while keeping the final answer, (3) measure whether the model's confidence or answer distribution shifts—if the reasoning is merely decorative, perturbations should have minimal effect; if it is causally relevant, perturbations should degrade performance. Concurrently, measure whether reasoning steps that reference specific visual details (e.g., "the angle at vertex A is 45°") are grounded in the image by occluding those regions and measuring answer stability. The paper's qualitative examples (Figures 6, 8) show impressive-looking reasoning, but systematic faithfulness evaluation would determine whether this is genuine or a form of elaborate confabulation that happens to correlate with correct answers. A negative finding—that the model's reasoning is largely unfaithful despite high accuracy—would significantly change how practitioners should use and interpret Kimi-VL-Thinking's outputs.

Scaling law analysis across MoE activation budgets for multimodal capability. The paper provides one data point: 2.8B activated parameters. The natural extension is a scaling study that trains Kimi-VL variants at, say, 1B, 2.8B, and 7B activated parameters (all with identical architecture, vision encoder, training data, and pipeline), measuring performance across the 24-benchmark suite at each scale. The specific question: does the efficiency advantage of MoE over dense architectures hold across scales, or is there a crossover point where dense models catch up? The paper reports that Kimi-VL's training throughput is ~60% higher than a 7B dense VLM (Section 2.5), suggesting MoE advantages persist, but a controlled scaling study would reveal whether the gap widens (if expert specialization compounds with scale) or narrows (if routing becomes a bottleneck). Such a study would also answer whether the parameter ceiling the paper acknowledges (Section 5: "too limited to address highly specialized or domain-specific problems") is primarily a function of activated parameters (in which case scaling helps) or total parameters (in which case the MoE design already mitigates it, and scaling activated parameters may be less impactful than scaling the expert count).

Combining PRM-guided search with long-thinking for verifier-guided chain-of-thought optimization. The paper's thinking model generates reasoning traces autoregressively without step-level verification—it can produce incorrect reasoning that happens to lead to a correct answer (or vice versa) without correction during generation. The process reward model (PRM) approach from the reference paper—where each reasoning step is scored and search algorithms prune unpromising branches—could be combined with Kimi-VL-Thinking by training a PRM on the model's own reasoning traces (using Monte Carlo rollout supervision, as in the reference paper's Section 5.1) and then using beam search over reasoning steps at inference time. The key question: would step-level verification reduce the incidence of unfaithful reasoning and improve accuracy beyond what raw token budget scaling achieves? A concrete experiment: on MathVision, compare accuracy at matched total token budgets between (a) standard long-thinking (autoregressive generation with a fixed budget) and (b) PRM-guided beam search over reasoning steps (where each step is scored, and only top-K continuations survive). The reference paper found that beam search over PRM scores helped on medium-difficulty problems but over-optimized on easy ones—the interaction between difficulty and search could be characterized for multimodal reasoning using Kimi-VL's difficulty distribution. Since Kimi-VL already has a strong base PRM-equivalent capability (it can evaluate correctness—this is what the RL reward signal captures), training an explicit PRM is feasible using the same training data pipeline.

Robustness to adversarial visual perturbations that exploit native-resolution encoding. MoonViT's native-resolution processing is presented as a strength, but it may create unique vulnerabilities: since the model sees fine detail, subtle adversarial perturbations at the pixel level (e.g., modifying a few characters in an OCR-heavy document, slightly altering the label on a UI button, adding imperceptible noise to a math diagram) could have outsized effects compared to fixed-resolution encoders that lose such detail to resizing. A stress-testing study would: create adversarial variants of InfoVQA, ScreenSpot-Pro, and MathVision inputs by introducing targeted perturbations—changing a few digits in financial tables, slightly shifting UI element positions, adding misleading text in image margins—and measuring whether Kimi-VL's accuracy degrades more sharply than fixed-resolution baselines (Qwen2.5-VL-7B, Gemma-3-12B-IT). A finding that native-resolution models are more vulnerable to fine-grained adversarial perturbations would not invalidate the approach but would identify a deployment risk that high-stakes applications need to mitigate. The paper provides no robustness evaluation of any kind.

Practical Applications and Downstream Use Cases

On-device or edge-deployed document understanding for resource-constrained environments. The combination of MoE efficiency (2.8B activated parameters) and strong OCR performance (InfoVQA: 83.2, OCRBench: 867/1000) makes Kimi-VL viable for deployment on hardware where a 7B+ dense model would exceed memory or latency budgets. A concrete scenario: a mobile application that scans multi-page financial documents, legal contracts, or academic papers and answers user questions about their content. The 128K context window enables processing documents up to ~100 pages in a single forward pass (MMLongBench-Doc: 35.1 for the instruct model, 42.1 for the thinking variant), while the ~180-token average output on simple queries (Section 4.3) means most user interactions are lightweight. The efficiency gain over dense alternatives is quantifiable: the paper reports ~60% higher training throughput than a 7B dense VLM, and at inference, activating 2.8B vs. 7B parameters per token translates to approximately 2.5× fewer FLOPs per generated token for the same sequence length.

GUI automation agents for desktop and web task execution. Kimi-VL achieves 8.22 on OSWorld (vs. GPT-4o's 5.03) and 10.4 on WindowsAgentArena (vs. GPT-4o's 9.4)—leading among open-source models on multi-step agent interaction benchmarks. The ScreenSpot-Pro score of 34.5 (instruct) and 52.8 (thinking-2506) on 4K-resolution screenshots demonstrates practical UI element grounding at resolutions matching real desktop deployments. A downstream application: an open-source desktop assistant that executes multi-step tasks (e.g., "organize my downloads folder by file type and create a summary spreadsheet") by perceiving the screen, planning actions, and interacting with UI elements. The thinking variant's chain-of-thought capability (Figure 10 demonstrates step-by-step Chrome settings navigation) provides interpretable action plans that users can review before execution, reducing the trust barrier for automated desktop control. The key advantage over proprietary alternatives (GPT-4o, Claude Computer Use) is local deployment—sensitive screen content never leaves the user's device.

Automated video content indexing and retrieval at scale. Kimi-VL's video understanding benchmarks (Video-MME without subtitles: 67.8, leading all efficient VLMs; MLVU MCQ: 74.2, state-of-the-art including over GPT-4o's 64.6; EgoSchema: 78.5, significantly ahead of GPT-4o's 72.2) and demonstrated video scene splitting capability (Figure 11) enable automated processing of large video archives—corporate training libraries, surveillance footage, content moderation pipelines, educational video repositories. The model can segment hour-long videos into scenes with timestamps and natural-language descriptions, then answer questions about specific events distributed across the video (Video-MMU: 65.2 for the thinking-2506 variant). A concrete deployment: a media archive system where users search for "the scene where the presenter discusses the quarterly revenue decline" across thousands of earnings call recordings, with the model pre-processing videos into indexed scene descriptions and retrieving relevant segments at query time. The 128K context with 100% NIAH recall up to 64K tokens (Table 2) means the model can hold approximately 30–60 minutes of video frames in working memory for dense analysis, with only mild degradation (91.7% video NIAH) in the 64K–128K range for hour-plus content.

Multimodal educational reasoning with controllable explanation depth. The thinking variant's test-time scaling behavior (Figure 13) enables a tutoring application where the explanation depth adapts to student needs: for a struggling student on a geometry problem, allocate 16K thinking tokens for a detailed step-by-step walkthrough with explicit theorem citation (as in Figure 8); for a student checking their work, allocate 1K tokens for concise verification. MathVista (80.1 for the thinking-2506 variant, state-of-the-art among open-source models) and MathVision (56.9, exceeding models with 10×+ more parameters) demonstrate the reasoning quality available. The token efficiency on simple queries (180 tokens average on MMBench) means the system does not waste computation generating verbose explanations when students ask straightforward fact-checking questions. The interpretable reasoning traces provide pedagogical value beyond answer correctness—students can see the model's planning, evaluation, and error-correction processes (the cognitive patterns taught during long-CoT SFT warmup), which model effective problem-solving strategies. The open-source release means educational platforms can fine-tune on domain-specific curricula without vendor lock-in.

When to Prefer This Method

The paper positions Kimi-VL against both open-source dense VLMs (Qwen2.5-VL, Gemma-3) and proprietary systems (GPT-4o, GPT-4o-mini), with specific tradeoffs emerging from the benchmark comparisons in Tables 3–5. The decision rule is not a universal matrix but a set of capability-cost comparisons informed by the paper's evidence:

Prefer Kimi-VL (instruct or thinking-2506 variant) when OCR quality, high-resolution UI grounding, or fine-grained visual perception is the primary requirement. The gap is largest here: 83.2 on InfoVQA vs. 43.8 for Gemma-3-12B-IT, 34.5–52.8 on ScreenSpot-Pro vs. 0.8 for GPT-4o (though the GPT-4o result uses degraded tooling; see footnote qualification in Section 4.1.5), and 92.8 on ScreenSpot-V2. These are not marginal advantages—on tasks where reading small text or precisely localizing UI elements is the bottleneck, Kimi-VL dominates its weight class.
Prefer Kimi-VL-Thinking-2506 when mathematical reasoning with visual context is needed and test-time thinking can be allocated. MathVista 80.1 exceeds all compared models in Table 4 including GPT-4o (63.8), Qwen2.5-VL-72B (74.8), and QVQ-72B-Preview (71.4). MathVision 56.9 exceeds Gemma-3-27B-IT (35.5) and QVQ-72B (35.9). The cost is thinking tokens (up to 16K for hard problems), but the thinking-2506 variant's 20% token reduction over the original thinking model means this cost is declining as training improves.
Prefer larger dense or proprietary models (GPT-4o, O1, Kimi k1.5) when broad college-level knowledge across multiple disciplines is the primary requirement and efficiency is secondary. On MMMU, Kimi-VL-Thinking-2506 scores 64.0 vs. GPT-4o's 69.1 and O1's 74.8. On MMMU-Pro, 46.3 vs. GPT-4o's 51.7 and O1's 51.1. The parameter ceiling (Section 5) is most evident on knowledge-intensive benchmarks where raw parameter count and training data diversity provide advantages that test-time scaling cannot fully compensate for.
Prefer Kimi-VL when local/private deployment is required—for sensitive documents, proprietary workflows, or offline operation. The paper's open-source release, MoE efficiency, and strong agent capabilities make it the leading option among models that can run entirely on user-controlled hardware. GPT-4o and Kimi k1.5 cannot serve this use case at all; dense open-source models can but at 2.5–4× higher per-token inference cost for comparable or weaker performance on key metrics.
The thinking budget is a tunable dial, not a fixed cost. For latency-sensitive applications with mixed difficulty, the model can be deployed with a low default budget (e.g., 1K tokens, sufficient for MathVista's ~67% accuracy) and escalated to higher budgets only when the initial response is low-confidence or the problem type is known to benefit from extended reasoning. The paper does not provide confidence estimation or automatic escalation logic, but the token efficiency on simple tasks (180 tokens on MMBench) means a conservative strategy that starts with a generous budget does not catastrophically waste compute—the model naturally generates short responses when the problem is simple, even if allowed more tokens.