Kimi k1.5: Scaling Reinforcement Learning with LLMs

ArXiv: 2501.12599

🎯 Pitch

A plain long-context RL setup—no Monte Carlo tree search, no value functions—matches o1’s reasoning by simply training with 128k-token rollouts. The same long-thinking ability is then compressed into short-CoT models that beat GPT‑4o and Claude Sonnet 3.5 by up to 550% on AIME and LiveCodeBench.

---

1. Executive Summary

This technical report presents the training recipe and infrastructure design of Kimi k1.5, a multi-modal LLM trained with reinforcement learning that achieves state-of-the-art reasoning performance by scaling context length during RL—matching OpenAI's o1 on multiple benchmarks (77.5 on AIME, 96.2 on MATH 500, 94th percentile on Codeforces, 74.9 on MathVista). The approach establishes a simplistic, effective RL framework that relies on two key mechanisms: long context scaling (extending the RL context window to 128k tokens, using partial rollouts to reuse segments of previous trajectories across training iterations) and improved policy optimization (a variant of online mirror descent with length penalty, curriculum sampling, and prioritized sampling—without Monte Carlo tree search, value functions, or process reward models). The report also presents long2short methods that transfer the thinking priors from long-CoT models to short-CoT models via model merging, shortest rejection sampling, DPO, and a dedicated long2short RL phase with length penalty, yielding state-of-the-art short-CoT results—60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench—that outperform GPT-4o and Claude Sonnet 3.5 by up to +550%, establishing that long-context RL capabilities can be effectively distilled into token-efficient models without sacrificing the reasoning gains achieved through extended chain-of-thought exploration.

2. Context and Motivation

The Core Problem: Pretraining Data Is Finite, But Reasoning Requires Unbounded Improvement

The paper opens by identifying a fundamental constraint on the dominant paradigm for scaling language models. The pretraining approach—training on next-token prediction with proportionally scaled model parameters and data—has been validated by scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) as a reliable path to improved intelligence. However, this approach hits a hard ceiling:

"this approach is limited to the amount of available high-quality training data (Villalobos et al. 2024; Muennighoff et al. 2023)"

The paper frames this not as a temporary bottleneck but as a structural limitation. High-quality training data—human-authored textbooks, research papers, competition problems, curated code—is a finite resource. While synthetic data generation can partially alleviate this, the core insight is that static datasets cannot teach a model to explore: they contain correct reasoning paths but not the trial-and-error process that produced those paths, the dead ends that were abandoned, or the backtracking that corrected initial mistakes. A model trained solely on correct solutions never learns how to recover from errors, which is precisely the skill needed for complex, novel reasoning problems.

This motivates the paper's central goal: to explore reinforcement learning as a new axis for continued scaling that is not constrained by pre-existing static datasets. The key phrase in the introduction is:

"Using RL with LLMs, the model learns to explore with rewards and thus is not limited to a pre-existing static dataset."

The theoretical significance is that this shifts the scaling paradigm from data-limited pretraining to exploration-driven self-improvement. The practical significance is that if successful, it would enable models to continue improving on reasoning tasks even after exhausting available training corpora, by generating their own training signal through interaction with verifiable reward functions.

The Gap: Prior Published RL Work Has Not Produced Competitive Results

The paper makes a striking claim in its abstract:

"prior published work has not produced competitive results"

This is not merely an observation—it is the central gap the paper aims to fill. Despite the theoretical promise of RL for LLMs (demonstrated in controlled settings by Ouyang et al., 2022, for instruction following), no published work prior to this report had achieved reasoning performance competitive with frontier models like OpenAI's o1 using RL training. The paper positions itself explicitly against this backdrop: it is not proposing RL as a speculative direction but claiming to have solved the practical challenges that prevented prior efforts from reaching competitive performance.

Why had prior RL efforts fallen short? The paper identifies several interconnected reasons through its design choices and ablation studies:

1. Context length was too short for complex reasoning. The paper frames context length as the central scaling dimension for RL. Prior work typically operated with limited context windows (often 2k–8k tokens), which constrained the model's ability to generate extended chains of thought involving planning, reflection, error identification, and backtracking. The paper's key insight is that these cognitive processes—which the paper explicitly names as planning, evaluation, reflection, and exploration (Section 2.2)—require substantial token budgets to unfold naturally. Short context windows force the model to compress its reasoning, losing the very behaviors that make long-CoT valuable.

2. Policy optimization methods were not adapted for the long-CoT setting. The paper's ablation comparing its online mirror descent variant against ReST (Gulcehre et al., 2023) in Figure 10 reveals that the choice of optimization algorithm matters significantly for long-CoT generation. ReST—which iteratively fits the best response without applying negative gradients to penalize incorrect responses—shows substantially worse sample complexity. The paper interprets this as evidence that negative gradients are crucial for teaching the model what not to do: identifying and penalizing incorrect reasoning paths so the model learns to avoid dead ends. In domains where ReST was previously studied, the performance gap between it and other RL methods was not pronounced, suggesting that long-CoT generation has unique optimization requirements.

3. Infrastructure for long-context RL was not available. Training with 128k-token context windows during RL—where the model generates entire trajectories, receives rewards, and updates its policy—poses severe computational challenges. Long trajectories can dominate rollout workers, causing stragglers that leave other GPUs idle. The paper's partial rollout technique (Section 2.6.2) and hybrid deployment system (Section 2.6.3) are presented as solutions to problems that had previously made long-context RL infeasible at scale. Without these infrastructure innovations, prior work was effectively capped at context lengths too short to exhibit the emergent reasoning behaviors the paper seeks.

4. Reward hacking and evaluation reliability were not adequately addressed. The paper devotes Section 2.1 to prompt set curation, specifically targeting the problem that "some complex reasoning problems may have relatively simple and easily guessable answers, leading to false positive verification—where the model reaches the correct answer through an incorrect reasoning process." This is a subtle but devastating failure mode for RL: if the reward signal can be gamed by superficial patterns, the policy will learn to produce those patterns rather than genuine reasoning. Prior work that did not carefully filter for evaluability would have suffered from this degradation in training signal quality, producing models that appeared to improve on training metrics but failed to generalize.

Prior Approaches and Their Limitations

The paper situates itself against two broad families of prior work:

Planning-augmented CoT methods. These approaches explicitly construct search trees over reasoning steps, guided by value functions or process reward models. The paper cites Tree of Thoughts (Yao et al., 2024), inference scaling laws work (Wu et al., 2024; Snell et al., 2024), and related planning methods. The core insight of these approaches is that by exploring multiple reasoning paths and using critics to evaluate partial solutions, the model can find solutions that would be missed by greedy single-path generation.

However, the paper identifies specific shortcomings with this family:

• Complex parallelization requirements at deployment: Tree search requires maintaining and evaluating multiple partial solutions simultaneously, which is computationally expensive and difficult to parallelize efficiently compared to auto-regressive generation. This makes deployment in production systems challenging.

• Dependence on separate critic models: Methods using value functions or process reward models require training an additional model (the critic) alongside the policy. This adds complexity to the training pipeline and introduces potential distribution shift between the critic's training distribution and the policy's evolving output distribution.

• The paper's alternative framing: The paper proposes a conceptual unification in Section 2.3.1 that treats planning algorithms as mappings over flattened reasoning sequences. Since thoughts and feedback are both language sequences, and search history can be concatenated into context, a sufficiently capable auto-regressive model with a long enough context window could implicitly perform the search in its own forward pass. The paper articulates this elegantly:

"Rather than explicitly constructing a search tree and implementing a planning algorithm, we could potentially train a model to approximate this process. Here, the number of thoughts (i.e., language tokens) serves as an analogy to the computational budget traditionally allocated to planning algorithms."

This framing is the intellectual foundation for the paper's "simplistic framework"—by scaling context length instead of building explicit search machinery, the model can learn to plan, evaluate, and backtrack through its own chain of thought without external critics or tree structures.

Prompt-based CoT and self-correction methods. Prior work on prompting models to "think step by step" (Wei et al., 2022) and to self-correct (Madaan et al., 2023; Bai et al., 2022) demonstrated that language models could produce more accurate reasoning when prompted to show their work. However, these methods have well-documented limitations:

• They rely on the model's pre-existing capabilities—prompting cannot teach new reasoning strategies, only elicit existing ones.
• Self-correction through prompting alone is largely ineffective for complex reasoning (Huang et al., 2023), because the model lacks training signal to distinguish good corrections from bad ones.
• The reasoning produced is typically linear and lacks the exploration, backtracking, and error recovery that characterize human problem-solving on difficult tasks.

The paper positions its approach as synthesizing the strengths of both families while avoiding their weaknesses. The model should auto-regressively sample reasoning during inference (avoiding complex parallelization), but should have learned through RL the planning skills that planning algorithms explicitly implement—including error identification, backtracking, and solution refinement. The key distinction the paper draws:

"a key distinction from simple prompt-based methods is that the model should not merely follow a series of reasoning steps. Instead, it should also learn critical planning skills including error identification, backtracking and solution refinement by leveraging the entire set of explored thoughts as contextual information."

How This Paper Positions Itself

The paper's positioning can be understood along four axes:

Against planning-augmented methods: The paper claims that long context scaling + RL can achieve what explicit tree search achieves, but with a simpler architecture. This is not presented as a theoretical proof but as an empirical demonstration—the results in Table 2 show performance matching o1 without Monte Carlo tree search, value functions, or process reward models. The implication is that the complexity of planning algorithms may be unnecessary if the model can learn to conduct implicit search through its own extended chain of thought.

Against prior RL work: The paper claims to be the first to produce competitive results with RL training, attributing prior failures to insufficient context length, suboptimal policy optimization, and infrastructure limitations. The detailed ablation studies—comparing against ReST (Figure 10), analyzing curriculum sampling (Figure 9), and demonstrating the relationship between context length and performance (Figures 5 and 6)—are designed to substantiate this claim by isolating which ingredients matter.

Against pure pretraining: The paper frames RL as complementary to pretraining, not a replacement. The training pipeline described in Section 2 includes pretraining, vanilla SFT, and long-CoT SFT before RL. The RL phase builds on capabilities established during pretraining and SFT, refining them through exploration. This positions RL as an "unlocking" mechanism rather than a ground-up training paradigm.

For the long2short paradigm: The paper's long2short results (Figure 7) represent a novel contribution that extends beyond the long-CoT model itself. The insight is that the reasoning capabilities developed during long-context RL can be distilled into shorter, more token-efficient models without losing most of the performance gain. This addresses a practical criticism of long-CoT models—their high inference cost—and shows that the benefits of RL training can be realized even under tight token budgets. The paper explicitly frames this as potentially iterative:

"it is possible to combine long2short methods with long-CoT RL in an iterative way to further increase token efficiency and extract the best performance out of a given context length budget."

This iterative vision—long-CoT RL to develop reasoning, long2short to compress it, then further long-CoT RL on the compressed model—suggests a self-reinforcing cycle that the paper presents as a direction for future work but clearly views as the natural extension of its framework.

The Unstated Motivation: Reconciling o1's Opaque Success

While the paper does not state this explicitly, a significant contextual motivation is the release of OpenAI's o1 model, which demonstrated dramatic reasoning improvements through undisclosed training methods. The AI community was left speculating about what techniques produced these gains—was it tree search? Process reward models? Specialized architectures? The Kimi k1.5 report can be read as a response to this opacity: it demonstrates that competitive reasoning performance can be achieved through a well-understood combination of long-context RL and careful data engineering, without the complex techniques many assumed were necessary. The paper's emphasis on a "simplistic framework" and its detailed disclosure of training recipes, infrastructure design, and ablation studies positions it as a transparency contribution as much as a technical one.

3. Technical Approach

3.1 Reader Orientation

The Kimi k1.5 system is a multi-modal language model trained through a multi-stage pipeline that culminates in large-scale reinforcement learning on reasoning tasks, where the model generates extended chains of thought (long-CoT) and receives rewards based on the correctness of its final answers. The core problem it solves is how to improve an LLM's complex reasoning capabilities beyond the limits of static pretraining data. The "shape" of the solution is to scale the context window during RL to 128k tokens, allowing the model to learn planning, reflection, error identification, and backtracking behaviors through trial-and-error exploration, and then to distill these learned reasoning capabilities into shorter, more token-efficient models through long2short techniques—all without relying on explicit tree search, value functions, or process reward models.

3.2 Big-Picture Architecture (Diagram in Words)

The Kimi k1.5 training pipeline consists of five major stages, with four supporting infrastructure components that enable large-scale RL:

Training Stages:

1. Pretraining (Section 2.5.1, Appendix B): Trains a base Transformer decoder on a diverse multimodal corpus (text + vision) across three sub-phases: vision-language pretraining (language foundation, then gradual multimodal integration), cooldown (consolidating capabilities with curated and synthetic data), and long-context activation (extending sequence processing to 131,072 tokens).

2. Vanilla Supervised Fine-Tuning (Section 2.5.2): Trains the pretrained model on approximately 1 million text examples (general QA, coding, math/science, creative writing, long-context tasks) plus 1 million text-vision examples, using rejection sampling for reasoning tasks where rule-based verification is available.

3. RL Prompt Set Curation (Section 2.1): Filters and selects prompts for RL training based on three criteria: diverse coverage across STEM/coding/reasoning domains, balanced difficulty (assessed by the SFT model's pass rate on 10 sampled answers), and accurate evaluability (excluding easy-to-hack prompts where answers can be guessed without reasoning, identified by testing whether a model can guess the correct answer within 8 attempts without CoT).

4. Long-CoT Supervised Fine-Tuning (Section 2.2): Constructs a small, high-quality warmup dataset of long-CoT reasoning paths through prompt engineering (resembling rejection sampling but focused on generating extended reasoning). These paths encode cognitive processes—planning, evaluation, reflection, exploration. Lightweight SFT on this dataset primes the model for RL.

5. Reinforcement Learning (Section 2.3): The central training phase. The model generates CoT solutions for prompts, receives rewards based on answer correctness, and updates its policy using a variant of online mirror descent. Key mechanisms: long context scaling (128k tokens), partial rollouts (reusing trajectory segments across iterations), length penalty (to control verbosity), and adaptive sampling strategies (curriculum + prioritized). Runs iteratively with a reference model that updates each iteration.

Infrastructure Components:

• Rollout Workers + Central Master (Section 2.6.1): Coordinate trajectory generation during RL, storing experiences in a replay buffer.
• Reward Models (Section 2.3.5): Evaluate answer correctness—a Chain-of-Thought RM for math (98.5% accuracy vs. 84.4% for classic RM), a code execution sandbox for coding, and rule-based verifiers for other domains.
• Partial Rollout System (Section 2.6.2): Handles long trajectories by breaking them into segments across iterations, reusing prior segments from the replay buffer to avoid re-generation.
• Hybrid Deployment Framework (Section 2.6.3): Collocates training (Megatron) and inference (vLLM) on the same GPUs using Kubernetes Sidecar containers, with a checkpoint engine managing weight transfer between phases in under one minute.

Information Flow: Prompt set → Curated RL prompts → Long-CoT SFT model → RL loop (rollout workers generate trajectories using current policy → reward models evaluate correctness → trainer workers compute gradient updates → updated policy becomes reference for next iteration) → Long-CoT model. For short-CoT: Long-CoT model → Long2short methods (model merging, shortest rejection sampling, DPO, long2short RL) → Short-CoT model.

3.3 Roadmap for the Deep Dive

• First, the formal RL problem setting and the conceptual unification of planning algorithms with auto-regressive generation (Section 2.3.1), because this establishes the theoretical motivation for why long-context RL can substitute for explicit tree search.
• Second, the policy optimization algorithm—the variant of online mirror descent (Section 2.3.2)—since this is the mathematical engine driving all learning and has specific properties (off-policy correction, $\ell_2$-regularization, no value network) that distinguish it from standard policy gradient methods.
• Third, the length penalty mechanism (Section 2.3.3), which addresses the overthinking problem that naturally emerges during long-CoT RL and is essential for practical deployment.
• Fourth, the sampling strategies—curriculum and prioritized sampling (Section 2.3.4)—because they determine which problems the model trains on at each iteration and significantly impact training efficiency.
• Fifth, the training recipe details (Section 2.3.5), including reward modeling for math, test case generation for coding, and vision data composition, as these supply the reward signal that drives all RL.
• Sixth, the long2short methods (Section 2.4), which compress the long-CoT model's reasoning into token-efficient short-CoT models.
• Seventh, the RL infrastructure (Section 2.6), particularly partial rollouts and hybrid deployment, because these engineering innovations make 128k-context RL feasible at scale and are presented as key enabling contributions.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methodology paper whose core idea is that scaling context length during RL, combined with improved policy optimization and infrastructure, enables LLMs to learn complex reasoning behaviors (planning, reflection, backtracking) without explicit search algorithms, and that these capabilities can be distilled into shorter models.

---

3.4.1 RL Problem Setting and the Unification of Planning with Auto-Regressive Generation

The paper begins by formalizing the RL training setup and then makes a conceptual move that underpins its entire approach: reframing planning algorithms as operations over flattened language sequences, which can therefore be approximated by a sufficiently capable auto-regressive model with a long enough context window.

Training objective. Given a dataset $D = \{(x_i, y^*_i)\}_{i=1}^n$ of problems $x_i$ and ground truth answers $y^*_i$, the goal is to train a policy model $\pi_\theta$ to generate correct solutions. For complex reasoning, the model uses chain of thought (CoT): a sequence of intermediate reasoning steps $z = (z_1, z_2, \ldots, z_m)$ bridges the problem $x$ and the final answer $y$. At inference time, thoughts are sampled auto-regressively: $z_t \sim \pi_\theta(\cdot|x, z_1, \ldots, z_{t-1})$, followed by $y \sim \pi_\theta(\cdot|x, z_1, \ldots, z_m)$. The notation $y, z \sim \pi_\theta$ denotes this full sampling procedure.

Planning algorithms as sequence mappings. The paper describes planning algorithms (like Tree of Thoughts) that explicitly construct a search tree $T$ where each node represents a partial solution $s = (x, z_{1:|s|})$. A critic model $v$ provides feedback $v(x, z_{1:|s|})$ evaluating progress toward the solution. The planning algorithm selects the most promising node for expansion based on this feedback, growing the tree iteratively until a full solution is derived.

The paper then makes a crucial reframing. Since both thoughts and feedback can be represented as language sequences, and the algorithm's decision about which node to expand next is a function of the entire search history, we can view the planning algorithm as a mapping:

$A(\cdot|z_1, z_2, \ldots)$

This mapping directly acts on a sequence of reasoning steps. All information stored in the search tree is flattened into the full context. The paper articulates the implication:

"Rather than explicitly constructing a search tree and implementing a planning algorithm, we could potentially train a model to approximate this process. Here, the number of thoughts (i.e., language tokens) serves as an analogy to the computational budget traditionally allocated to planning algorithms."

Why this matters. This reframing is not merely philosophical—it is the justification for the entire technical approach. If extended CoT tokens function analogously to search steps, then scaling context length during RL training allows the model to learn search-like behaviors (exploring alternatives, identifying dead ends, backtracking, refining solutions) through the same auto-regressive mechanism it uses for everything else. The model does not need a separate tree structure or critic network because the tree and the critic are implicitly represented in the sequence of tokens the model generates and conditions on.

RL objective. Given a reward model $r$ that assigns $r(x, y, y^*) \in \{0, 1\}$ based on whether the answer $y$ matches the ground truth $y^*$ (either through rule-based verification or a learned reward model), the optimization objective is:

$\max_\theta \mathbb{E}_{(x,y^*) \sim D, (y,z) \sim \pi_\theta} \left[ r(x, y, y^*) \right]$

where $(x, y^*)$ is a problem-answer pair sampled from the training dataset $D$, $(y, z)$ is a CoT-and-answer sampled from the current policy $\pi_\theta$, and $r(x, y, y^*)$ is the binary reward (1 if correct, 0 otherwise).

What it computes: the expected reward over the distribution of problems in the training set and the distribution of CoT solutions generated by the current policy. In operational terms: for each problem, the model generates a reasoning trace and answer; the answer is checked against the ground truth; the model is updated to increase the probability of reasoning traces that lead to correct answers.

Why this form: the expectation over both the data distribution and the policy distribution is the standard RL objective. Critically, the reward depends only on the final answer's correctness, not on intermediate steps. This means the model receives a single scalar signal per full trajectory—it must learn on its own which intermediate reasoning choices contributed to success or failure. This is a harder credit assignment problem than step-level rewards would provide, but the paper argues (Section 2.3.2) that it is actually beneficial because it forces the model to explore diverse paths and learn recovery strategies rather than being penalized for every wrong turn.

---

3.4.2 Policy Optimization: Online Mirror Descent Variant

The paper applies a variant of online mirror descent (OMD) as its policy optimization algorithm. This is not standard PPO or REINFORCE—it has specific properties tailored to the long-CoT RL setting.

Iterative formulation. At iteration $i$, the current model $\pi_{\theta_i}$ serves as a reference policy. The optimization problem for the next policy $\pi_\theta$ is:

$\max_\theta \mathbb{E}_{(x,y^*) \sim D} \left[ \mathbb{E}_{(y,z) \sim \pi_\theta} [r(x, y, y^*)] - \tau \cdot \text{KL}(\pi_\theta(x) \| \pi_{\theta_i}(x)) \right]$

where $\tau > 0$ is a regularization parameter controlling how far the new policy can deviate from the reference, and $\text{KL}(\pi_\theta(x) \| \pi_{\theta_i}(x))$ is the Kullback-Leibler divergence between the new and reference policies' output distributions for problem $x$.

What it computes: the objective maximizes expected reward while penalizing deviation from the reference policy. The KL term acts as a trust region—it prevents the policy from changing too drastically in a single iteration, which is critical for stable RL with large models.

Why this form: pure reward maximization without regularization can cause the policy to collapse to a narrow distribution (e.g., always producing the same high-reward output), losing the diversity needed for continued exploration. The KL penalty maintains proximity to the reference policy, which has been shaped by all previous iterations and thus encodes accumulated knowledge. This is the standard motivation for KL-regularized RL (used in RLHF), but it takes on additional importance here because exploration is central to the paper's approach—the model must continue trying diverse reasoning paths throughout training.

Closed-form solution. The KL-regularized objective has a known closed-form optimal policy:

$\pi^*(y, z|x) = \pi_{\theta_i}(y, z|x) \exp(r(x, y, y^*) / \tau) / Z$

where $Z = \sum_{y', z'} \pi_{\theta_i}(y', z'|x) \exp(r(x, y', y^*) / \tau)$ is the normalization factor (partition function).

Taking the logarithm of both sides yields a constraint that any optimal policy must satisfy, for all $(y, z)$:

$r(x, y, y^*) - \tau \log Z = \tau \log \frac{\pi^*(y, z|x)}{\pi_{\theta_i}(y, z|x)}$

What this constraint means: the reward for a response (shifted by the log-partition function) must equal $\tau$ times the log-ratio of the optimal policy's probability to the reference policy's probability. This is the key relationship that enables off-policy learning: it connects the reward signal to a policy ratio that can be evaluated on any data, not just data from the current policy.

Surrogate loss. The constraint motivates the following squared-error surrogate loss, which can be optimized using samples from the reference policy (off-policy data):

$L(\theta) = \mathbb{E}_{(x,y^*) \sim D} \left[ \mathbb{E}_{(y,z) \sim \pi_{\theta_i}} \left[ \left( r(x, y, y^*) - \tau \log Z - \tau \log \frac{\pi_\theta(y, z|x)}{\pi_{\theta_i}(y, z|x)} \right)^2 \right] \right]$

Approximating $\tau \log Z$. The partition function $Z$ is intractable to compute exactly (it requires summing over all possible responses). The paper uses $k$ samples $(y_1, z_1), \ldots, (y_k, z_k) \sim \pi_{\theta_i}$ to approximate it:

$\tau \log Z \approx \tau \log \left( \frac{1}{k} \sum_{j=1}^k \exp(r(x, y_j, y^*) / \tau) \right)$

The paper further notes that using the empirical mean of sampled rewards $\bar{r} = \text{mean}(r(x, y_1, y^*), \ldots, r(x, y_k, y^*))$ as a proxy for $\tau \log Z$ yields effective practical results.

Why this approximation works: as $\tau \to \infty$ (weaker regularization), $\tau \log Z$ approaches the expected reward under $\pi_{\theta_i}$ because the exponential terms flatten out. Using $\bar{r}$ as a baseline is computationally simpler and empirically effective—it centers the rewards, reducing variance in the gradient estimates.

Final gradient. Taking the derivative of the surrogate loss with respect to $\theta$, for a batch of problems where $k$ responses are sampled per problem using the reference policy $\pi_{\theta_i}$:

$\frac{1}{k} \sum_{j=1}^k \left( \nabla_\theta \log \pi_\theta(y_j, z_j|x) \cdot (r(x, y_j, y^*) - \bar{r}) - \frac{\tau}{2} \nabla_\theta \left( \log \frac{\pi_\theta(y_j, z_j|x)}{\pi_{\theta_i}(y_j, z_j|x)} \right)^2 \right)$

What it computes, term by term:

1. $\nabla_\theta \log \pi_\theta(y_j, z_j|x) \cdot (r(x, y_j, y^*) - \bar{r})$: This is the standard policy gradient with a mean baseline. It increases the log-probability of responses with above-average reward and decreases the log-probability of responses with below-average reward. The baseline $\bar{r}$ reduces variance by centering the advantages.

2. $-\frac{\tau}{2} \nabla_\theta (\log \pi_\theta / \pi_{\theta_i})^2$: This is an $\ell_2$-regularization term that penalizes the squared log-ratio between the new and reference policies. It prevents individual probability ratios from becoming too extreme, which corresponds to staying close to the reference policy in KL divergence.

Relationship to standard policy gradient. For those familiar with REINFORCE or PPO, this gradient resembles an off-policy regularized policy gradient:

• It uses samples from $\pi_{\theta_i}$ (off-policy) rather than from $\pi_\theta$ (on-policy), making it more sample-efficient because data from previous iterations can be reused.
• The $\ell_2$-regularization on log-ratios serves a similar purpose to PPO's clipping—preventing destructive large updates—but through a squared penalty rather than a hard clip.
• The mean baseline is a standard variance-reduction technique.

Design choice: no value network. The paper explicitly excludes a separate value network (critic), noting:

"While this design choice significantly improves training efficiency, we also hypothesize that the conventional use of value functions for credit assignment in classical RL may not be suitable for our context."

Rationale for omitting the value function (Section 2.3.2, extended argument). The paper provides a concrete example to justify this decision. Suppose the model has generated a partial CoT $(z_1, \ldots, z_t)$ and faces two possible next steps: $z_{t+1}$ (leads directly to the correct answer) and $z'_{t+1}$ (contains errors but the model could later recover). An oracle value function would assign higher value to $z_{t+1}$ and penalize $z'_{t+1}$ through negative advantage. However, from a learning perspective, exploring $z'_{t+1}$ and recovering from its errors is extremely valuable—it teaches the model error identification, backtracking, and solution refinement. The paper states:

"By using the justification of the final answer derived from a long CoT as the reward signal, the model can learn the pattern of trial and error from taking $z'_{t+1}$ as long as it successfully recovers and reaches the correct answer."

This is a critical design philosophy: the goal is not to maximize training accuracy (which a value function would help with) but to develop generalizable problem-solving strategies. Step-level credit assignment via a value function would discourage the exploration that generates the diversity of experience needed to learn these strategies. The final-answer-only reward signal allows the model to discover that error recovery is possible and valuable, because a trajectory that makes a wrong turn but then corrects itself still receives a positive reward.

Iteration mechanics. The paper describes the iterative procedure:

1. Sample a batch of problems from $D$.
2. For each problem, sample $k$ responses using the current reference policy $\pi_{\theta_i}$.
3. Compute rewards for each response.
4. Compute the gradient using Equation 3 and update parameters to $\theta_{i+1}$.
5. The updated model $\pi_{\theta_{i+1}}$ becomes the reference policy for the next iteration.
6. The optimizer is reset at the start of each iteration (since each iteration solves a different optimization problem due to the changing reference policy).

---

3.4.3 Length Penalty

A natural consequence of training models to generate long chains of thought with final-answer-only rewards is that the model learns that longer responses tend to produce higher rewards (more reasoning steps = more opportunities to catch errors). The paper observes:

"an overthinking phenomenon that the model's response length significantly increases during RL training. Although this leads to better performance, an excessively lengthy reasoning process is costly during training and inference, and overthinking is often not preferred by humans."

Length reward formulation. Given $k$ sampled responses $(y_1, z_1), \ldots, (y_k, z_k)$ for problem $x$, let $\text{len}(i)$ be the token length of response $i$, and define:

• $\text{min\_len} = \min_i \text{len}(i)$: the shortest response among the $k$ samples.
• $\text{max\_len} = \max_i \text{len}(i)$: the longest response among the $k$ samples.

The raw length score for response $i$ is:

$\lambda = 0.5 - \frac{\text{len}(i) - \text{min\_len}}{\text{max\_len} - \text{min\_len}}$

This score ranges from 0.5 (for the shortest response, where $\text{len}(i) = \text{min\_len}$) to -0.5 (for the longest response, where $\text{len}(i) = \text{max\_len}$). It is a normalized linear penalty: shorter responses get positive scores, longer responses get negative scores.

Why normalize by min and max within each batch: this makes the penalty adaptive to the specific problem's difficulty. For a hard problem where all responses are long, the penalty is distributed around the batch's natural length distribution rather than an absolute threshold. For an easy problem with naturally short responses, the same relative penalization applies.

Length reward assignment: The length reward is then conditioned on correctness:

$\text{len\_reward}(i) = \begin{cases} \lambda & \text{if } r(x, y_i, y^*) = 1 \\ \min(0, \lambda) & \text{if } r(x, y_i, y^*) = 0 \end{cases}$

What this means operationally:

• For correct responses: the full length score $\lambda$ is applied. Short correct responses get positive bonuses (up to +0.5); long correct responses get penalties (down to -0.5). This promotes concise correct reasoning.
• For incorrect responses: only non-positive length scores are applied—$\min(0, \lambda)$. Short incorrect responses receive 0 length reward (they are not rewarded for being short if they're wrong); long incorrect responses receive negative penalties (down to -0.5). This explicitly penalizes long-winded wrong answers, which are doubly undesirable (incorrect AND expensive).

Why condition on correctness: the paper wants to promote short correct responses without accidentally promoting short incorrect responses. If short incorrect responses received positive length rewards, the model might learn to give up quickly rather than persist through difficult reasoning. The $\min(0, \lambda)$ clamp ensures that incorrect responses are never rewarded for brevity, only penalized for verbosity.

Integration with the original reward. The length reward is added to the original binary reward $r(x, y_i, y^*)$ with a weighting parameter (not explicitly specified in the paper). The combined reward is used in the policy gradient computation.

Warmup schedule. The paper notes:

"In our preliminary experiments, length penalty may slow down training during the initial phases."

To address this, the length penalty is gradually introduced:

1. Phase 1: Standard policy optimization without length penalty—the model is allowed to develop long-CoT reasoning behaviors freely.
2. Phase 2: Constant length penalty applied for the remainder of training—once the model has learned to generate useful long CoTs, the penalty constrains further growth.

This warmup is crucial because applying length penalty too early would discourage the model from ever exploring long reasoning chains, preventing it from discovering the planning and reflection behaviors that require extended context.

---

3.4.4 Sampling Strategies

The paper introduces two adaptive sampling methods to improve RL training efficiency: curriculum sampling and prioritized sampling. Both exploit signals about problem difficulty that are naturally available during RL training.

Available difficulty signals. The paper identifies two sources of difficulty information:

1. Domain/tag-based difficulty: The training data includes problems from different sources—math competition problems are inherently harder than primary school math problems. The paper developed a tagging system (Section 2.1) to categorize prompts by domain and discipline.
2. Empirical success rate: Because RL samples each problem multiple times across iterations, the paper tracks the success rate $s_i$ for each problem $i$ (the fraction of sampled responses that are correct). This is an online, model-specific difficulty metric.

Curriculum sampling. The strategy is described as:

"We start by training on easier tasks and gradually progress to more challenging ones."

The motivation is practical:

"Since the initial RL model has limited performance, spending a restricted computation budget on very hard problems often yields few correct samples, resulting in lower training efficiency."

During early iterations, the model rarely produces correct answers on hard problems, so most samples receive zero reward and contribute little learning signal—the policy gradient for these samples is small because all responses in a batch may be incorrect (making $r - \bar{r} \approx 0$). Curriculum sampling concentrates the training budget on problems where the model can actually get positive signal.

Implementation (operational description): The paper describes using the full dataset $D$ for a warm-up phase, then focusing training solely on hard questions. The ablation in Figure 9 shows the transition at iteration 24 for the experiment—before iteration 24, uniform sampling from the mixed easy/hard dataset; after iteration 24, sampling only from hard problems. The curriculum approach outperforms the uniform baseline, which the paper attributes to the model developing foundational reasoning on easier problems before tackling harder ones.

Prioritized sampling. In addition to curriculum sampling, the paper uses prioritized sampling:

"We track the success rates $s_i$ for each problem $i$ and sample problems proportional to $1 - s_i$, so that problems with lower success rates receive higher sampling probabilities."

What this means: if a problem has success rate $s_i = 0.1$ (the model gets it right only 10% of the time), its sampling weight is $1 - 0.1 = 0.9$. If another problem has success rate $s_i = 0.9$, its weight is $1 - 0.9 = 0.1$. The sampling probability for problem $i$ in a batch is proportional to $1 - s_i$.

Why proportional to $1 - s_i$ rather than using a threshold: this creates a continuous prioritization where problems the model struggles with most get the most attention, but all problems have non-zero probability (unless $s_i = 1$). This maintains some exposure to problems the model already performs well on, preventing catastrophic forgetting, while focusing the majority of training budget on weak areas.

Why this helps: from an RL perspective, problems with low success rates have higher-variance gradients (some responses are correct, some incorrect, creating larger $r - \bar{r}$ values) and higher potential for improvement. Problems with very high success rates contribute almost no gradient signal because nearly all responses are correct (small $r - \bar{r}$). Prioritized sampling redirects compute toward the problems that actually drive learning.

Combined effect. Curriculum sampling provides a coarse, scheduled difficulty progression (easy first, then hard), while prioritized sampling provides fine-grained, dynamic difficulty weighting (within the current difficulty tier, focus on the hardest problems). The two strategies are complementary: curriculum determines when to introduce hard problems, and prioritization determines which hard problems to focus on.

---

3.4.5 Training Recipe Details: Reward Modeling, Test Cases, and Vision Data

The RL training depends on accurate reward signals. The paper details how rewards are generated for three domains: math (where answer equivalence is non-trivial), coding (where test cases may not be available), and vision (where data composition requires careful curation).

Reward modeling for math. Evaluating math answers is challenging because equivalent answers can have different surface forms (e.g., $a^2 - 4$ vs. $(a+2)(a-2)$). The paper develops two reward model approaches:

1. Classic RM: Following the InstructGPT (Ouyang et al., 2022) methodology, a value-head reward model is fine-tuned on approximately 800k data points. Input: question + reference answer + model's response. Output: a single scalar indicating correctness. The value head is a linear layer on top of the final hidden state that produces a scalar prediction.

2. Chain-of-Thought RM: Building on recent work (Ankner et al., 2024; McAleese et al., 2024), this RM first generates a step-by-step reasoning process analyzing whether the response matches the reference answer, then outputs a correctness judgment in JSON format. Also trained on approximately 800k CoT-labeled examples.

Accuracy comparison (from manual spot checks):

• Classic RM: approximately 84.4% accuracy
• Chain-of-Thought RM: approximately 98.5% accuracy

Why CoT RM performs better: by generating explicit reasoning about answer equivalence, the model can handle cases that require mathematical transformation to verify—e.g., recognizing that $\frac{2}{\sqrt{2}}$ equals $\sqrt{2}$, or that two different factored forms represent the same polynomial. The classic RM must make this judgment from a single embedding vector, which is a harder learning problem.

The paper adopts the Chain-of-Thought RM for RL training to ensure more correct feedback.

Test case generation for coding. Many coding problems from the web lack test cases, which are necessary to evaluate solution correctness during RL. The paper designs an automatic test case generation pipeline:

1. Focus: Problems that do not require a special judge (standard input-output problems where correctness is determined by comparing program output to expected output). Ground truth solutions are assumed to be available.

2. Generator: The base Kimi k1.5 model generates test cases using the CYaRon library (a test case generation library). Input to the generator: the CYaRon usage statement + the problem description. Output: test cases (input-output pairs).

3. Validation procedure:

   • Generate 50 test cases per problem.
   • Randomly sample 10 ground truth submissions (correct solutions to the problem).
   • Run each test case against all 10 submissions.
   • A test case is deemed valid if at least 7 out of 10 submissions produce matching outputs (consensus ≥ 70%). This filters out ambiguous or incorrectly generated test cases.
   • A problem is included in the training set if at least 9 out of 10 submissions pass the entire set of selected (valid) test cases.

4. Statistics (from a sample of 1,000 online contest problems):

   • 614 problems do not require a special judge.
   • 463 test case generators produced at least 40 valid test cases.
   • 323 problems were ultimately included in the training set.

Why the 7/10 and 9/10 thresholds: the 7/10 threshold for individual test cases ensures that the test case has a clear expected output—if most correct solutions agree, the test case is well-defined. The 9/10 threshold for problem inclusion ensures that the full test suite correctly evaluates solution correctness—if most correct solutions pass all tests for a given problem, the test suite is reliable as a reward signal.

Vision data composition. The vision RL data is sourced from three categories:

1. Real-world data: Science questions across grade levels requiring graphical comprehension, location guessing tasks requiring visual perception and inference, and data analysis involving complex charts. These improve real-world visual reasoning.

2. Synthetic visual reasoning data: Artificially generated, including procedurally created images and scenes targeting specific visual reasoning skills—spatial relationships, geometric patterns, object interactions. Provides controlled environments for testing and an endless supply of training examples.

3. Text-rendered data: Textual content (documents, code snippets, structured data) converted into visual format (images). Purpose: ensure the model provides consistent responses whether input is pure text or text rendered as images (screenshots, photos). Enhances handling of text-heavy images.

Why text-rendered data matters: in real-world deployment, users may upload screenshots of problems rather than typing them. If the model was trained only on text inputs for math problems, it might fail when the same problem appears as an image. Text-rendered data forces the model to maintain modality-invariant reasoning—the answer should be the same regardless of whether the input arrives as text or as an image of text. This is a form of modality alignment that goes beyond standard vision-language training.

---

3.4.6 Long2Short: Context Compression for Short-CoT Models

The long-CoT model achieves strong reasoning performance but consumes many test-time tokens. The paper presents four methods to transfer the thinking priors from the long-CoT model to more token-efficient short-CoT models.

Model merging. This is the simplest method:

"We merge the two models by simply averaging their weights."

Given a long-CoT model with parameters $\theta_{\text{long}}$ and a short-CoT model with parameters $\theta_{\text{short}}$, the merged model has parameters $\theta_{\text{merged}} = (\theta_{\text{long}} + \theta_{\text{short}}) / 2$.

What this does operationally: this is weight-space interpolation. It requires no additional training. The merged model inherits some of the long-CoT model's reasoning capabilities while retaining some of the short-CoT model's conciseness.

Why averaging works (the paper's observation): model merging has been found useful for maintaining generalization ability (Yang et al., 2024). The paper discovers its effectiveness for token efficiency specifically—the merged model produces shorter responses than the pure long-CoT model while outperforming the pure short-CoT model. The mechanism is not fully understood, but likely involves the long-CoT model's weights encoding reasoning strategies that can be partially activated even without generating the full extended chain of thought.

Shortest rejection sampling. This method exploits natural length variation in the long-CoT model's outputs:

"This method samples the same question $n$ times (in our experiments, $n = 8$) and selects the shortest correct response for supervised fine-tuning."

For each training problem, the long-CoT model generates $n$ responses. Among those that are correct, the one with the fewest tokens is selected as the training target. The short-CoT model is then fine-tuned (SFT) on these shortest-correct-response pairs.

Why this works: the long-CoT model sometimes produces concise correct solutions even though its average response is verbose. By selecting only these concise correct examples, the SFT dataset teaches the short-CoT model to produce correct reasoning without unnecessary verbosity. This is essentially a form of data filtering that biases the training distribution toward brevity.

DPO (Direct Preference Optimization). This method uses pairwise preference data:

"The shortest correct solution is selected as the positive sample, while longer responses are treated as negative samples, including both wrong longer responses and correct longer responses (1.5 times longer than the chosen positive sample)."

Operational details:

1. For each training problem, the long-CoT model generates multiple responses.
2. The shortest correct response is designated as the chosen (positive) sample.
3. Longer responses are designated as rejected (negative) samples. Two types of negative samples are used:
   • Wrong longer responses: incorrect answers with longer CoT.
   • Correct but longer responses: correct answers where the response is at least 1.5 times the length of the chosen sample.
4. These chosen-rejected pairs form the preference data for DPO training (Rafailov et al., 2024).

Why include correct-but-long responses as negatives: this teaches the model that given a choice between two correct solutions—one concise and one verbose—the concise one is preferred. Without this signal, the model might learn to produce correct answers but remain verbose. The 1.5× threshold prevents penalizing responses that are only marginally longer (which might just be natural variation) while still steering the model toward conciseness.

Long2short RL. This is a dedicated RL phase applied after the standard RL training:

"After a standard RL training phase, we select a model that offers the best balance between performance and token efficiency to serve as the base model, and conduct a separate long2short RL training phase."

In this phase:

1. The length penalty from Section 2.3.3 is applied.
2. The maximum rollout length is significantly reduced to further penalize responses that exceed the desired length while possibly correct.

What distinguishes long2short RL from standard RL with length penalty: the key difference is the reduced maximum rollout length. In standard RL, the model can generate very long responses and then receive length penalties—but the exploration still happens within a long context window. In long2short RL, the model is physically prevented from generating beyond the reduced length limit. This forces the model to find reasoning strategies that fit within the token budget, essentially learning to compress its reasoning rather than simply being penalized for verbosity.

Results comparison (Figure 7). The paper compares all methods on token efficiency:

• k1.5-short w/ rl (long2short RL): achieves 60.8 on AIME 2024 with 3,272 average tokens.
• k1.5-shortest: achieves 88.2 on MATH500 with token counts comparable to other short models.
• All k1.5-series models (orange in Figure 7) demonstrate superior token efficiency compared to baseline models (blue).

The long2short RL method demonstrates the highest token efficiency among the proposed methods, suggesting that explicit RL training with length constraints is more effective than post-hoc methods (DPO, model merging, rejection sampling) for compressing reasoning capabilities.

Iterative vision. The paper suggests that long2short methods can be combined with long-CoT RL iteratively:

"it is possible to combine long2short methods with long-CoT RL in an iterative way to further increase token efficiency and extract the best performance out of a given context length budget."

The implied cycle: long-CoT RL → long2short compression → use compressed model as base for next round of long-CoT RL → further compression → repeat. Each cycle could push the Pareto frontier of performance vs. token efficiency outward.

---

3.4.7 RL Infrastructure: Partial Rollouts and Hybrid Deployment

The paper presents infrastructure innovations as key enablers of long-context RL at scale. These are not peripheral engineering details but central contributions that made the approach feasible.

Iterative synchronous RL framework (Figure 3a). The system operates in cycles, each consisting of:

1. Rollout phase: Rollout workers, coordinated by a central master, generate trajectories by running the current policy model on prompts from the training set. These trajectories are stored in a replay buffer that disrupts temporal correlations.

2. Training phase: Trainer workers sample from the replay buffer, compute rewards (via reward models and the code execution sandbox), calculate gradient updates using the policy optimization algorithm (Section 2.3.2), and update model weights.

3. Weight synchronization: Updated weights flow back to rollout workers for the next iteration.

The replay buffer's role: by storing trajectories across iterations, the buffer enables off-policy learning—the gradient computation in Equation 3 uses samples from the reference policy $\pi_{\theta_i}$, which may differ from the current policy $\pi_\theta$. The buffer also shuffles data to disrupt temporal correlations, which is standard practice in deep RL to reduce variance.

Partial rollouts (Section 2.6.2, Figure 3b). This is the key innovation for handling long-CoT trajectories:

"Partial rollouts is a key technique that effectively addresses the challenge of handling long-CoT features by managing the rollouts of both long and short trajectories."

The problem it solves: without partial rollouts, long trajectories would monopolize rollout workers. If one worker is generating a 128k-token response while others finish their short responses quickly, the system wastes GPU resources waiting for the slowest worker (the straggler problem). Partial rollouts break long trajectories into segments processed across multiple iterations.

Mechanism (operational description):

1. A fixed output token budget caps each rollout trajectory. If a trajectory reaches this limit before completion, it is truncated.
2. The unfinished portion is saved to the replay buffer with its current state (the partial CoT generated so far).
3. In the next iteration, the partial trajectory is continued from where it left off, using the updated policy. The previous segments (from iterations $n-m$ to $n-1$) are reused from the buffer without re-generation.
4. Only the current iteration's segment requires on-policy computation.

Why this works with the policy optimization algorithm: because the gradient in Equation 3 uses samples from the reference policy, it is off-policy compatible. A trajectory generated partially under $\pi_{\theta_{i-2}}$ and partially under $\pi_{\theta_{i-1}}$ can still be used for training at iteration $i$, because the importance weighting is computed against $\pi_{\theta_i}$. This compatibility between partial rollouts and off-policy learning is what makes the approach viable.

Computational savings: instead of rolling out the entire response from scratch each iteration, the system processes and stores segments incrementally. This significantly reduces redundant computation—segments that haven't changed (because they were generated under a similar policy) are reused.

Repeat detection: the partial rollout system also identifies repeated sequences in generated content and terminates them early. Detected repetitions can be assigned additional penalties, discouraging the model from generating redundant content (a common failure mode where the model loops on the same reasoning step).

Hybrid deployment (Section 2.6.3, Figure 4). The RL training process alternates between training and inference phases, and the paper developed a system to share GPUs between both workloads.

The phases:

1. Training Phase: Megatron (training framework) runs training. After completion, it offloads GPU memory and prepares to transfer weights.
2. Inference Phase: vLLM (inference framework) starts with dummy weights, receives the latest weights from Megatron via Mooncake (a KVCache-centric transfer system), and performs rollout generation. After rollout completion, all vLLM processes are halted.
3. Subsequent Training Phase: GPU memory is released from vLLM, Megatron onloads memory, and training resumes.

Deployment strategy: Kubernetes Sidecar containers share all available GPUs, collocating both training and inference workloads in one pod. The primary advantages:

• Prevents training nodes from idling while waiting for inference nodes (both workloads share the same devices).
• Training and inference can iterate independently with distinct deployed images.
• The architecture is not limited to vLLM—other inference frameworks can be integrated.

Checkpoint engine: a shim process that manages the vLLM lifecycle, exposing HTTP APIs for triggering operations. It coordinates:

• Weight conversion: Megatron checkpoints (with Pipeline Parallelism and Expert Parallelism) are converted to Hugging Face format (retaining only Tensor Parallelism) in shared memory.
• Weight transfer: Mooncake transfers checkpoints between peer nodes over RDMA (Remote Direct Memory Access).
• Process lifecycle: vLLM is terminated and restarted between phases (rather than attempting to fully offload GPU memory, which is challenging due to CUDA graphs, NCCL buffers, and NVIDIA driver allocations).

Timing: the system achieves less than one minute from training to inference phase transition, and approximately ten seconds for the reverse transition. This low overhead is critical for maintaining high GPU utilization across the iterative RL loop.

Code sandbox (Section 2.6.4). For coding problems, a secure execution environment evaluates model-generated code against test cases. Key optimizations:

• Crun as container runtime: reduces container startup times from 0.12s (Docker) to 0.04s.
• Cgroup reusing: pre-creates cgroups to avoid the bottleneck of creating/destroying them per container in high-concurrency scenarios.
• Disk usage optimization: overlay filesystem with an upper tmpfs layer for high-speed ephemeral storage.
• Throughput: maximum containers started per second on a 16-core machine: 120 (sandbox) vs. 27 (Docker).

These optimizations are essential because RL generates and evaluates thousands of code samples per iteration—container startup overhead would otherwise dominate the evaluation time.

4. Key Insights and Innovations

Innovation 1: Context Length as the Primary Scaling Dimension for RL-Based Reasoning

The paper's most conceptually distinctive move is reframing the problem of improving LLM reasoning through RL not as a question of better search algorithms or reward models—the focus of virtually all prior work on inference-time compute scaling (Snell et al., 2024; Wu et al., 2024; Yao et al., 2024)—but as fundamentally a question of context length scaling. This is not an incremental adjustment to existing RL pipelines; it is a redefinition of what the core scaling axis should be.

Prior to this work, the dominant assumption in the field was that improving reasoning required explicit mechanisms: tree search over solution paths (Yao et al., 2024), process reward models trained on human-labeled step-level data (Lightman et al., 2023), or value functions for credit assignment (Snell et al., 2024). The Kimi k1.5 paper challenges this assumption at its foundation. The key conceptual move appears in Section 2.3.1: since both thoughts and feedback can be represented as language sequences, and planning algorithms operate over flattened search histories, an auto-regressive model with a sufficiently long context window can implicitly perform the search through its own generated tokens. The number of tokens becomes analogous to the computational budget allocated to explicit search algorithms. This is a unification argument: it claims that what explicit tree search does through structured exploration of a branching state space, a long-context model can do through sequential token generation that encodes exploration, evaluation, and backtracking in natural language.

The significance of this reframing extends beyond the architecture choice. It changes what the field should optimize for. If context length is the primary bottleneck—not search algorithm design, not reward model quality—then the research agenda shifts from "how do we build better planning algorithms?" to "how do we scale context length during RL efficiently?" This is precisely what the paper's infrastructure contributions (partial rollouts, hybrid deployment) are designed to enable. The evidence supporting this claim comes from Figures 5 and 6, which demonstrate a strong correlation between response length and accuracy across multiple benchmarks, with more difficult benchmarks exhibiting steeper slopes. The paper explicitly states: "Our final run of k1.5 scales to 128k context length and observes continued improvement on hard reasoning benchmarks" (Section 3.3). The fact that performance continues to improve with context length—rather than plateauing—suggests that this scaling axis is not yet saturated, distinguishing it from model size scaling which shows diminishing returns.

This innovation's significance is both theoretical (it provides a unified view of planning and auto-regressive generation) and practical (it eliminates the need for complex multi-component systems). However, it is important to note that this is an empirical claim, not a proven equivalence: the paper does not theoretically prove that auto-regressive generation can replicate all tree search behaviors, only that the resulting performance matches or exceeds what explicit search methods achieve. Whether there exist reasoning problems that fundamentally require explicit branching rather than sequential token generation remains an open question.

---

Innovation 2: Negative Gradients Are Essential for Learning Complex Reasoning Strategies

The paper's comparison between its online mirror descent variant and ReST (Gulcehre et al., 2023) in Figure 10 yields a finding that is not merely a performance comparison but a diagnostic insight about what makes RL work for reasoning. ReST iteratively fits the best response sampled from the current model—it provides positive signal (increase probability of good outputs) but no negative signal (do not decrease probability of bad outputs). The paper's method applies negative gradients that actively penalize incorrect responses. The performance gap is substantial and consistent across all evaluated benchmarks (OMNI-MATH500, MATH500, AIME2024, AIMO2024, ChatGLMMath, GAOKAO, GPQA, and several subject-specific benchmarks).

This is a significant finding because prior work on ReST (Gulcehre et al., 2023) had shown that positive-only self-training was competitive with more complex RL methods in other domains. The paper explicitly notes this discrepancy: "the performance gap between ReST and other RL-based methods is not as pronounced in other domains." The implication is that long-CoT reasoning has unique optimization requirements that make negative gradients disproportionately valuable.

Why would negative gradients matter more for reasoning than for other tasks? The paper's argument (Section 2.3.2) is that learning to generate long CoT requires the model to learn what not to do: which reasoning paths lead to dead ends, which approaches waste tokens on unproductive exploration, which error patterns are common. Positive-only training can teach the model to reproduce successful reasoning traces, but it cannot teach the model to avoid failure modes because the model never receives a signal that a particular reasoning pattern is harmful. In contrast, negative gradients explicitly reduce the probability of responses that received zero reward, teaching the model a decision boundary between productive and unproductive reasoning strategies.

This finding has broader implications for the RL-for-reasoning research agenda. It suggests that the choice of policy optimization algorithm is not a minor implementation detail but a central design decision that determines whether the model can learn the full spectrum of reasoning behaviors. Methods that only reinforce successes may be fundamentally limited in their ability to teach error recovery, backtracking, and self-correction—precisely the behaviors that distinguish long-CoT reasoning from simple step-by-step prompting. The paper positions this as a "crucial" finding (Section 3.5), and the breadth of benchmarks where the gap appears strengthens the claim that this is a general phenomenon rather than a dataset-specific artifact.

---

Innovation 3: The Long2short Paradigm as Capability Compression

The long2short methods introduced in Section 2.4 represent a conceptual contribution that goes beyond the specific techniques (model merging, shortest rejection sampling, DPO, long2short RL). The innovation is the paradigm itself: the idea that reasoning capabilities developed through expensive long-context RL can be systematically compressed into models that generate much shorter CoTs while preserving most of the performance gain. This is not simply a distillation technique—it is a reframing of what long-CoT training achieves.

The standard view of long-CoT models treats their verbosity as an inherent cost of their improved reasoning: you get better answers, but you pay for it in tokens. The Kimi k1.5 paper challenges this framing by demonstrating that the reasoning capability and the reasoning verbosity are partially separable. The long-CoT RL phase develops the capability—the model learns planning, reflection, error identification, and backtracking. The long2short phase then compresses this capability into a more token-efficient form, effectively asking: "now that you know how to reason deeply, can you do it more concisely?"

Figure 7 provides the empirical evidence for this separability. The k1.5-short w/ rl model achieves 60.8 on AIME 2024 with only 3,272 average tokens, while the long-CoT model (k1.5-long) achieves its score with substantially more tokens. The entire k1.5 series (orange points in Figure 7) sits on a distinctly better Pareto frontier than baseline models (blue points)—for any given token budget, k1.5 models achieve higher accuracy, and for any given accuracy level, they use fewer tokens. This demonstrates that the compressed models are not simply trading off performance for efficiency; they are genuinely more capable per token.

The paradigm has significant practical implications. Long-CoT models, despite their strong performance, are expensive to deploy at scale—each query consumes thousands of additional tokens, increasing latency and cost. If the reasoning capabilities can be compressed into short-CoT models with minimal performance loss, the practical benefits of RL training become accessible for production deployments. The paper's suggestion that long2short and long-CoT RL could be combined iteratively (compress → retrain with RL → compress further) points toward a self-reinforcing cycle where each iteration pushes the capability-efficiency frontier outward.

This innovation matters because it addresses the most obvious criticism of long-CoT approaches: that they are impractical for real-world use. By demonstrating that the verbosity is not an inseparable property of the improved reasoning but can be compressed away, the paper makes the case that long-context RL is not just a research curiosity but a viable path to deployable reasoning improvements.

---

Innovation 4: Final-Answer-Only Reward as a Feature, Not a Limitation

The paper's decision to use only final-answer correctness as the reward signal—without step-level rewards, process reward models, or value functions—is presented not as a simplification to reduce engineering complexity but as a deliberate design choice that enables better exploration and learning of error recovery. This inverts the conventional wisdom in the reasoning literature.

Prior work on improving LLM reasoning has invested heavily in process supervision. Lightman et al. (2023) demonstrated that step-level human feedback labels produced better reasoning than outcome-level labels. Process reward models (PRMs) trained to evaluate intermediate steps became a standard tool (Snell et al., 2024; Wu et al., 2024). The implicit assumption was that more granular feedback is always better for credit assignment—if you can tell the model exactly which step went wrong, it will learn faster and more accurately.

The Kimi k1.5 paper challenges this assumption with a specific argument (Section 2.3.2): step-level credit assignment via value functions would actively penalize the exploration that generates diverse reasoning paths. If the model takes a wrong turn but later recovers and reaches the correct answer, a value function would assign negative advantage to that wrong turn, discouraging the model from ever making that mistake again. But from a learning perspective, the experience of making the mistake and recovering is precisely what teaches the model error identification and backtracking. The paper states:

"exploring $z'_{t+1}$ is extremely valuable for training the model to generate long CoT... the model can learn the pattern of trial and error from taking $z'_{t+1}$ as long as it successfully recovers and reaches the correct answer."

This is a fundamentally different philosophy of what RL should optimize for. The goal is not to maximize training accuracy—which step-level rewards would help with—but to develop generalizable problem-solving strategies that transfer to unseen problems. Final-answer-only rewards create an incentive structure where the model is free to explore diverse approaches, including those that initially go wrong, because the only thing that matters is whether it eventually gets the right answer. This freedom to explore is what enables the emergence of planning, reflection, and backtracking behaviors.

The paper's ablation comparing against ReST (Figure 10) provides indirect support for this claim: positive-only training (ReST) underperforms the method that applies both positive and negative gradients. But the more fundamental claim—that final-answer-only rewards are better than step-level rewards for learning reasoning strategies—is not directly tested through an ablation comparing outcome rewards against process rewards. The argument remains a hypothesis grounded in the paper's design philosophy rather than an empirically verified superiority. This is a notable gap: the paper does not train a comparison model with process reward models to demonstrate that the final-answer-only approach actually produces better generalization. The claim should be understood as a well-motivated design choice with supporting reasoning, rather than a proven advantage.

Nevertheless, the innovation at the conceptual level is significant: it reframes the absence of process supervision not as a limitation to be overcome but as an enabling condition for the kind of exploration that produces robust reasoning strategies. If correct, this has major implications for how RL training pipelines should be designed—suggesting that efforts to build better process reward models may be misdirected, and that the field should instead focus on creating environments where models can explore freely and learn from outcomes.

---

Innovation 5: Difficulty-Adaptive Training Through Online Success Rate Tracking

The paper's curriculum and prioritized sampling strategies (Section 2.3.4) are not merely efficiency improvements—they represent a dynamic, model-aware approach to training data selection that contrasts with the static data mixing strategies used in most LLM training pipelines.

In standard supervised fine-tuning and even most RL fine-tuning work, the training data distribution is fixed in advance: practitioners decide on a mixture of domains and difficulty levels, and the model trains on samples drawn uniformly from that mixture. This treats all training examples as equally valuable at all stages of training. The Kimi k1.5 paper challenges this by making difficulty relative to the model's current capabilities rather than an intrinsic property of the problem. The difficulty of a problem is measured by the model's own success rate on it ($s_i$), which evolves during training. This creates a feedback loop: as the model improves on certain problems, those problems naturally receive lower sampling priority, redirecting compute toward problems where the model still struggles.

The curriculum sampling strategy takes this further by introducing a temporal dimension: easy problems first, hard problems later. The Figure 9 ablation shows that this scheduled progression outperforms uniform sampling from the mixed dataset. The paper attributes this to the practical reality that "spending a restricted computation budget on very hard problems often yields few correct samples, resulting in lower training efficiency" during early training. But the deeper insight is that the model needs to develop foundational reasoning capabilities before it can productively learn from its failures on hard problems. Presenting a novice model with competition-level math problems produces mostly incorrect responses with near-zero reward for all samples, generating no useful gradient signal. The same problems become productive training examples once the model has developed basic reasoning competence.

This approach has intellectual connections to curriculum learning in machine learning (Bengio et al., 2009) and to automatic curriculum generation in RL (Florensa et al., 2017), but applies these ideas in the specific context of LLM reasoning with online success rate tracking. The innovation is the synthesis: using the model's own evolving performance as both the difficulty metric and the sampling weight, creating an adaptive training distribution that continuously focuses on the frontier of the model's capabilities. This is a form of automatic.zone of proximal development—the model trains most heavily on problems just beyond its current reach, where learning is most efficient.

The significance extends beyond the specific implementation. It suggests a general principle for RL training of LLMs: the optimal data distribution is dynamic and model-dependent, not static and dataset-dependent. Future work could extend this principle to more continuous difficulty metrics, multi-dimensional difficulty (e.g., separating "requires more knowledge" from "requires deeper reasoning"), or difficulty estimation without requiring repeated sampling (which currently adds computational overhead).

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on multiple benchmarks spanning text, reasoning, and vision modalities. Text benchmarks: MMLU (Hendrycks et al., 2020; 57 subjects across STEM, humanities, social sciences), IF-Eval (Zhou et al., 2023; 500+ prompts with verifiable instructions), CLUEWSC (Xu et al., 2020; Chinese coreference resolution), C-EVAL (Huang et al., 2023; 13,948 multiple-choice questions across 52 disciplines). Reasoning benchmarks: HumanEval-Mul (Cassano et al., 2022; HumanEval translations across 8 programming languages), LiveCodeBench (Jain et al., 2024; contamination-free coding benchmark with live updates, v4 for short-CoT, v5 for long-CoT), Codeforces (online judge platform, percentile of ELO rating), AIME 2024 (competition math problems), MATH-500 (Lightman et al., 2023; 500 problems across algebra, calculus, probability, etc.). Vision benchmarks: MMMU (Yue et al., 2024; 11.5K multimodal questions from college exams across 6 academic fields), MATH-Vision (Wang et al., 2024; 3,040 math problems with visual contexts from real competitions), MathVista (Lu et al., 2023; mathematical and visual reasoning tasks requiring fine-grained visual understanding and compositional reasoning). The paper does not specify a single held-out test split for internal RL training—the benchmarks serve as the evaluation set—so there is no explicit cross-validation protocol described for the main results.

• Base model(s). The Kimi k1.5 base model is a Transformer decoder variant with multimodal capabilities (vision + language), pretrained on a diverse multilingual and multimodal corpus (English, Chinese, Code, Mathematics & Reasoning, Knowledge domains for text; Captioning, Image-text Interleaving, OCR, Knowledge, QA for vision). The model architecture supports interleaved images and text as input. For the scaling analysis in Section 3.3 and Section 3.5, the paper uses two model sizes—"Small Size" and "Large Size"—trained on the same dataset to study the interaction between model scale and context length scaling. The paper states the base model is "representative of contemporary LLMs" but does not disclose exact parameter counts. The long-CoT model is obtained through the full training pipeline (pretraining → vanilla SFT → long-CoT SFT → RL). The short-CoT model is obtained from the long-CoT model via long2short methods.

• Metrics. For math benchmarks (MATH-500, AIME 2024), the metric is Exact Match (EM) or Pass@1—the fraction of problems for which the model's final answer matches the ground truth. For coding benchmarks (HumanEval-Mul, LiveCodeBench), Pass@1 measures whether the generated code passes all test cases. For Codeforces, the metric is percentile of ELO rating, computed using majority voting on generated code snippets with model-generated test cases. For MMLU, C-EVAL, and MMMU, the metric is Exact Match on multiple-choice questions. For IF-Eval, the metric is Prompt Strict accuracy—the fraction of prompts where all verifiable instructions are followed. For CLUEWSC, the metric is Exact Match on coreference decisions. For vision benchmarks (MathVista, MATH-Vision), Pass@1 measures correctness on visual reasoning tasks. The AIME 2024 short-CoT result is reported as "averaged over 8 runs" (Section 3.4).

• Baselines. The paper compares against multiple frontier models, both open-source and proprietary. For long-CoT (Table 2): QwQ-32B Preview, OpenAI o1-mini, QVQ-72B Preview, OpenAI o1. For short-CoT (Table 3): Qwen2.5 72B-Instruct, LLaMA-3.1 405B-Instruct, DeepSeek V3, Qwen2-VL, Claude-3.5-Sonnet-1022, GPT-4o-0513. For the long2short analysis (Figure 7), baseline comparisons include Claude 3.5, DeepSeek V3, GPT-4o-0513, and Qwen2.5-72B-Inst. For the RL ablation studies, the primary baseline is ReST (Gulcehre et al., 2023)—a self-training method that iteratively fits the best response without applying negative gradients to incorrect responses. For curriculum sampling (Figure 9), the baseline is uniform sampling from the mixed easy/hard dataset without curriculum adjustments.

• Generation budget / compute accounting. The paper does not use a formal "generation budget" framework analogous to the Snell et al. (2024) paper's analysis of test-time compute scaling. Instead, the relevant axes of comparison are: (1) model size (Small vs. Large in Figure 8), (2) context length / response length (the primary scaling dimension studied in Figures 5, 6, and 8), (3) training iterations (Figures 5, 9, 10), and (4) token efficiency at inference (Figures 7 and 8, where accuracy is plotted against mean response length in tokens). For training compute, the paper describes infrastructure optimizations (partial rollouts, hybrid deployment) but does not report total FLOPs, GPU-hours, or training compute budgets for the RL phase. The long2short comparison in Figure 7 is the only place where inference-time token cost is explicitly quantified and plotted as an independent variable: the x-axis shows mean token length, and the y-axis shows accuracy, creating a Pareto frontier of token efficiency.

• Cross-validation / statistical protocol. The paper does not describe a cross-validation or train/test split for the RL prompt set beyond the curation process in Section 2.1. For the main benchmark results in Tables 2 and 3, the model is evaluated on the standard test sets of each benchmark. The AIME 2024 short-CoT result specifies averaging over 8 runs. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any result. The Codeforces percentile uses majority voting but does not specify the number of samples. For the internal training metrics (Figures 5, 9, 10), the shaded areas in Figure 5 represent the 95th percentile of response length. The paper acknowledges that the IFEval score in Table 3 "derived from an intermediate model" and states it will update scores based on the final model.

Main Quantitative Results

Long-CoT Model Performance

The long-CoT model achieves state-of-the-art results across all evaluated benchmarks, matching or exceeding OpenAI's o1 (Table 2). Headline numbers:

• AIME 2024: 77.5 Pass@1 (vs. 74.4 for o1, 63.6 for o1-mini, 50.0 for QwQ-32B Preview). This represents a +3.1 point absolute improvement over o1.
• MATH-500: 96.2 Exact Match (vs. 94.8 for o1, 90.6 for QwQ-32B, 90.0 for o1-mini). This is +1.4 points above o1.
• Codeforces: 94th percentile (tied with o1, vs. 88 for o1-mini, 62 for QwQ-32B). This matches o1's performance exactly.
• LiveCodeBench: 62.5 Pass@1 (vs. 67.2 for o1, 53.1 for o1-mini, 40.6 for QwQ-32B). This is -4.7 points below o1 but +9.4 above o1-mini.
• MathVista: 74.9 Pass@1 (vs. 71.0 for o1, 71.4 for QVQ-72B Preview). This is +3.9 points above o1.
• MMMU: 70.0 Pass@1 (vs. 77.3 for o1, 70.3 for QVQ-72B Preview). This is -7.3 points below o1 but roughly matches QVQ-72B.
• MathVision-Full: 38.6 Pass@1 (vs. 35.9 for QVQ-72B Preview; o1 score not reported). This is +2.7 points above QVQ-72B.

The pattern is that k1.5 long-CoT is competitive with or exceeds o1 on math-heavy reasoning benchmarks (AIME, MATH-500, MathVista) and coding competitions (Codeforces), while trailing on the multimodal understanding benchmark MMMU and on LiveCodeBench. The MMMU gap (-7.3 vs. o1) is notable and suggests that the vision-language joint training may not fully match o1's multimodal capabilities despite the paper's emphasis on multimodal data incorporation.

Short-CoT Model Performance

The short-CoT model achieves state-of-the-art results among models of comparable type, substantially outperforming GPT-4o and Claude 3.5 Sonnet on reasoning benchmarks (Table 3). Headline numbers:

• AIME 2024: 60.8 Pass@1 (vs. 9.3 for GPT-4o, 16.0 for Claude 3.5 Sonnet, 23.3 for Qwen2.5 72B-Inst, 39.2 for DeepSeek V3). This is a +550% relative improvement over GPT-4o and +280% over Claude 3.5 Sonnet, as claimed in the abstract. The comparison against DeepSeek V3 (+21.6 points absolute, +55% relative) is also substantial.
• MATH-500: 94.6 Exact Match (vs. 74.6 for GPT-4o, 78.3 for Claude 3.5 Sonnet, 80.0 for Qwen2.5 72B-Inst, 90.2 for DeepSeek V3). This is +20.0 points above GPT-4o and +4.4 points above DeepSeek V3.
• LiveCodeBench: 47.3 Pass@1 (vs. 33.4 for GPT-4o, 36.3 for Claude 3.5 Sonnet, 31.1 for Qwen2.5 72B-Inst, 40.5 for DeepSeek V3). This is +13.9 points above GPT-4o and +6.8 above DeepSeek V3.
• MMLU: 87.4 EM (vs. 87.2 for GPT-4o, 88.3 for Claude 3.5 Sonnet, 85.3 for Qwen2.5 72B-Inst, 88.5 for DeepSeek V3). Performance is roughly at parity—within ~1 point of all baselines except Qwen2.5.
• MMMU: 68.0 Pass@1 (vs. 69.1 for GPT-4o, 66.4 for Claude 3.5 Sonnet, 64.5 for Qwen2-VL). Slightly below GPT-4o (-1.1) but above Claude 3.5 Sonnet (+1.6).
• MathVista: 70.1 Pass@1 (vs. 63.8 for GPT-4o, 65.3 for Claude 3.5 Sonnet, 69.7 for Qwen2-VL). Above all baselines, +6.3 over GPT-4o.

The short-CoT model's strength is concentrated on reasoning benchmarks where the gap over frontier models is dramatic (AIME, MATH-500, LiveCodeBench), while performance on general knowledge and understanding benchmarks (MMLU, MMMU, MathVista) is at parity or slightly ahead of baselines. This is consistent with the paper's narrative: RL training specifically enhances reasoning capabilities, and the long2short methods successfully distill these reasoning gains into token-efficient form. The +550% claim from the abstract specifically refers to the AIME comparison against GPT-4o (60.8 vs. 9.3).

A notable pattern in the short-CoT results is that DeepSeek V3 is the strongest baseline on reasoning tasks among comparison models (39.2 on AIME, 90.2 on MATH-500, 40.5 on LiveCodeBench). The k1.5 short-CoT's advantage over DeepSeek V3 is significant but less dramatic than over GPT-4o or Claude 3.5 Sonnet. This suggests that DeepSeek V3 may incorporate some form of reasoning-specific training (the paper does not speculate on this), making it the most competitive baseline.

Long Context Scaling: Response Length and Performance

The paper's central scaling claim is that context length is the primary dimension along which RL reasoning performance improves. Figures 5 and 6 provide the empirical evidence.

Figure 5 tracks training accuracy and response length across training iterations for 12 internal benchmarks (total, OMNI-MATH500, MATH500, AIMO2024, AIME2024, ChatGLMMath, GAOKAO, GPQA, Biology, Chemistry, Physics, KAOYAN) using a mid-sized model. Key observations:

• Across all benchmarks, both accuracy and mean response length increase concurrently as training progresses through ~150 iterations.
• The rate of increase differs by benchmark difficulty: AIME2024 and AIMO2024 show the steepest increases in both accuracy (from near zero to ~0.4-0.5) and token length (from ~5,000 to ~20,000-30,000 tokens). MATH500, which is easier, starts at higher accuracy (~0.8) and shows more moderate length growth (from ~2,000 to ~10,000 tokens).
• The shaded area (95th percentile of response length) widens substantially for harder benchmarks, indicating that difficult problems elicit more variable-length responses—some problems require very long CoTs while others can be solved more concisely.
• The GAOKAO benchmark (Chinese college entrance exam) and ChatGLMMath show patterns similar to MATH500: high initial accuracy with moderate length growth.

Figure 6 formalizes the correlation between response length and accuracy by plotting accuracy against mean token length for each training checkpoint, with a trend line and slope for 8 benchmarks (total, OMNI-MATH500, MATH500, AIMO2024, AIME2024, ChatGLMMath, GAOKAO, GPQA). The slopes (reported in scientific notation as slope: X.XXe-05) represent the percentage-point accuracy gain per additional token of mean response length. Key observations:

• All slopes are positive, confirming that longer responses correlate with higher accuracy across all benchmarks.
• The steepest slopes are for the hardest benchmarks: GPQA (4.24e-05), AIME2024 (3.40e-05), AIMO2024 (3.33e-05), OMNI-MATH500 (3.05e-05). This means that on the hardest problems, additional tokens provide the largest accuracy gains per token.
• The shallowest slopes are for easier benchmarks: MATH500 (1.36e-05), GAOKAO (1.49e-05). Easier problems benefit less from additional tokens because the model already solves most of them correctly with shorter responses.
• The total benchmark (first panel) shows a slope of 2.46e-05, representing the average across all difficulty levels.

The paper states in Section 3.3: "Our final run of k1.5 scales to 128k context length and observes continued improvement on hard reasoning benchmarks." This claim is supported by the trend in Figure 5 showing no plateau in accuracy or length at the end of training (iteration ~150), and by Figure 6 showing roughly linear relationships between length and accuracy without obvious diminishing returns. However, the paper does not present a direct ablation showing performance at different maximum context lengths (e.g., 16k vs. 32k vs. 64k vs. 128k) to isolate context length scaling from other training dynamics.

Model Size vs. Context Length Scaling

The paper addresses whether scaling context length through RL can substitute for scaling model size. Figure 8 plots accuracy vs. mean response length for Small and Large model variants on OMNI-MATH500, AIME2024, MATH500, and AIMO2024, with trend lines for each size. Key observations:

• Large model initially outperforms Small model at shorter response lengths across all benchmarks. For example, on AIME2024 at ~2,000 tokens, Large achieves ~0.20 accuracy vs. Small at ~0.12.
• Small model can match Large model's performance by generating longer responses. On OMNI-MATH500, Small at ~4,500 tokens achieves accuracy comparable to Large at ~3,500 tokens. On MATH500, Small at ~5,000 tokens reaches accuracy comparable to Large at ~3,000 tokens.
• Large model has better token efficiency: The slopes for Large are steeper than for Small across all benchmarks. For example, on AIME2024: Large slope = 8.10e-05 vs. Small slope = 5.90e-05; on AIMO2024: Large slope = 6.84e-05 vs. Small slope = 3.25e-05. This means the Large model gets more accuracy improvement per additional token of response length.
• Large model has a higher performance ceiling: At the maximum response lengths shown, Large achieves higher absolute accuracy than Small on all benchmarks except MATH500 (where they converge at ~0.90-0.925).

The paper interprets these results as showing that "if one targets the best possible performance, scaling the context length of a larger model has a higher upper bound and is more token efficient. However, if test-time compute has a budget, training smaller models with a larger context length may be viable solutions" (Section 3.5). This is a nuanced finding: context length scaling and model size scaling are partially substitutable but not fully equivalent—larger models extract more value from additional context.

Long2Short: Token Efficiency Results

Figure 7 presents the long2short comparison on MATH500 and AIME2024, plotting accuracy against average token length for multiple k1.5 variants and baseline models. Key observations:

• All k1.5 variants (orange) occupy a superior Pareto frontier compared to all baseline models (blue). For any given token budget, k1.5 variants achieve higher accuracy; for any target accuracy, they use fewer tokens.
• k1.5-short w/ rl (long2short RL) achieves the best token efficiency among k1.5 variants: 60.8 on AIME2024 with 3,272 average tokens, and ~92.5 on MATH500 with ~800 tokens.
• k1.5-shortest achieves the highest MATH500 accuracy (~94) with modest token usage (~500-600 tokens), but underperforms k1.5-short w/ rl on AIME2024 (~48 vs. 60.8).
• k1.5-long achieves the highest absolute performance on both benchmarks but requires substantially more tokens (~1,200 tokens for MATH500, ~5,000 tokens for AIME2024).
• k1.5-short w/ dpo, k1.5-short w/ merge, and k1.5-short w/ merge + rs occupy intermediate positions on the Pareto frontier—better than baselines but not as efficient as long2short RL.
• Baseline models (GPT-4o, Claude 3.5 Sonnet, DeepSeek V3, Qwen2.5-72B-Inst) cluster at lower token counts but also lower accuracy. GPT-4o achieves ~75 on MATH500 with ~400 tokens and ~9 on AIME2024 with ~400 tokens. DeepSeek V3 achieves ~90 on MATH500 with ~800 tokens and ~39 on AIME2024 with ~800 tokens.

The quantitative gap between k1.5-short w/ rl and baselines is stark: on AIME2024, k1.5-short w/ rl (60.8 at 3,272 tokens) vs. GPT-4o (9.3 at ~400 tokens) represents a 6.5× accuracy improvement at 8× the token cost. The paper's +550% claim refers to the accuracy ratio, not token-normalized performance. The Pareto frontier representation in Figure 7 makes it clear that k1.5 models genuinely dominate in token efficiency—they achieve accuracies that baselines cannot reach at any token budget.

Ablation: RL vs. ReST (Figure 10)

The paper compares its online mirror descent variant against ReST across 12 benchmarks (OMNI-MATH500, MATH500, AIMO2024, AIME2024, ChatGLMMath, GAOKAO, GPQA, k12-biology, k12-chemistry, k12-physics, KAOYAN, Total). Key observations:

• The paper's method ("Ours", blue) consistently outperforms ReST (orange) across all benchmarks and throughout training (0-50 steps).
• The gap is largest on the hardest benchmarks: AIME2024 shows a substantial divergence where Ours reaches ~0.40 accuracy while ReST plateaus at ~0.25; AIMO2024 shows Ours at ~0.30 vs. ReST at ~0.25.
• On easier benchmarks (MATH500, GAOKAO), the gap is present but narrower. MATH500: Ours reaches ~0.90 vs. ReST at ~0.87. GAOKAO: Ours reaches ~0.88 vs. ReST at ~0.86.
• On GPQA, the gap emerges early and widens: Ours reaches ~0.24 vs. ReST at ~0.12 by step 50.
• The "Total" panel (aggregated across all benchmarks) shows Ours reaching ~0.64 vs. ReST at ~0.52 by step 50—a 12 percentage point gap.

The paper interprets this as evidence that "negative gradients markedly enhance the model's efficiency in generating long CoT" and that "the choice of policy optimization algorithm is crucial." The consistency across all 12 benchmarks strengthens this claim—this is not a benchmark-specific artifact but a robust pattern. However, the paper does not report the ReST implementation details (temperature, number of samples per iteration, how the "best response" is selected), which makes it difficult to assess whether the comparison is fair. ReST's performance could depend on these hyperparameters.

Ablation: Curriculum Sampling (Figure 9)

The paper compares curriculum sampling (uniform sampling from full dataset for 24 iterations, then only hard problems) against a uniform sampling baseline. The result is shown on an unnamed benchmark ("Accuracy" on y-axis, iteration on x-axis, with the transition at iteration 24 marked by a vertical dashed line). Key observations:

• Pre-transition (iterations 0-24): both methods follow similar trajectories, starting around 0.33 and rising to ~0.40 (curriculum) and ~0.38 (baseline). The gap is small during this phase.
• Post-transition (iterations 24-40): the curriculum method accelerates sharply, reaching ~0.62 at iteration 40, while the baseline continues a slower climb to ~0.52. The gap widens from ~0.02 at iteration 24 to ~0.10 at iteration 40.
• The slope change at iteration 24 for the curriculum method is visually obvious—the curve bends upward more sharply compared to the baseline.

The paper states this improvement can be attributed to the method's ability to "progressively challenge the model, allowing it to develop a more robust understanding and competency in handling complex problems" (Section 3.5). A caveat: the result is shown on a single benchmark and a single curriculum schedule (transition at iteration 24). The paper does not ablate the transition point or compare different curriculum schedules, so the optimality of this specific schedule is not established.

Critical Assessment

The experimental analysis provides strong support for the paper's central empirical claims but has several important limitations that constrain the generality and confidence of the conclusions.

Claim: Long context scaling during RL produces continued improvement on hard reasoning benchmarks.

The evidence for this claim comes primarily from Figures 5 and 6, which show concurrent growth in accuracy and response length across training iterations. The correlation is clear and consistent across 12 benchmarks. However, what the experiments demonstrate is a correlation between response length and accuracy during RL training, not necessarily a causal effect of context length on reasoning capability. There are alternative interpretations of the data:

1. Selection effect: Longer responses may be more accurate not because length enables better reasoning, but because problems that the model happens to solve correctly also happen to elicit longer responses (reverse causality). The model may be generating longer responses because it is engaging more deeply with problems it partially understands, rather than the length itself causing the accuracy.

2. Confounding by training iteration: Both accuracy and length increase with training iteration. The length-accuracy correlation could be partially driven by a third variable (the model's general improvement through RL) rather than a direct causal link between length and accuracy.

The paper does not present the most direct test of this claim: an experiment where the same model checkpoint is evaluated at different forced context lengths (e.g., by varying the maximum generation length at inference time) to isolate the effect of length on accuracy. The scaling analysis in Figure 5 uses checkpoints from different training iterations, where many factors beyond length are changing (model weights, training data distribution, optimizer state). The Figure 8 comparison between Small and Large models is more informative because it shows length-accuracy relationships within each model size class, but it still uses different checkpoints.

A stronger test would be: take a fixed checkpoint, generate responses at different maximum lengths, and measure accuracy. If longer maximum lengths produce higher accuracy from the same model, that would directly demonstrate the causal effect of context length. The paper does not report such an experiment.

Moreover, the paper claims scaling to 128k context length but does not report what accuracy was achieved at 128k vs. at shorter lengths (e.g., 32k, 64k). The "continued improvement on hard reasoning benchmarks" is inferred from the non-saturation of training curves in Figure 5, not from a controlled length ablation.

Claim: The online mirror descent variant substantially outperforms ReST because negative gradients enable more efficient learning.

Figure 10 provides consistent evidence across 12 benchmarks. This is the most convincing ablation in the paper because the comparison is at the same training steps, using (presumably) the same model initialization and data. The performance gap is large and robust. However, several qualifications are necessary:

1. Implementation details are missing. The paper does not specify how ReST was implemented—what sampling temperature was used, how many responses were generated per problem, how the "best response" was selected, whether the ReST model was trained from the same initialization, or whether ReST used the same number of gradient steps per iteration. These implementation choices can substantially affect ReST's performance.

2. The gap may reflect an exploration-exploitation tradeoff. ReST's positive-only approach may simply explore less, which could be addressed by different sampling strategies (higher temperature, more samples) rather than being an inherent limitation of the algorithm class.

3. The claim that negative gradients are specifically valuable for learning error recovery behaviors (the conceptual argument in Section 2.3.2) is not directly tested. The paper does not analyze the generated CoTs from ReST vs. Ours to show that Ours exhibits more backtracking, error identification, or recovery patterns. The performance gap could be due to other factors (better exploration, more effective use of data, different effective learning rates).

Claim: The long2short methods transfer reasoning capabilities from long-CoT to short-CoT models with high token efficiency.

Figure 7 provides compelling evidence that k1.5 short-CoT variants occupy a superior Pareto frontier compared to baseline models. However, several aspects of this claim deserve scrutiny:

1. The baselines are not retrained with the same data. The k1.5 models benefit from the full Kimi training pipeline (pretraining data, SFT data, RL prompt set), which may be of higher quality or different composition than what the baseline models were trained on. The superior token efficiency could partially reflect better pretraining or SFT data rather than the specific long2short methods.

2. The long2short methods use the long-CoT model as a teacher. This means the short-CoT models are effectively distilled from a stronger model. A fairer comparison might involve training a short-CoT model directly on the same training data without the long-CoT teacher, to isolate the contribution of the distillation process from the contribution of the underlying training data and RL phase.

3. The iterative long2short vision (long-CoT RL → long2short → further long-CoT RL) is presented as a promising direction but is not experimentally validated. The paper does not show results from multiple cycles of this process.

Claim: Curriculum sampling improves training efficiency.

Figure 9 shows a clear benefit on a single benchmark with a single curriculum schedule (transition at iteration 24). The limitations:

1. Single benchmark, single schedule. The paper does not report curriculum sampling results on multiple benchmarks or with different transition points. The optimal transition point likely depends on the benchmark difficulty distribution and model size.

2. The baseline uses uniform sampling from the mixed dataset. A stronger baseline would be a non-curriculum method that also focuses on hard problems but without the easy-first warmup—e.g., prioritized sampling alone from the start. This would isolate the specific contribution of the temporal curriculum (easy → hard) from the contribution of simply training more on hard problems.

Overarching experimental limitations:

1. No confidence intervals or statistical testing. Almost none of the figures report error bars, confidence intervals, or p-values. For a paper making strong comparative claims (matching o1, outperforming GPT-4o by +550%), the absence of any statistical rigor is a significant weakness. The AIME 2024 short-CoT result mentions averaging over 8 runs, but no variance is reported. The main benchmark tables (Tables 2 and 3) report single numbers without any indication of measurement uncertainty.

2. No disclosure of model sizes or training compute. The paper does not report the parameter count of k1.5, the number of GPUs used, the total training FLOPs, or the wall-clock time of RL training. This makes it impossible to compare the efficiency of the approach against alternatives (e.g., how does RL training compute compare to the compute used to train the baseline models?). The "Small" and "Large" model sizes in Figures 5 and 8 are not quantified.

3. No test set contamination analysis. Given that the RL prompt set is curated from web sources and may overlap with benchmark test sets, the paper should report decontamination results. LiveCodeBench is specifically designed to be contamination-free through live updates, but the other benchmarks (AIME, MATH-500, MMLU) may have training set overlap that inflates results.

4. Single model family. All results are from the Kimi k1.5 model family. There is no evidence that the findings—particularly the context length scaling relationship and the effectiveness of the specific policy optimization algorithm—generalize to other model architectures or pretraining recipes.

5. The claims about not needing MCTS, value functions, or PRMs are not experimentally validated. The paper does not train a comparison model that does use MCTS, value functions, or PRMs to show that these techniques are unnecessary. The claim rests on the implicit comparison that k1.5's performance matches o1 (which may or may not use these techniques—its training method is undisclosed). Without a controlled ablation where these components are added and shown not to improve performance, the "simplistic framework" claim is more of a design philosophy than an empirically validated advantage.

6. The RL prompt set curation is critical but not ablated. Section 2.1 describes careful filtering to ensure diverse coverage, balanced difficulty, and accurate evaluability. The paper does not show what happens if these curation steps are omitted—how much does the easy-to-hack filtering matter? How much does the difficulty balancing matter? Without these ablations, the reader cannot assess whether the prompt set curation is a minor optimization or a critical enabler.

7. Missing comparison: long-CoT RL vs. explicit search with the same base model. The paper argues that long-context RL can substitute for explicit planning algorithms, but does not implement a planning-augmented baseline (e.g., Tree of Thoughts with the same base model) and compare it against the RL-trained model. Such a comparison would directly test the claim that the "simplistic framework" achieves what explicit search achieves.

In summary, the experimental results strongly support the claim that Kimi k1.5 achieves state-of-the-art reasoning performance through a training pipeline involving long-context RL and long2short compression. The evidence for why this works—that context length scaling rather than algorithmic complexity is the key factor, that negative gradients are essential, that final-answer-only rewards are sufficient—is suggestive but not rigorously isolated through controlled experiments. The paper's contributions are primarily in demonstrating what is possible (competitive performance without complex planning machinery) rather than in establishing why each component is necessary or optimal.

6. Limitations and Trade-offs

The Absence of Controlled Ablations Weakens the Central "Simplistic Framework" Claim

The paper's most prominent conceptual claim — that strong reasoning performance can be achieved without Monte Carlo tree search, value functions, or process reward models — rests entirely on an implicit comparison against undisclosed methods (OpenAI's o1) rather than on controlled experiments within the Kimi training pipeline. The paper never trains a comparison model that does use MCTS, value functions, or PRMs on the same base model, with the same data, at the same compute budget, and shows that these components fail to improve performance. The claim appears in the abstract and is reiterated throughout Section 2.3.1:

"we show that strong performance can be achieved without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models."

There is no experiment that supports this "without relying on" framing as a demonstrated advantage — the paper shows that the chosen approach works, not that the alternatives are unnecessary or inferior.

The consequence is that a practitioner reading this report cannot determine whether the simplified framework is actually preferable to more complex alternatives, or whether it simply suffices given the scale of training compute and data available to the Kimi team. It is possible that adding process reward models or explicit tree search to the same training pipeline would yield further improvements beyond the reported numbers — the paper provides no evidence either way. The "simplistic framework" narrative may lead practitioners to avoid techniques that could be beneficial in their own settings, particularly if they have access to fewer resources where efficient credit assignment (via value functions or PRMs) might matter more.

What evidence exists in the paper: The only comparison provided is external benchmarking against o1 (Table 2), which achieves comparable or slightly better/worse numbers depending on the benchmark. This tells us that k1.5 and o1 have similar final performance, not that the methods used to achieve that performance are equivalent or that the omitted techniques are unnecessary. The ablation comparing against ReST (Figure 10) tests a different hypothesis (positive-only vs. positive+negative gradients), not the value of MCTS or PRMs. The paper does not report a single experiment where PRM-guided search or value-function-based credit assignment is added to the k1.5 pipeline and measured against the baseline.

Mitigation status: The paper does not acknowledge this as a limitation. The "simplistic framework" is presented as an unqualified strength. Future work would need to implement the omitted techniques within the same infrastructure and compare at matched compute budgets to substantiate the claim.

---

Long-CoT RL Training Compute Is Undisclosed and Likely Enormous

The paper provides extensive detail on infrastructure design (Section 2.6) — the hybrid deployment system, partial rollouts, checkpoint engine, code sandbox — but discloses zero quantitative information about the computational cost of training. The reader is not told: the parameter count of k1.5, the number of GPUs used, the total training FLOPs, the wall-clock duration of RL training, the number of iterations in the final run, or the cost of the long-CoT SFT phase. The only hint about scale comes from the "Small" and "Large" model comparison (Section 3.5, Figure 8) and the mention of scaling to 128k context length, neither of which is quantified in absolute terms.

The consequence is that the paper's results are not reproducible and its efficiency claims are unevaluable. A practitioner cannot estimate whether the reported performance improvements are feasible within their compute budget. The field cannot compare the cost-effectiveness of long-context RL against alternative approaches (e.g., scaling pretraining, using explicit search with a larger model) because the cost of the RL phase is unknown. The paper's central narrative — that context length scaling during RL is a viable new axis for continued improvement — depends on the efficiency of that scaling, not just its existence. Without cost data, the reader cannot assess whether the context-length-to-accuracy relationship in Figures 5 and 6 represents an efficient use of compute or an extremely expensive one. The improvement from iteration 0 to iteration 150 could require days of training on thousands of GPUs or months — the paper provides no way to distinguish these scenarios.

What evidence exists in the paper: The paper provides extensive infrastructure details (partial rollouts in Section 2.6.2, hybrid deployment in Section 2.6.3, code sandbox performance in Section 2.6.4) that are described as key enablers, suggesting that the training is computationally intensive. The use of partial rollouts specifically addresses the problem of long trajectories monopolizing resources, implying that this was a practical bottleneck. The code sandbox reports container startup times (0.04 seconds) and throughput (120 containers/sec on 16 cores), which hints at the scale of code evaluation but does not translate to total compute. The training curves in Figure 5 span ~150 iterations for a "mid-sized model," but the duration of one iteration and the model size are unspecified.

Mitigation status: The paper does not acknowledge the absence of compute disclosure as a limitation. This is an unusual omission for a technical report that otherwise provides detailed methodology. A partial mitigation is the infrastructure description itself, which would allow a well-resourced team to build a similar system, but the lack of scale information means each team would need to independently discover the required compute budget through trial and error.

---

No Evidence That Long Context Scaling Is Causal Rather Than Correlational

The paper's title and central thesis position context length scaling as the key mechanism underlying performance improvement. Section 3.3 states:

"We scale the context window of RL to 128k and observe continued improvement of performance with an increased context length."

The supporting evidence (Figures 5 and 6) demonstrates a correlation between response length and accuracy across training iterations. However, the paper does not present the experiment that would establish causality: taking a single fixed model checkpoint, generating responses at different maximum context lengths, and measuring whether longer allowed responses produce higher accuracy. Without this, the observed correlation admits multiple interpretations:

• Length causes accuracy: The model uses additional tokens to perform more reasoning steps, plan, backtrack, and verify, leading to correct answers.
• Accuracy causes length: The model generates longer responses on problems it partially understands (engaging more deeply) and shorter responses on problems it finds confusing (giving up quickly), producing the correlation.
• A confounder (training progress) causes both: As RL training improves the model's general reasoning capability, it both solves more problems correctly and generates longer responses (perhaps because it has learned that longer reasoning is rewarded), producing a spurious correlation.

The consequence of this ambiguity is that the paper's central design recommendation — invest in scaling context length during RL — may be targeting the wrong mechanism. If the correlation is not causal, then the infrastructure investment in partial rollouts and long-context RL training (which the paper presents as its primary technical contribution) might be unnecessary — similar performance could perhaps be achieved through other means (better reward modeling, different optimization, more training iterations at shorter context lengths) without the complexity of 128k-context RL. A practitioner following the paper's approach might invest heavily in long-context infrastructure without clear evidence that the length itself — rather than the training process that happened to produce longer responses — is responsible for the gains.

What evidence exists in the paper: Figures 5 and 6 show within-training correlations. Figure 8 shows length-accuracy relationships for Small and Large model variants, but still uses different checkpoints across training. The paper does not report a single experiment where context length is the independent variable, manipulated at inference time for a fixed model. The statement about scaling to 128k observing "continued improvement" (Section 3.3) is an observation about training dynamics, not a controlled length ablation.

Mitigation status: The paper does not acknowledge this gap between correlational and causal evidence. The language throughout the paper treats the length-accuracy relationship as causal (e.g., "the number of thoughts serves as an analogy to the computational budget traditionally allocated to planning algorithms," Section 2.3.1). A simple experiment — evaluating a fixed checkpoint at maximum generation lengths of 4k, 8k, 16k, 32k, 64k, and 128k tokens — would directly test the causal claim and is conspicuously absent.

---

Difficulty Estimation Cost and the Online Nature of RL Sampling Overhead

The paper's curriculum and prioritized sampling strategies (Section 2.3.4) depend on knowing the difficulty of each training problem. Difficulty is measured by the model's success rate $s_i$, which requires sampling each problem multiple times to estimate. The paper also uses model-based difficulty assessment during prompt set curation (Section 2.1):

"for every prompt, an SFT model generates answers ten times using a relatively high sampling temperature. The pass rate is then calculated and used as a proxy for the prompt's difficulty"

This sampling is performed for every prompt in the training set. During RL training, success rates $s_i$ are tracked across iterations to compute sampling weights proportional to $1 - s_i$. While the paper frames these as improving training efficiency, it does not account for the cost of obtaining the difficulty estimates themselves. Generating 10 samples per prompt during curation, plus the ongoing sampling needed to maintain accurate success rate estimates during RL, represents a substantial computational overhead that is not factored into any reported efficiency metric.

The consequence is that the reported benefits of curriculum and prioritized sampling (Figure 9, and the general claim that these strategies improve training) are measured after the difficulty estimation cost has been paid, making the net efficiency gain unclear. In the extreme case, the cost of difficulty estimation could exceed the efficiency gain from adaptive sampling, making the strategy net-negative in total compute. This is analogous to the difficulty estimation problem identified in Snell et al. (2024), where generating 2048 samples per question to estimate difficulty cost more than the largest test-time compute budgets studied. The Kimi paper's approach is more efficient (10 samples for curation, online tracking during RL), but the total cost across potentially thousands of training problems and hundreds of iterations is still unquantified.

What evidence exists in the paper: The paper provides details on the difficulty estimation procedure (10 samples per prompt during curation, Section 2.1) and the success rate tracking mechanism (Section 2.3.4), but does not report the total computational cost of these procedures or compare training efficiency with and without the estimation overhead factored in. The curriculum sampling ablation (Figure 9) compares against uniform sampling but does not report the total compute used by each method including difficulty estimation.

Mitigation status: The paper does not acknowledge this accounting gap. The difficulty estimation procedures are presented as costless optimizations. A partial mitigation is that the online success rate tracking during RL is inherent to the training process (the model generates samples anyway, and success rates can be updated from training rollouts), but the initial curation cost (10 samples × number of curated prompts) is an additional, unaccounted expense.

---

Evaluation Lacks Statistical Rigor, Jeopardizing the +550% and o1-Matching Claims

The paper makes strong comparative claims — "outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%)" (abstract) and "matching OpenAI's o1" (abstract) — based on single-number benchmark results without any reported measure of statistical uncertainty. Across Tables 2 and 3, Figures 5-10, and all ablation studies, the paper reports no confidence intervals, standard deviations, standard errors, or p-values. The only exception is the AIME 2024 short-CoT result, which notes "averaged over 8 runs" (Section 3.4) but does not report the variance across those runs.

The consequence is that the reader cannot assess whether the reported performance differences are statistically reliable or within the range of sampling noise. This is particularly concerning for:

• The +550% claim on AIME 2024: 60.8 vs. 9.3 is a dramatic difference, but AIME 2024 has only 30 questions. A difference of a few correct answers substantially changes the percentage. If k1.5 short-CoT's true Pass@1 is 60.8 ± 5 (95% CI: 51-71) and GPT-4o's is 9.3 ± 5 (95% CI: 0-19), the gap is large but the precise multiplier is highly uncertain.
• The "matching o1" claim: On MATH-500, 96.2 vs. 94.8 is a 1.4-point difference. With a 500-question test set, this represents 481 vs. 474 correct answers — a difference of 7 questions. Without confidence intervals, it is unclear whether this gap is statistically significant or within sampling error.
• The ablation comparisons (Figures 9 and 10): The curriculum sampling ablation (Figure 9) shows a final gap of ~0.10 on an unspecified benchmark with an unspecified number of test questions. The statistical reliability of this gap is unknown.

What evidence exists in the paper: The paper reports only point estimates. The shaded area in Figure 5 represents the 95th percentile of response length, not confidence intervals for accuracy. No other figure reports variability. The benchmark descriptions in Appendix C provide test set sizes (AIME 2024: unspecified but typically 30 questions; MATH-500: 500; MMLU: ~14,000; MMMU: 11,500; LiveCodeBench: unspecified), which allow the reader to roughly estimate binomial confidence intervals, but the paper does not perform this analysis.

Mitigation status: The paper does not acknowledge the absence of statistical reporting as a limitation. This is a weakness shared with many LLM technical reports, but it is particularly consequential here because the paper's primary contribution is a comparative claim (matching o1, outperforming GPT-4o) rather than an absolute capability demonstration. A partial mitigation is that the consistency across multiple benchmarks — k1.5 outperforms baselines on reasoning tasks across AIME, MATH-500, LiveCodeBench, and Codeforces — makes it unlikely that all results are statistical flukes. However, the specific magnitude of improvement and the claim of matching o1 on individual benchmarks remain uncertain.

---

The Long2short Comparison Does Not Control for Training Data and Compute Advantages

The long2short token efficiency analysis (Figure 7) shows that k1.5 short-CoT variants achieve substantially better accuracy-token tradeoffs than GPT-4o, Claude 3.5 Sonnet, DeepSeek V3, and Qwen2.5-72B-Inst. The paper presents this as evidence that the long2short methods (model merging, shortest rejection sampling, DPO, long2short RL) effectively transfer reasoning capabilities from long-CoT to short-CoT models. However, the comparison does not control for two critical confounding factors:

1. Training data differences. The k1.5 models benefit from the full Kimi training pipeline, including a curated pretraining corpus (Appendix B), a multi-stage SFT dataset (~1M text + 1M vision examples, Section 2.5.2), a carefully filtered RL prompt set (Section 2.1), and the long-CoT SFT warmup dataset (Section 2.2). The baseline models were trained on different, non-overlapping datasets. The superior token efficiency of k1.5 short-CoT could partially reflect higher-quality pretraining or SFT data rather than the specific long2short distillation methods.

2. The long-CoT model as teacher. The long2short methods all use the long-CoT model — which has already undergone extensive RL training — to generate training data or as a merging partner. This means the short-CoT model is effectively distilled from a stronger teacher. A fairer baseline would be: train a short-CoT model directly on the same curated RL prompt set using the same RL algorithm but with a short context limit, without access to the long-CoT teacher. This would isolate whether the long2short process adds value beyond what direct short-context RL training would achieve on the same data.

The consequence is that the paper's claim about the effectiveness of long2short methods conflates the contribution of the distillation process with the contribution of the underlying training data and the long-CoT teacher model. A practitioner who implements the long2short techniques but with a weaker base model or less curated training data may not see the same token efficiency gains.

What evidence exists in the paper: Figure 7 is the only comparison. All k1.5 variants (orange points) cluster at the upper-left of the Pareto frontier, while all baseline models (blue points) cluster at the lower-right. The gap between these clusters reflects everything that differs between the Kimi training pipeline and the baseline models' training pipelines — not just the long2short methods. The paper reports no ablation where the same base model is trained with short-context RL from scratch on the same data and compared against the long2short-distilled model.

Mitigation status: The paper does not acknowledge this confounding as a limitation. The long2short results are presented as a clean demonstration of method effectiveness. A partial mitigation is that the comparison does show that the overall k1.5 pipeline (including pretraining, SFT, RL, and long2short) produces superior token efficiency compared to existing models, which is practically relevant even if the attribution to specific long2short methods is uncertain. However, the paper's specific claims about model merging, DPO, and long2short RL being effective relative to each other (the comparison among orange points in Figure 7) are less confounded, as these variants share the same underlying training pipeline and differ only in the compression method.

The paper suggests iterative long2short as future work:

"it is possible to combine long2short methods with long-CoT RL in an iterative way to further increase token efficiency" (Section 4)

This vision — if realized — would make the teacher-student relationship endogenous (the teacher improves through cycles of RL and compression), partially addressing the concern about reliance on a separately trained long-CoT model. However, this remains speculative and untested in the current report.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a reframing of the primary scaling dimension for LLM reasoning: it shifts attention from model size and algorithmic complexity (tree search, process reward models, value functions) toward context length as the central axis of improvement during reinforcement learning. This is not a paradigm shift in the Kuhnian sense—the underlying technologies (Transformers, policy gradient methods, chain-of-thought prompting) remain the same—but it is a substantial reorientation of what the field should optimize for and invest in.

The reframing operates at three levels. First, at the conceptual level, the paper's argument that an auto-regressive model with a sufficiently long context window can implicitly perform the search that explicit planning algorithms implement (Section 2.3.1) provides a unifying lens that collapses two previously distinct research threads—planning-augmented decoding and RL-based reasoning training—into a single framework. If correct, this implies that the complexity of Monte Carlo tree search, value functions, and process reward models is not merely unnecessary but potentially counterproductive, because these techniques require engineering effort that could instead be directed toward scaling context length and improving policy optimization. The paper does not prove this equivalence—no controlled experiment compares explicit search against implicit long-CoT RL on the same base model—but it establishes the hypothesis in a compelling enough form to shift the burden of proof: the field must now demonstrate that explicit planning algorithms add value beyond what long-context RL achieves, rather than assuming their necessity.

Second, at the methodological level, the paper demonstrates that a relatively simple RL pipeline—online mirror descent with length penalty, curriculum sampling, and final-answer-only rewards—can produce reasoning performance matching the best known proprietary systems (OpenAI's o1). This is significant because prior published RL work had not approached this level of reasoning performance. The paper's detailed disclosure of its training recipe, infrastructure design, and ablation studies provides a template that other research groups can replicate, modify, and improve upon. This lowers the barrier to entry for RL-based reasoning research, which had previously been concentrated in a small number of industrial labs with the resources to develop complex multi-component systems. The paper's message is effectively: "you don't need MCTS, value functions, or PRMs—you need long context, good policy optimization, and careful data curation."

Third, at the practical level, the long2short paradigm (Section 2.4) resolves a tension that had been brewing in the reasoning community. Long-CoT models achieve impressive accuracy but are expensive to deploy; short-CoT models are cheap but less capable. Prior work treated this as an unfortunate tradeoff to be managed. The Kimi k1.5 paper demonstrates that the reasoning capabilities developed through long-context RL are partially separable from the verbosity of the reasoning process itself—they can be compressed into shorter, more token-efficient models through model merging, rejection sampling, DPO, and dedicated long2short RL. The k1.5-short w/ rl model achieves 60.8 on AIME 2024 with 3,272 average tokens (Figure 7), while the long-CoT model requires substantially more tokens for its performance. This means organizations can invest in expensive long-context RL training to develop reasoning capabilities, then deploy compressed models that realize most of the gain at a fraction of the inference cost. The paper's suggestion of iterative long2short cycles (long-CoT RL → compression → further long-CoT RL) points toward a self-reinforcing process where each cycle pushes the capability-efficiency frontier outward.

Prior contradictions this work helps reconcile. The paper resolves the tension between two bodies of prior work: (1) studies showing that explicit search algorithms (tree search, PRM-guided beam search) improve reasoning performance (Yao et al., 2024; Snell et al., 2024; Wu et al., 2024), and (2) the practical difficulty and cost of deploying these algorithms in production systems, which has limited their adoption. The Kimi k1.5 paper's reframing—that search can be internalized into the model's own chain of thought through RL training with long context—suggests that both perspectives were partially correct. Search is valuable, but it need not be implemented as an external algorithm; it can be learned as a behavior. This also explains why process reward models have shown mixed results in prior work: if the model can learn to evaluate its own intermediate steps through the same mechanism it uses for everything else (generating language tokens that assess progress), a separately trained critic may add marginal value beyond what the model can do internally.

The paper also helps reconcile the conflicting findings around self-correction. Prior work found that prompting models to self-correct was largely ineffective for complex reasoning (Huang et al., 2023), while other work found benefits in specific settings (Madaan et al., 2023). The Kimi k1.5 results suggest that self-correction can be effective, but it must be learned through RL with appropriate reward structure and context length, not merely prompted. The model needs to experience trial-and-error trajectories during training—making mistakes, identifying them, and recovering—to develop the capability. Prompting alone cannot teach this because the model has no training signal distinguishing good corrections from bad ones. This reframes the self-correction question from "can models do it?" (the prompting literature's focus) to "how should we train models to do it?" (the RL literature's focus).

Research directions that become more attractive. The paper makes long-context RL infrastructure a first-class research problem. Prior work on LLM training infrastructure focused primarily on pretraining efficiency (model parallelism, pipeline parallelism, gradient accumulation). The Kimi k1.5 paper demonstrates that RL training imposes fundamentally different demands—alternating training and inference phases, handling trajectories of widely varying lengths, reusing partial computation across iterations—that require specialized solutions (partial rollouts, hybrid deployment, weight transfer optimization). This opens a new subfield at the intersection of systems and ML: RL-aware training infrastructure for LLMs. The paper's specific techniques (Section 2.6) provide a starting point, but many optimization opportunities remain unexplored, including more sophisticated trajectory reuse strategies, adaptive token budgets that vary by problem difficulty, and tighter integration between the rollout and training phases.

The paper also makes the scaling behavior of context length during RL a central empirical question. Prior scaling laws work (Kaplan et al., 2020; Hoffmann et al., 2022) focused on pretraining—how performance scales with model parameters, data tokens, and compute. The Kimi k1.5 results suggest that a new scaling law is operative during RL: performance scales with context length (Figures 5 and 6), and this scaling interacts with model size (Figure 8). Characterizing this scaling relationship precisely—what functional form it follows, whether it exhibits power-law behavior, whether it saturates—would provide the same kind of principled guidance for RL training budgets that Chinchilla provides for pretraining budgets.

Research directions that become less attractive. If the paper's central claim is correct—that long-context RL can substitute for explicit planning algorithms—then research on process reward models, value functions for reasoning, and Monte Carlo tree search for LLMs faces a higher burden of proof. These techniques must now demonstrate that they provide benefits beyond what long-context RL with final-answer rewards achieves, not merely that they improve over baseline prompting or best-of-N sampling. The paper does not prove that PRMs and MCTS are unnecessary—again, no controlled comparison exists—but it shifts the default assumption from "these techniques are needed for strong reasoning" to "these techniques must justify their additional complexity." Research on process reward models, in particular, may need to pivot from demonstrating that step-level feedback improves over outcome-level feedback (which was the contribution of Lightman et al., 2023) to demonstrating that step-level feedback improves over long-context RL with outcome rewards, which is a substantially higher bar.

Similarly, the paper's success with final-answer-only rewards makes research on dense reward shaping for reasoning less urgent. If a binary success/failure signal, combined with sufficient context length and good policy optimization, produces state-of-the-art reasoning, then the effort to design intermediate rewards (e.g., rewarding specific reasoning subgoals, penalizing specific error types) may yield diminishing returns. The paper's argument that sparse rewards enable beneficial exploration (Section 2.3.2) further suggests that denser rewards could be actively harmful by constraining the model's exploration of diverse reasoning paths.

Follow-Up Research This Work Enables

Causal isolation of context length from training progress. The paper demonstrates a correlation between response length and accuracy during RL training (Figures 5 and 6), but does not establish causality. A direct follow-up experiment would take a fixed checkpoint of the k1.5 long-CoT model and evaluate it at multiple maximum generation lengths (e.g., 4k, 8k, 16k, 32k, 64k, 128k tokens) on AIME 2024 and MATH-500, measuring whether forcing the model to generate longer responses at inference time improves accuracy. If accuracy is flat or nearly flat across lengths (once the model has enough tokens to finish its reasoning), then the length-accuracy correlation during training was driven by confounding (better models happen to generate longer responses) rather than by a causal effect of length on capability. If accuracy continues to improve with forced length, the causal claim is supported. A stronger variant would test whether truncating responses at different lengths during RL training (effectively training with different context budgets) produces proportionally different final performance, which would establish a causal relationship between training-time context length and capability.

Controlled comparison against explicit planning algorithms on the same base model. The paper's central "simplistic framework" claim—that MCTS, value functions, and PRMs are unnecessary—requires a controlled experiment the paper does not perform. A strong follow-up would take the same pretrained and SFT-trained base model used for k1.5, train one variant with the paper's long-context RL approach, and train a second variant that adds process reward model-guided beam search (similar to Snell et al., 2024) or Monte Carlo tree search (similar to Yao et al., 2024) during RL training, at matched total compute budgets. The key measurement is whether the explicit search variant achieves higher final accuracy, better sample efficiency, or different reasoning behaviors compared to the long-context RL variant. If explicit search provides no benefit (or hurts), the paper's claim is substantiated. If explicit search provides complementary benefits—for example, improving performance on problems where the model struggles to generate long CoTs that self-correct, or reducing the context length needed for a given accuracy level—then the optimal approach may be a hybrid that uses explicit search during training but not at inference, or that uses PRM-guided exploration to generate better training trajectories for the long-CoT model.

Iterative long2short cycling with measurement of capability compression efficiency. The paper proposes iterative long2short as a promising direction (Section 4) but does not evaluate it. A concrete follow-up experiment would run three cycles of: (1) long-CoT RL training from the current best model, (2) long2short compression using the best method identified in the paper (long2short RL), (3) use the compressed model as the initialization for the next cycle. At each cycle, measure: the long-CoT model's accuracy on AIME 2024 and MATH-500, the compressed model's accuracy and average token length, and the total compute spent in that cycle. The key questions are: does performance continue to improve across cycles, or does it saturate? Does token efficiency improve (same accuracy at lower token cost, or higher accuracy at the same token cost)? Is the improvement per unit of compute increasing, constant, or diminishing across cycles? If the iterative process shows continued improvement, it would establish long2short as a genuine capability amplifier rather than a one-time compression step, with profound implications for how reasoning models are developed.

Length penalty design space exploration and its effect on reasoning quality. The paper introduces a specific length penalty formulation (Section 2.3.3) with a warmup schedule, but does not ablate the penalty design. A systematic follow-up would compare: (1) the paper's batch-relative normalization ($\lambda = 0.5 - (\text{len} - \text{min\_len})/(\text{max\_len} - \text{min\_len})$) against an absolute token penalty (fixed cost per token), (2) the correctness-conditioned penalty (applying $\min(0, \lambda)$ to incorrect responses) against an unconditional penalty, (3) different warmup schedules (no warmup, early warmup, late warmup), and (4) different penalty strengths (the weighting parameter mentioned but not specified in the paper). The outcomes of interest are not just final accuracy and token length, but qualitative properties of the generated reasoning: does the penalty change how the model reasons (e.g., more concise chains with fewer restatements, vs. truncated chains that fail to complete reasoning)? Do different penalty designs affect different difficulty levels differently? The paper's observation that length penalty can slow initial training suggests a delicate balance between exploration (which requires token budget) and efficiency (which penalizes verbosity)—understanding this tradeoff quantitatively would guide practitioners in setting the penalty parameters for their own domains.

Negative result: testing whether the paper's approach fails on domains without clean verifiability. The Kimi k1.5 system depends on accurate reward signals—the Chain-of-Thought RM for math (98.5% accuracy), test case execution for coding, and rule-based verification for other domains (Section 2.3.5). A critical stress test would apply the same long-context RL pipeline to a domain where verification is noisy, subjective, or expensive: for example, open-ended essay writing (where "correctness" is multi-dimensional and hard to automate), creative problem-solving (where multiple valid solutions exist), or long-horizon planning (where intermediate feedback is sparse and delayed). The hypothesis—suggested by the paper's reliance on clean rewards for prompt set curation (Section 2.1, filtering out easy-to-hack prompts) and reward modeling (Section 2.3.5)—is that performance gains will degrade as reward noise increases. Quantifying this degradation curve (accuracy vs. reward model accuracy) would establish the boundary conditions for the paper's approach and identify which application domains can benefit from long-context RL today versus which require better reward modeling first.

Difficulty estimation without sampling overhead. The paper's curriculum and prioritized sampling strategies depend on knowing per-problem success rates $s_i$, which requires sampling each problem multiple times. The paper does not account for this cost. A practical follow-up would train a lightweight difficulty predictor—a small model or linear probe on the base model's embeddings—that takes only the problem text as input and predicts the model's expected success rate. Such a predictor could be trained on the success rate data that the RL system already collects (from initial sampling), and then used to estimate difficulty for new problems without requiring additional samples. The evaluation would compare: (1) curriculum/prioritized sampling using the predictor's difficulty estimates vs. using ground-truth success rates from ongoing sampling, measuring both final accuracy and total compute (including difficulty estimation cost), and (2) whether the predictor generalizes across domains (math → coding → vision) or needs to be domain-specific. If a cheap difficulty predictor can approximate the benefits of online success rate tracking, it would make the adaptive sampling strategies practical for deployments where the cost of per-problem sampling is prohibitive.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of reasoning models through long2short compression. Organizations deploying LLMs for reasoning-heavy applications (automated math tutoring, competitive programming assistance, scientific problem-solving) face a direct cost-quality tradeoff: long-CoT models are accurate but expensive (each query consumes thousands of additional tokens), while short-CoT models are cheap but less capable. The paper's long2short results (Figure 7) demonstrate that this tradeoff can be substantially improved: k1.5-short w/ rl achieves 60.8 on AIME 2024 at 3,272 average tokens, compared to the long-CoT model's performance at ~5,000 tokens, and dramatically outperforms GPT-4o (9.3 at ~400 tokens) and Claude 3.5 Sonnet (16.0 at ~400 tokens) at comparable or modestly higher token costs. For a deployment handling 1 million AIME-level queries per month, switching from a long-CoT model (5,000 tokens/query) to the compressed short-CoT model (3,272 tokens/query) would reduce inference costs by ~35% while maintaining the same performance tier—or alternatively, deploying the compressed model instead of a GPT-4o-level baseline would provide 6.5× higher accuracy at ~8× the per-query token cost, a dramatically better point on the cost-quality Pareto frontier. Organizations can apply the paper's long2short recipes to their own base models and domain-specific reasoning tasks to find the optimal operating point for their accuracy requirements and budget constraints.

Self-improving data generation pipelines for reasoning datasets. The paper's RL training process naturally generates large volumes of diverse reasoning trajectories—both correct and incorrect—at varying difficulty levels. These trajectories can be repurposed as training data for smaller, specialized models. For example, a math education company could: (1) fine-tune the long-CoT k1.5 model (or a smaller variant) on their proprietary problem bank using the paper's RL recipe, (2) use the trained model to generate thousands of step-by-step solutions with natural variation in approach and difficulty, (3) filter for correct solutions using the Chain-of-Thought RM approach (Section 2.3.5), and (4) distill these into a compact student-facing model that provides detailed explanations at low inference cost. The paper's finding that the model learns to explore diverse reasoning paths—including backtracking and error recovery—means the generated training data will include examples of how to recover from mistakes, not just clean solution paths. This is particularly valuable for educational applications where showing students common errors and their corrections is pedagogically important. The curriculum sampling strategy (Section 2.3.4) further enables targeted data generation: focus the RL model's exploration on problems at the appropriate difficulty level for the target student population.

On-device or edge deployment of advanced reasoning through iterative compression. The iterative long2short vision suggested in the paper—cycles of long-CoT RL followed by compression—offers a path to deploy sophisticated reasoning capabilities on hardware-constrained devices. A cloud provider could: (1) train a large long-CoT model using the full k1.5 pipeline on their cloud infrastructure, (2) compress it to a short-CoT model using long2short RL, (3) deploy the compressed model to edge devices (phones, laptops, embedded systems), (4) collect anonymized queries and failure cases from the edge deployment, and (5) use these to seed the next cycle of long-CoT RL training, targeting the specific weaknesses observed in production. Each cycle would improve both the capability ceiling (through RL on the long-CoT model) and the efficiency floor (through compression for edge deployment). The paper's infrastructure innovations—particularly partial rollouts (Section 2.6.2) and the hybrid training/inference deployment (Section 2.6.3)—provide the architectural template for the cloud-side training, while the compressed models meet the latency and memory constraints of edge inference. The key metric for such a system would be the rate of improvement per cycle: does performance on edge-deployed models asymptote after a few cycles, or does iterative refinement continue to yield gains?