LIMO: Less is More for Reasoning

ArXiv: 2502.03387

🎯 Pitch

A model fine-tuned on just 817 high-quality examples crushes the AIME math competition (57.1% accuracy), annihilating systems trained on 100× more data and even outperforming OpenAI’s o1-preview. The secret is realizing that modern LLMs already possess the knowledge—they just need a few carefully crafted 'cognitive templates' to unlock it, making massive datasets obsolete for complex reasoning.

---

1. Executive Summary

This paper introduces the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis), which posits that in foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning can emerge through minimal but strategically designed demonstrations of cognitive processes — specifically, only a few hundred carefully curated examples that encourage extended deliberation during inference. Using Qwen2.5-32B-Instruct fine-tuned on just 800 high-quality question–solution pairs selected through a multi-stage filtration pipeline (prioritizing problem difficulty and reasoning chain quality via metrics like elaborated reasoning, self-verification, and exploratory approaches), LIMO achieves 63.3% on AIME24 and 95.6% on MATH500, surpassing models trained on datasets 100× larger — NuminaMath-100k (6.5% AIME24) and OpenThoughts-114k (50.2% AIME24) — and outperforming QwQ-32B-Preview and OpenAI-o1-preview, establishing that the threshold for eliciting competition-level reasoning is bounded by pre-trained knowledge completeness and exemplar quality rather than data quantity, with the approach failing only on models lacking rich mathematical pre-training (Qwen1.5-32B-Chat drops to 9.2% AIME24).

2. Context and Motivation

The Core Problem: Challenging the Assumption That More Data Always Means Better Reasoning

The paper addresses a fundamental assumption that has quietly hardened into dogma in the LLM post-training community: the belief that complex reasoning — especially mathematical reasoning at competition levels — requires massive supervised fine-tuning datasets containing tens or hundreds of thousands of examples. The authors explicitly frame this as their target in Section 1:

"While recent work has shown that LLMs can be effectively aligned with user preferences through relatively small amounts of instruction data (Zhou et al., 2023), teaching models to reason—particularly in mathematics and programming—is widely believed to require vastly more training examples (Paster et al., 2023; Yue et al., 2024)."

This is not a strawman. The prevailing paradigm in mathematical reasoning post-training genuinely operates at scale: NuminaMath-CoT provides 860k problem-solution pairs (Li et al., 2024b), MetaMath bootstraps questions for large-scale fine-tuning (Yu et al., 2024), and MAmmoTH2 scales instruction data from the web (Yue et al., 2024). The working assumption is that mastering multi-step logical deduction, domain knowledge application, and structured solution paths requires exposure to an enormous diversity of worked examples — essentially, that the model needs to "see" enough reasoning patterns to internalize the cognitive moves required for novel problems.

The paper's counterclaim is provocative: this data-intensive paradigm may no longer be necessary. The argument is not that training data is irrelevant, but rather that two recent developments have fundamentally changed the equation — (1) foundation models now encode vastly more mathematical knowledge during pre-training than even 1–2 years ago, and (2) inference-time computation scaling (long chain-of-thought reasoning) provides the computational workspace for models to unpack and apply that pre-trained knowledge. Together, these developments shift the challenge from knowledge acquisition (teaching the model concepts it doesn't know) to knowledge elicitation (activating capabilities that are already latent in the weights). And the latter, the paper argues, may require far fewer examples — potentially just hundreds — if those examples are carefully designed to demonstrate the process of extended deliberation.

Why This Problem Matters: Practical, Theoretical, and Economic Stakes

This question matters on multiple fronts:

Practical and economic significance. Training on massive curated datasets is expensive in multiple currencies: compute cost (running SFT on hundreds of thousands of long reasoning traces), engineering effort (curating, filtering, and verifying solution quality at scale), and environmental cost. If 99% of this data is redundant — if the same or better mathematical reasoning can be elicited with 800 examples instead of 100,000 — the savings cascade through every stage of the model development pipeline. The paper's Figure 1 makes this explicit, visually comparing LIMO's 800-sample training set against NuminaMath's massive corpus and showing LIMO dramatically outperforming it. For organizations building mathematical reasoning models, the implication is that investment should shift from data quantity engineering (scraping and generating ever-larger problem banks) toward data quality engineering (identifying the minimal set of exemplars that maximally activate latent capabilities).

Theoretical significance. The paper engages with a deeper debate about the nature of LLM reasoning that has been simmering across multiple papers. On one side, a body of work suggests that supervised fine-tuning leads primarily to memorization rather than true generalization for reasoning tasks (Zhang et al., 2024; Xu et al., 2024; Chu et al., 2025). Mirzadeh et al. (2024) showed that LLMs exhibit performance variance when responding to different instantiations of the same question — change the numerical values and accuracy drops, suggesting the model has memorized problem templates rather than learned underlying mathematical principles. This raises uncomfortable questions about whether SFT-based reasoning models are "true reasoners" or sophisticated pattern matchers (Kambhampati, 2024).

The LIMO Hypothesis proposes a potential resolution: if reasoning degradation under large-scale SFT comes from training on low-quality examples (uncurated reasoning chains, trivial problems that don't engage extended deliberation), then training on a small number of high-quality examples that emphasize deep cognitive processes might produce models that genuinely generalize — not because they've seen more patterns, but because they've been shown how to think through a few exemplary demonstrations. The paper's out-of-distribution results (45.8% absolute improvement across diverse benchmarks, Section 6.2) are presented as evidence for this interpretation: if the model were merely memorizing, cross-benchmark transfer from AIME-style training to Chinese Gaokao problems or GPQA would be minimal.

Connecting pre-training and inference-time research. The paper sits at the intersection of two largely separate research threads. The pre-training scaling community has focused on what goes into the model (mathematical corpora quality and quantity: Azerbayev et al., 2024; Shao et al., 2024; Wang et al., 2024). The inference-time scaling community has focused on what happens during generation (longer reasoning chains, test-time search: OpenAI, 2024; Snell et al., 2024; Guo et al., 2025). The LIMO Hypothesis explicitly ties these together: it claims that the synergy between rich pre-trained knowledge (what the model already knows) and inference-time computation space (the cognitive workspace to unpack that knowledge) is what makes data-efficient reasoning elicitation possible. This is a theoretical bridge between two communities that rarely talk to each other.

Where Prior Approaches Fall Short

The paper identifies specific limitations across three axes of prior work:

1. Large-scale SFT for reasoning assumes data quantity is the lever. The dominant post-training paradigm (exemplified by MetaMath (Yu et al., 2024), MAmmoTH (Yue et al., 2023), MAmmoTH2 (Yue et al., 2024), and NuminaMath (Li et al., 2024b)) operates on the implicit theory that scaling supervised examples monotonically improves reasoning. The paper's direct experiments contradict this: models trained on NuminaMath-100k (a random 100k subset of NuminaMath-CoT) achieve only 6.5% on AIME24 and 32.3% average across all benchmarks — actually worse than the untrained base model's 49.9% average (Table 1). This is a striking result: more data, carefully annotated at scale, produced catastrophic degradation. OpenThoughts-114k fares better (50.2% AIME24, 58.3% average) but still underperforms LIMO's 800 examples by roughly 20 percentage points of average accuracy. These are not minor efficiency differences — they suggest that the relationship between data quantity and reasoning quality is fundamentally non-monotonic, and that large-scale SFT with uncurated or semi-curated data can actively damage reasoning capabilities.

2. The field conflates alignment data efficiency with reasoning data efficiency. The paper explicitly cites LIMA (Zhou et al., 2023) as inspiration — demonstrating that 1,000 carefully curated instruction-following examples can achieve strong alignment. But the paper notes an important caveat:

"whether this lesson can be applied to reasoning tasks remains uncertain, given the potential high computational complexity of such tasks (Merrill & Sabharwal, 2024; Xiang et al., 2025)."

Alignment tasks (following instruction formats, maintaining helpful/safe tone) might be fundamentally different from reasoning tasks (multi-step logical deduction, domain knowledge integration, solution structuring). The computational complexity argument — that reasoning requires navigating exponentially large solution spaces, while format-following is lower-dimensional — suggests that the LIMA lesson might not transfer. The paper positions itself as testing this very question: can the "less is more" philosophy extend from alignment to reasoning, and if so, what constitutes "quality" in the reasoning domain? Prior to this work, no one had systematically addressed this, leaving a gap between the alignment community's quality-over-quantity findings and the reasoning community's scale-default approach.

3. Existing data curation for reasoning focuses on problem volume, not solution quality. Some prior work has emphasized the importance of data quality in mathematical reasoning training (Zhou et al., 2024; Yu et al., 2024), but the quantity of data used in these efforts is still "much larger compared to that in LIMA" (Section 2.3). These approaches typically filter for problem difficulty or correctness but don't deeply interrogate what makes a reasoning chain high-quality — the specific cognitive features (elaborated reasoning, self-verification, exploratory approach, adaptive granularity) that the LIMO paper operationalizes into its scoring rubric in Section 3.1.2. The prior assumption was that any correct solution chain to a challenging problem was adequate training material. The LIMO paper argues this is false: even among correct solutions to identically difficult problems, reasoning chain quality dramatically affects downstream model performance (Figure 3, showing a consistent gradient from L1 to L5 solution quality levels).

4. The relationship between pre-trained knowledge and post-training data requirements is undertheorized. The literature has documented that better pre-trained models perform better after fine-tuning (obviously), but the paper's claim is stronger: richer pre-trained knowledge changes the nature of what fine-tuning needs to accomplish. When the model already possesses mathematical knowledge (concepts, procedures, common problem structures) from pre-training, fine-tuning's role shifts from teaching to activating — showing the model, through exemplars, how to marshal its existing knowledge into extended deliberative reasoning. This reframes what constitutes an effective training example: it's not one that introduces new facts, but one that demonstrates cognitive process — how to structure reasoning, verify intermediate steps, explore alternatives, and backtrack when stuck. This theoretical reframing has no direct precedent in the reasoning literature.

How This Paper Positions Itself

The paper positions itself as a direct extension of the LIMA philosophy into the reasoning domain, but with a crucial theoretical addition: the identification of two necessary conditions that make data-efficient reasoning elicitation possible. LIMA showed that for alignment, quality matters more than quantity; LIMO argues that for reasoning, quality matters more than quantity if and only if the model possesses sufficient pre-trained knowledge and the training examples effectively utilize inference-time computation space.

This dual-condition structure is important because it explains when the approach should work and when it should fail. The paper doesn't claim universal data efficiency — it claims conditional efficiency, bounded by pre-training quality. This is directly tested in Section 6.3.3 (Figure 5): the same 800 LIMO examples produce 63.3% AIME24 accuracy on Qwen2.5-32B-Instruct (which had enhanced mathematical pre-training) but only 9.2% on Qwen1.5-32B-Chat (same architecture, earlier generation with weaker math pre-training). The 54.1 percentage point gap isolates the pre-training knowledge effect, providing strong negative evidence that supports the conditional nature of the hypothesis.

The paper also positions itself against the dominant inference-time scaling narrative. Rather than treating test-time computation (long reasoning chains, search, RL-based reasoning training) as an alternative to training data altogether, the LIMO framework treats it as an enabling condition for data efficiency: the training examples need to demonstrate extended deliberation so the model learns to occupy that inference-time computational workspace. This connects to the work of Guo et al. (2025) (DeepSeek-R1) and OpenAI (2024) (o1), which showed that models trained with reinforcement learning to generate long chain-of-thought exhibit self-reflection, verification, and backtracking. LIMO's contribution is showing that these long-CoT patterns can be demonstrated through examples (via SFT) rather than discovered through RL, dramatically reducing the data and compute requirements — if the pre-training foundation is sufficiently rich.

Finally, the paper implicitly challenges the memorization-vs-generalization framing that has colored recent evaluations of SFT-based reasoning models (Mirzadeh et al., 2024; Zhang et al., 2024; Chu et al., 2025). These critiques focus on models trained with large-scale, often uncurated data. LIMO's out-of-distribution results — strong performance on Chinese-language exams (CHMath, Gaokao, Kaoyan), multi-disciplinary benchmarks (Minerva, GPQA), and diverse competition formats — suggest that the problem may not be SFT per se, but what kind of SFT data is used. High-quality exemplars that emphasize cognitive process over answer-memorization may produce models that genuinely generalize, making the data quality critique a meta-critique: the field's problems with memorization may be self-inflicted through poor data curation rather than inherent to supervised fine-tuning. This is a significant reframing of an active debate, and it positions the paper not just as an efficiency result but as a methodological intervention in how the community thinks about reasoning training.

3. Technical Approach

3.1 Reader Orientation

The system is a recipe for building a competition-level mathematical reasoning model through supervised fine-tuning on a deliberately tiny, ruthlessly curated dataset of only 800 question–solution pairs. It solves the problem of knowledge elicitation rather than knowledge acquisition: given a foundation model that already contains rich mathematical knowledge from pre-training, the goal is to activate that latent capability using minimal but strategically designed exemplars that demonstrate extended, deliberative cognitive processes — essentially, teaching the model how to think through hard problems by showing it a few hundred exemplary thinking traces rather than flooding it with tens of thousands of mediocre ones.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components arranged in a pipeline:

1. Candidate Pool — a massive collection of millions of mathematical problems from diverse sources (NuminaMath-CoT, DeepScaleR, AIME historical exams, MATH, Chinese examination papers) that serves as the raw material for curation.

2. Question Selection Pipeline — a two-stage difficulty-based filter that winnows millions of problems down to 2,125 challenging ones (the "LIMO-Pool") by first eliminating problems that a baseline math model solves trivially, then keeping only problems that a strong reasoning model solves rarely (1–3 successes in 32 attempts).

3. Reasoning Chain Construction and Scoring System — for each of the 2,125 surviving problems, multiple state-of-the-art reasoning models generate solution traces, which are then scored by a rule-based metric that quantifies four dimensions of reasoning quality (elaborated reasoning, self-verification, exploratory approach, adaptive granularity); the highest-scoring solution per problem is selected, and the top 800 problem–solution pairs by composite quality score form the LIMO dataset.

4. Supervised Fine-Tuning — Qwen2.5-32B-Instruct is trained on these 800 examples using standard SFT with full-parameter optimization and a specific set of hyperparameter choices designed for rapid adaptation to high-quality reasoning demonstrations.

Information flows linearly: raw problems → difficulty filtering → solution generation from reasoning models → quality scoring → top-800 selection → SFT training → evaluation model.

3.3 Roadmap for the Deep Dive

• First, the LIMO Hypothesis formalization (Section 3, opening text), because every subsequent design choice — which questions to select, which solutions to keep, how to score them — is derived from this hypothesis about what makes data-efficient reasoning elicitation possible.

• Second, the Question Selection Pipeline (Section 3.1.1), because it determines which problems enter the candidate pool, and the difficulty-based filtering logic directly operationalizes the hypothesis's emphasis on challenging problems that require extended inference-time computation.

• Third, Reasoning Chain Construction and the Quality Scoring Rubric (Section 3.1.2), because it determines which solutions are selected for each surviving problem, and the four-dimensional scoring metric is the paper's concrete answer to "what makes a reasoning chain high-quality."

• Fourth, the Training Recipe (Section 4), because understanding the hyperparameters and optimization choices explains how the 800 examples are actually absorbed by the model, and several choices (no warmup, 15 epochs on a tiny dataset, cosine decay) are deliberate deviations from standard practice that are motivated by the data characteristics.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methodology paper whose core idea is that the relationship between training data quantity and reasoning quality is non-monotonic — beyond a small threshold of well-chosen exemplars, additional data can actively harm performance by diluting the training signal with low-quality reasoning patterns — and that the key to data-efficient reasoning is selecting problems that are hard enough to require extended deliberation and solutions that demonstrate how to deliberate effectively, rather than merely arriving at correct answers.

---

The LIMO Hypothesis Formalization

The paper opens Section 3 with a formal statement of the Less-Is-More Reasoning Hypothesis:

"In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes."

This is not presented as a theorem to be proved but as a falsifiable empirical claim that the rest of the paper's experiments are designed to test. The hypothesis has two necessary conditions:

• Premise I: The latent presence of prerequisite knowledge within the model's parameter space. If the model doesn't already "know" the mathematical concepts, procedures, and problem structures needed to solve competition-level problems, no amount of clever exemplar curation can compensate — the knowledge must be there for fine-tuning to elicit it.

• Premise II: The quality of reasoning chains that precisely decompose complex problems into detailed, logical steps, making the cognitive process explicit and traceable. It is not sufficient that training examples be correct; they must demonstrate how to think — the meta-cognitive moves (verification, backtracking, exploration) that constitute expert mathematical reasoning.

The hypothesis makes a specific prediction about where data efficiency should work and where it should fail: on models with rich mathematical pre-training (Premise I satisfied), a small number of high-quality exemplars (Premise II satisfied) should produce strong reasoning. On models with weak mathematical pre-training, even the same exemplars should fail. This prediction is tested directly in Section 6.3.3 (Figure 5).

Crucially, the hypothesis reframes what fine-tuning is doing: it is not teaching the model new knowledge but rather showing it how to use knowledge it already has. This distinction explains why the approach is so sensitive to pre-training quality and why data quantity is less important than data character — the model already possesses the conceptual vocabulary; what it needs is demonstrations of syntax for combining those concepts into extended reasoning chains.

The paper also formalizes the reasoning task itself as a mapping: given a question $q \in Q$, the model must produce an answer $a \in A$ via a reasoning chain $r \in R$ consisting of intermediate steps $\{s_1, s_2, ..., s_n\}$, formalized as:

$f : Q \rightarrow R \times A$

where $Q$ is the space of questions, $R$ is the space of reasoning chains (sequences of intermediate reasoning steps), and $A$ is the space of final answers.

What it computes: the function $f$ takes a natural-language mathematical question and outputs both a step-by-step reasoning trace and a final answer. This decomposition matters because the paper's central claim is that the quality of $r$ — not just the correctness of $a$ — determines training effectiveness.

Why this form: by explicitly separating $R$ (the reasoning process) from $A$ (the answer), the formulation emphasizes that the training objective is not answer-memorization (learning the mapping $Q \rightarrow A$ directly) but process-learning (learning how to construct $R$ such that $A$ follows logically). This directly connects to the paper's operationalization of reasoning chain quality in Section 3.1.2.

The dataset construction goal is then stated as producing a deliberately small, high-quality set:

$D = \{(q_i, r_i, a_i)\}_{i=1}^N$

with $N = 800$, where each triple is selected to maximize the elicitation of the model's latent reasoning capabilities given the two premises.

---

The Candidate Pool and Problem Sources

Before any filtering, the paper assembles a massive pool of mathematical problems from five distinct sources (Section 3.1.1, Figure 2). This is not a random collection — each source is chosen to provide specific characteristics that serve the filtering pipeline's goals:

• NuminaMath-CoT (Li et al., 2024b): described as "featuring meticulously annotated problems from high school to advanced competition levels." This provides a large base of problems with existing solution annotations spanning a wide difficulty range, ensuring coverage of standard competition mathematics topics.

• DeepScaleR (Luo et al., 2025): "consists of approximately 40,000 unique mathematics problem-answer pairs." This contributes additional problem diversity, particularly in the competition-to-olympiad difficulty range.

• AIME historical examination problems before 2024: "known for its extremely challenging and integrative problems spanning multiple mathematical domains." AIME (American Invitational Mathematics Examination) problems are deliberately chosen because they represent the gold standard of difficulty — problems requiring multi-step reasoning that integrates concepts from algebra, geometry, number theory, and combinatorics. The "before 2024" restriction is critical: AIME 2024 is used as an evaluation benchmark, so including it in training would constitute data contamination. The paper explicitly verifies this separation through "thorough deduplication against all evaluation benchmarks with n-gram matching" (Section 3.1.1).

• MATH (Hendrycks et al., 2021): "encompassing various competitive mathematics problems from prestigious contests." MATH provides problems across five difficulty levels (1–5) and seven subjects, offering a structured difficulty gradient that is leveraged in the ablation studies (Section 6.3.2).

• Chinese examination papers: "including Chinese elementary, middle school, high school, and undergraduate-level exercises and examination papers." This source serves two purposes: it expands problem diversity beyond Anglophone competition formats, and it provides problems that will later serve as out-of-distribution evaluation benchmarks (CHMath, Gaokao, Kaoyan).

The initial corpus is described as "tens of millions of mathematical problems," though the exact count is not specified. This massive initial pool is necessary because the multi-stage filtration pipeline aggressively removes problems — the goal is to identify the tiny fraction that are simultaneously challenging enough, diverse enough, and solvable with high-quality reasoning chains.

---

Stage 1: Baseline Difficulty Filter (Coarse Filtering)

The first filtration stage (Section 3.1.1) uses Qwen2.5-Math-7B-Instruct (Yang et al., 2024), a 7-billion-parameter model specifically fine-tuned for mathematical reasoning using short chain-of-thought.

The procedure:

1. Present each problem from the candidate pool to Qwen2.5-Math-7B-Instruct.
2. Allow the model 4 attempts to solve each problem.
3. If the model produces a correct answer on any of the 4 attempts, exclude the problem.

The logic is straightforward: this model serves as a minimum difficulty bar. If a 7B short-CoT math model can solve a problem within 4 tries, the problem is too easy to warrant inclusion in the dataset. Such problems would not require the extended deliberation that the LIMO hypothesis identifies as essential for activating latent reasoning capabilities — a model could solve them through relatively shallow pattern-matching without engaging in deep cognitive processing.

Design choice: Why 4 attempts rather than 1? The paper doesn't explicitly justify this number, but it likely reflects a pragmatic balance: 1 attempt might miss problems where the 7B model occasionally succeeds by chance, making the filter too permissive; many more attempts would increase the computational cost of the already-expensive filtering pipeline. Four attempts provides a reasonable signal of whether the problem is within the "easy" regime for a competent but not extraordinary math model.

Design choice: Why Qwen2.5-Math-7B-Instruct specifically? This model is from the same family as the eventual base model (Qwen2.5-32B-Instruct), ensuring that the difficulty assessment is calibrated to the Qwen architecture's strengths and weaknesses. Using a model from a different family (e.g., Llama-based) might filter based on architecture-specific capabilities rather than genuine problem difficulty.

The output of Stage 1 is a substantially reduced problem set, though the paper does not report the exact number of surviving problems. The key property is that all remaining problems are ones that the 7B math model consistently fails on — they are, at minimum, non-trivial.

---

Stage 2: Rigorous Difficulty Assessment (Fine-Grained Filtering)

The second stage applies a much more stringent filter using a significantly more powerful model: DeepSeek-R1-Distill-Qwen-32B (Guo et al., 2025). This is a 32B-parameter model distilled from DeepSeek-R1, which was trained with reinforcement learning to generate long chain-of-thought reasoning with self-verification and backtracking — precisely the kind of extended deliberation the LIMO hypothesis values.

The procedure (Section 3.1.1):

1. For each problem surviving Stage 1, sample 32 solution attempts from DeepSeek-R1-Distill-Qwen-32B.
2. Compute the empirical success rate: the fraction of the 32 attempts that produce the correct answer.
3. Retain only problems where the success rate is between 1/32 and 3/32 — that is, problems that the strong model solves in exactly 1, 2, or 3 out of 32 attempts.

The logic is precise and carries several implications:

• Problems solved 0 times out of 32 are excluded. If even a top-tier 32B reasoning model cannot solve a problem after 32 attempts, the problem is likely too hard — beyond the capability frontier that the LIMO approach can elicit from the base model. Including such problems would provide training examples that the model cannot learn from (it lacks the prerequisite knowledge, violating Premise I of the LIMO Hypothesis).

• Problems solved 4+ times out of 32 are excluded. If the strong model can solve the problem reliably (≥4/32 ≈ 12.5% success rate), the problem is too easy — it doesn't force the kind of extended deliberation that serves as a useful cognitive template. The model can solve it without deploying the full suite of meta-cognitive strategies (self-verification, backtracking, exploration) that the LIMO dataset aims to demonstrate.

• Problems solved 1–3 times out of 32 are retained. This narrow window — roughly 3–9% success rate for a top-tier reasoning model — identifies problems that are at the edge of solvability. These problems require the model to marshal all its reasoning capabilities, try different approaches, verify intermediate results, and potentially backtrack from dead ends. They are exactly the problems whose solution traces will contain the rich cognitive processes that the LIMO hypothesis identifies as essential for elicitation.

Design choice: Why 32 attempts? The paper does not explicitly justify this number, but the statistical reasoning is clear: 32 samples provide enough resolution to distinguish roughly 3% increments in success rate (1/32 ≈ 3.1%, 2/32 ≈ 6.2%, 3/32 ≈ 9.4%), giving a reasonably fine-grained difficulty signal while keeping the computational cost manageable. With fewer samples (e.g., 8), the measurement would be too coarse to identify the narrow 1–3 success window. With more samples (e.g., 64), the cost would double without substantially improving the difficulty estimate's utility.

Design choice: Why DeepSeek-R1-Distill-Qwen-32B rather than the base DeepSeek-R1? The distilled version is from the Qwen architecture family, maintaining architectural consistency with the eventual base model while providing the reasoning capabilities of the full R1 model through distillation. This ensures the difficulty assessment is relevant to the Qwen model family.

This stage produces 2,125 problems, which the paper designates as the LIMO-Pool. These are the problems that survive both filtration stages — non-trivial enough to stump a 7B math model, and challenging-but-solvable for a top-tier 32B reasoning model.

Deduplication protocol. Before proceeding, the paper applies "thorough deduplication against all evaluation benchmarks with n-gram matching, confirming no overlap existed" (Section 3.1.1). This step is essential for the validity of subsequent evaluation claims, particularly the out-of-distribution generalization results. Without rigorous deduplication, strong performance on benchmarks like AIME24, MATH500, or Chinese examination papers could be attributed to training data contamination rather than genuine reasoning capability. The n-gram matching approach compares sequences of tokens between training and evaluation problems to detect near-duplicates, not just exact string matches — catching problems that differ in minor wording but are essentially identical.

---

Reasoning Chain Generation: Sources and Rationale

For each of the 2,125 problems in the LIMO-Pool, the paper generates multiple candidate solution traces using three state-of-the-art reasoning models (Section 3.1.2):

• DeepSeek R1 (the full model, Guo et al., 2025): trained with reinforcement learning to produce extended chain-of-thought reasoning with self-verification, reflection, and backtracking. This model represents the strongest available open-weight reasoning capability at the time of the paper's writing.

• DeepSeek-R1-Distill-Qwen-32B (Guo et al., 2025): the same model used for Stage 2 difficulty filtering. Its inclusion ensures solution diversity — different model checkpoints may produce qualitatively different reasoning approaches to the same problem.

• QwQ-32B (Team, 2025b): described as an independent reasoning model from the Qwen team, providing a third source of high-quality reasoning traces with potentially different stylistic characteristics.

For each problem, multiple solutions are sampled from each model (the exact number per model is not specified, but the total per problem is sufficient to allow selection among candidates). This multi-model, multi-sample approach is designed to produce a diverse set of reasoning approaches for each problem, from which the highest-quality trace can be selected.

Design choice: Why these three models? They represent the frontier of open-weight reasoning at the time, and their different training procedures (RL for R1, distillation for R1-Distill-Qwen, independent development for QwQ) should produce reasoning traces with different characteristics. This diversity is important because the subsequent quality filtering selects the best solution — having diverse candidates increases the probability that at least one solution per problem exhibits the desired cognitive qualities.

Design choice: Why generate solutions from reasoning models rather than human experts? Three likely reasons: (1) cost and scalability — generating 2,125+ solutions from human mathematicians would be prohibitively expensive; (2) distribution match — solutions generated by models are in the same distributional style as what the fine-tuned model will produce, reducing the domain gap between training demonstrations and inference-time behavior; (3) reproducibility — automated generation enables exact replication and systematic quality scoring.

---

The Quality Scoring Rubric: Four Dimensions of Reasoning Chain Excellence

This is the methodological heart of the paper. The authors conducted "a comprehensive analysis of these filtered solutions through collaborative examination" (Section 3.1.2) — all authors manually reviewed the generated reasoning chains — and through "careful observation and systematic review" identified four key characteristics that distinguish high-quality reasoning chains from merely correct ones.

The four dimensions and their operational definitions:

1. Elaborated Reasoning: "Comprehensive exploration of logical steps without premature conclusions." This dimension captures whether the solution fully unpacks its reasoning rather than jumping to conclusions through ellipsis or hand-waving. A solution with high elaborated reasoning shows the intermediate algebraic manipulations, explicitly states the logical justification for each step, and makes the inferential chain complete rather than leaving gaps for the reader to fill.

2. Self-Verification: "Regular validation of intermediate results and logical consistency." This captures meta-cognitive checking behaviors — does the solution pause to verify that a computed value is reasonable, re-check algebraic derivations, confirm that assumptions are satisfied before applying a theorem, or validate that the final answer satisfies the original problem constraints? These are behaviors that human mathematicians engage in extensively and that prior work (Guo et al., 2025; OpenAI, 2024) has identified as hallmarks of strong reasoning model outputs.

3. Exploratory Approach: "Consideration of multiple possibilities before reaching conclusions." This captures whether the solution considers alternative solution strategies, explicitly rejects dead-end approaches, or weighs multiple candidate answers before selecting the final one. Rather than presenting a single linear path from problem to answer, exploratory solutions show the process of search — trying one approach, recognizing it's unfruitful, pivoting to another.

4. Adaptive Granularity: "Appropriate detail level across simple and complex deductions." This captures whether the solution adjusts its level of detail based on the complexity of the current step — spelling out non-trivial derivations in full while moving quickly through routine algebraic simplifications. Poor solutions often have uniform granularity (everything in excessive detail, or everything sketched) regardless of the step's difficulty.

Design choice: Why these four dimensions? The paper argues they are derived inductively from examination of actual high-quality reasoning chains rather than imposed a priori. They correspond to distinct aspects of expert mathematical reasoning that are observable in solution texts: structural completeness (elaborated reasoning), error detection (self-verification), search strategy (exploratory approach), and communication efficiency (adaptive granularity). Together, they operationalize the difference between "a correct answer" and "an exemplary reasoning process."

---

The Rule-Based Scoring Mechanism

To operationalize these qualitative dimensions into a computable score, the paper implements a rule-based scoring system using keyword frequency analysis (Section 3.1.2). This is an important methodological choice — rather than using a learned quality estimator (which would require labeled quality data and could introduce its own biases) or relying solely on human judgment (which wouldn't scale), the paper extracts simple, interpretable proxies for each dimension.

The scoring formula:

$\text{Score} = 0.30 \times \text{Elaborated} + 0.25 \times \text{Exploratory} + 0.20 \times \text{Self-Verification} + 0.25 \times \text{Adaptive}$

where each component is computed from the solution text as follows (paraphrasing the paper's specification):

• Elaborated Reasoning (30% weight): Measured by solution length — longer solutions, all else equal, are assumed to contain more elaborated reasoning steps. The paper does not specify whether this is character count, token count, or word count, but the normalization step (described below) ensures fair comparison across solutions of different lengths.

• Self-Verification (20% weight): Measured by frequency of validation-related words — terms like "check," "verify," "confirm," "validate," and variants. The count of these terms in the solution text serves as a proxy for how often the solution pauses to verify its work.

• Exploratory Approach (25% weight): Measured by frequency of tentative expressions — terms like "perhaps," "might," "alternatively," "another approach," "we could also." These linguistic markers indicate consideration of multiple possibilities rather than a single confident path.

• Adaptive Granularity (25% weight): Measured by frequency of connective phrases — terms like "therefore," "since," "because," "consequently," "it follows that." These logical connectors indicate explicit step-to-step reasoning rather than implicit jumps, and their density serves as a proxy for how carefully the solution articulates its inferential structure.

Critical normalization step: The paper explicitly states that "all keyword frequencies were normalized by text length to ensure fair comparison across solutions of different sizes." Without this normalization, longer solutions would automatically score higher on all keyword-based dimensions simply by virtue of having more words. The normalization converts raw keyword counts into rates (keywords per unit length), which measures the density of verification, exploration, or logical connection rather than the absolute quantity.

What the scoring system computes: for each candidate solution to a problem, it produces a single scalar quality score that combines length (as a proxy for elaboration), verification keyword density, exploratory language density, and logical connector density, weighted 30/20/25/25. This score ranks solutions for the same problem, enabling selection of the highest-quality reasoning trace.

Why this form and not a learned quality model:

1. Interpretability: The rule-based score is transparent — one can inspect a solution and understand why it received its score by looking at length and keyword usage. This matters for a paper making a methodological claim about "quality," because a black-box quality estimator would make it impossible to verify that the selected solutions actually exhibit the claimed characteristics.

2. Efficiency: Computing keyword frequencies is trivially fast compared to running a separate LLM-based quality evaluation, enabling scoring of many candidate solutions per problem.

3. Avoiding circularity: A learned quality model would need training data — which would require human quality judgments, creating a bottleneck and introducing annotator subjectivity. The rule-based approach uses objective, computable features.

4. Theoretical alignment: The specific keyword categories are chosen to operationalize the four dimensions that the authors identified through qualitative analysis. The weighting reflects a prior judgment about the relative importance of each dimension, with elaborated reasoning (length) receiving the highest weight (30%) and self-verification receiving the lowest (20%).

Limitations of the rule-based approach (acknowledged implicitly): The scoring system is a proxy — solutions can score highly on keyword density without genuinely exhibiting the corresponding cognitive quality. A solution could contain many instances of "therefore" without actually making valid logical connections, or many "checks" that are perfunctory rather than substantive. The paper's reliance on keyword-based scoring means that the "quality" being optimized is an approximation of the underlying construct. However, for the paper's purpose — demonstrating that solution quality matters for downstream performance — this approximation is sufficient, particularly because it enables systematic, reproducible comparison across thousands of solutions.

---

Final Dataset Assembly: Top-800 Selection

From the 2,125 problems in the LIMO-Pool, the assembly procedure (Section 3.1.2) follows these steps:

1. For each problem, compute the quality score for each candidate solution (from DeepSeek R1, DeepSeek-R1-Distill-Qwen-32B, and QwQ-32B).
2. Select the single highest-scoring solution per problem.
3. Rank all 2,125 problem–solution pairs by their solution quality score.
4. Extract the top 800 pairs to form the LIMO Dataset.

Design choice: Why 800? The paper does not provide an explicit justification for this specific number, but Figure 7 in the experimental analysis shows that performance improvement exhibits diminishing returns beyond 800 samples — the gains from 800 to 1,200 samples on AIME24 are marginal (+0.9 percentage points), and 800 appears to be near the elbow of the performance curve. The choice of 800 may reflect both this empirical observation (discovered through the dataset size ablation in Section 6.3.5) and a deliberate symbolic parallel to LIMA's 1,000-sample dataset, slightly reduced to emphasize the "less is more" claim.

Design choice: Why select the highest-scoring solution per problem rather than multiple solutions per problem? If multiple solutions to the same problem exhibit different high-quality reasoning approaches (e.g., one using algebraic manipulation, another using geometric insight), including multiple solutions could demonstrate solution diversity. The paper's choice suggests a priority on quality density over approach diversity — for each problem, the model sees the single best reasoning chain, maximizing the per-example quality of the training signal. The diversity of approaches comes from having 800 different problems spanning multiple mathematical domains rather than multiple approaches to the same problem.

Knowledge point diversification. The paper mentions that during the filtration process, "knowledge point diversification" ensures "comprehensive coverage" (Section 3.1.1), though the specific mechanism for ensuring diversity across mathematical topics is not detailed in the dataset construction section. This likely means that the selection process considers topic coverage — algebra, geometry, number theory, combinatorics, etc. — to avoid over-representing any single mathematical domain, though whether this is enforced through explicit constraints or emerged naturally from the difficulty filtering is unclear from the text.

The resulting LIMO Dataset contains 800 question–solution pairs where:

• Questions are drawn from the challenging edge of solvability (1–3 successes in 32 attempts by a top reasoning model)
• Solutions are the highest-scoring reasoning chains from frontier reasoning models, selected by a four-dimensional quality rubric
• The dataset spans multiple mathematical competition sources and difficulty levels
• No evaluation benchmark problems are included (verified by n-gram deduplication)

---

SFT Training Configuration and Design Rationale

The training recipe (Section 4) applies standard supervised fine-tuning to Qwen2.5-32B-Instruct, but several hyperparameter choices are deliberate deviations from common practice, each motivated by the specific characteristics of the LIMO dataset.

Model specification: Qwen2.5-32B-Instruct (Qwen et al., 2025), a 32-billion-parameter instruction-tuned model from the Qwen2.5 family. The paper describes this model as having "enhanced pre-training data quality, particularly in mathematical and code-related content" (Section 6.3.3), satisfying Premise I of the LIMO Hypothesis.

Sequence length limit: 16,384 tokens. The paper states: "We set as our sequence length limit because all SFT response sequences remain under 16,384 tokens." This is an important detail: the LIMO solutions are long (they exhibit "elaborated reasoning," after all), but not so long that they exceed 16K tokens. Setting the limit to just above the maximum solution length avoids unnecessary padding or truncation while ensuring all training examples fit in full. For context, 16,384 tokens at roughly 4 characters per token is approximately 65,000 characters of reasoning text — substantially longer than what a typical short-CoT solution would require.

Optimization infrastructure: Full-parameter fine-tuning with DeepSpeed ZeRO-3 optimization (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023). ZeRO-3 partitions model parameters, gradients, and optimizer states across multiple GPUs, enabling full-parameter training of a 32B model without model parallelism complexity. FlashAttention-2 provides memory-efficient attention computation, reducing the GPU memory footprint of the long sequences in the LIMO dataset. Together, these infrastructure choices enable training the full 32B model without parameter-efficient methods (like LoRA) that might constrain the model's ability to fully absorb the reasoning patterns from the training data.

Learning rate and schedule: Learning rate of 5.0 × 10⁻⁶ ($5.0e^{-6}$) with a cosine decay schedule. This is a relatively low learning rate for SFT, appropriate for fine-tuning a model that is already instruction-tuned — higher learning rates risk catastrophic forgetting of the model's pre-existing capabilities.

No warmup phase. The paper explicitly states: "We deliberately omit the warmup phase to facilitate rapid adaptation to the high-quality reasoning examples in our dataset." This is a non-standard choice. Typically, SFT includes a warmup phase where the learning rate linearly increases from zero to the target value over the first few hundred steps, which stabilizes training by preventing large gradient updates on the first batches. Omitting warmup means the model receives full learning rate updates from the very first batch.

Why omit warmup? The justification — "to facilitate rapid adaptation" — suggests that the paper wants the model to aggressively fit the high-quality reasoning patterns from the start. Since the dataset is small (800 examples) and every example is carefully curated for quality, there is less risk of the model being destabilized by noisy or contradictory training signals in the first batches. The expected benefit is faster convergence and more complete absorption of the reasoning patterns in the limited training data.

Epochs and batch size: 15 epochs with batch size 64. For a dataset of 800 examples with batch size 64, each epoch consists of approximately 12.5 update steps (800/64 = 12.5, so roughly 13 steps per epoch with the last batch being smaller), totaling approximately 188–195 training steps across 15 epochs.

Why 15 epochs? Training for 15 epochs on only 800 examples means the model sees each training example 15 times. This is a high number of epochs by standard SFT standards — typical fine-tuning runs use 1–5 epochs on larger datasets to avoid memorization or overfitting. The high epoch count on this tiny dataset represents an explicit choice to deeply internalize the reasoning patterns, justified by the claim that the training examples are exemplary demonstrations of cognitive process rather than facts to be memorized — repeatedly exposing the model to the same high-quality reasoning patterns should reinforce the meta-cognitive skills rather than encouraging rote memorization (since the problems themselves are all unique, the model is learning reasoning patterns that generalize across problems rather than problem–answer associations).

Why batch size 64? A batch of 64 long reasoning sequences (up to 16K tokens each) represents a substantial memory requirement — up to 1 million tokens per batch. The paper chooses the largest batch size that fits in GPU memory given the ZeRO-3 and FlashAttention-2 optimizations. Larger batch sizes provide more stable gradient estimates, though with only ~13 steps per epoch, the batch-to-batch variance is inherently limited.

Cosine decay schedule. The learning rate follows:

$\eta(t) = \eta_0 \cdot \frac{1}{2}\left(1 + \cos\left(\frac{t}{T}\pi\right)\right)$

where $\eta_0 = 5.0 \times 10^{-6}$ is the initial learning rate, $t$ is the current training step, and $T$ is the total number of training steps (~188–195).

What it computes: the learning rate starts at $5.0 \times 10^{-6}$ and smoothly decays to near zero by the end of training, following a cosine curve. The rate changes most rapidly in the middle of training and most slowly near the beginning and end.

Why this form: cosine decay provides a smooth annealing that is gentler than step decay (which introduces discontinuities) and more aggressive than linear decay (which maintains higher learning rates through the midpoint). For a tiny dataset trained for many epochs, the smooth decay to near-zero learning rate at the end is important — it allows the model to settle into a stable minimum after the aggressive early fitting enabled by the no-warmup start.

Training objective: Standard autoregressive language modeling loss (next-token prediction) computed only on the response tokens (the reasoning chain + final answer). The prompt tokens (the question) are not included in the loss computation. This is standard for SFT — the model is trained to produce the high-quality reasoning chain conditioned on the question, not to predict the question itself.

---

Summary of Design Choices and Their Connections to the LIMO Hypothesis

The entire technical pipeline can be understood as an operationalization of the two premises of the LIMO Hypothesis:

• Premise I (pre-trained knowledge) is addressed through model selection (Qwen2.5-32B-Instruct with enhanced mathematical pre-training), validated by the Qwen1.5 comparison (Section 6.3.3), and reinforced through difficulty filtering that selects problems at the edge of the model's latent capability.

• Premise II (reasoning chain quality) is operationalized through the four-dimensional scoring rubric that selects solutions exhibiting elaborated reasoning, self-verification, exploratory approach, and adaptive granularity — cognitive patterns that the training recipe (no warmup, 15 epochs, cosine decay to near-zero) is explicitly designed to deeply internalize.

The key design tension in the pipeline is the reliance on keyword-based proxies for reasoning quality. The four quality dimensions are rich cognitive constructs; the scoring mechanism operationalizes them through simple lexical features (length, keyword densities). This is simultaneously a strength (transparent, reproducible, scalable) and a weakness (solutions can "game" the scoring without genuinely exhibiting high-quality reasoning). The paper's experimental validation — showing that L5 solutions (highest keyword scores) produce better downstream models than L1 solutions (lowest scores) in Figure 3 — provides evidence that the proxy is sufficiently correlated with genuine quality to be useful, even if imperfect.

4. Key Insights and Innovations

Innovation 1: The LIMO Hypothesis Reframes Reasoning from Knowledge Acquisition to Knowledge Elicitation

The paper's most fundamental intellectual move is not a new training technique but a reconceptualization of what post-training for reasoning is trying to accomplish. Prior to this work, the dominant mental model in the mathematical reasoning community treated fine-tuning as knowledge acquisition — the model needs to learn mathematical concepts, problem-solving procedures, and solution patterns from scratch through exposure to large numbers of worked examples. This was the implicit theory behind NuminaMath's 860k examples (Li et al., 2024b), MetaMath's bootstrapped question generation (Yu et al., 2024), and MAmmoTH2's web-scale instruction curation (Yue et al., 2024). The more examples the model sees, the more reasoning patterns it internalizes; scaling data is the primary lever.

The LIMO Hypothesis inverts this entirely. It proposes that in sufficiently pre-trained foundation models, the relevant mathematical knowledge is already present in the weights — encoded during pre-training through exposure to mathematical corpora, textbooks, scientific papers, and code. The challenge is not teaching but activating: showing the model how to marshal its latent knowledge into the extended, deliberative cognitive processes that constitute expert mathematical reasoning. The training examples serve not as lessons in mathematics but as cognitive templates — demonstrations of how to think through a hard problem — that teach the model to occupy the inference-time computational workspace that recent work (OpenAI, 2024; Guo et al., 2025; Snell et al., 2024) has shown is critical for complex reasoning.

This reframing has profound implications that cascade through every design choice in the paper. If the goal is activation rather than acquisition, then:

• Data quantity becomes secondary to data character. The model doesn't need to see every type of problem; it needs to see exemplary demonstrations of the cognitive moves (verification, backtracking, exploration) that constitute expert reasoning, and a few hundred such demonstrations may suffice.

• Data quality has a new, specific meaning. Quality is not merely correctness — it is the presence of visible meta-cognitive processes in the reasoning trace. This operationalizes directly into the paper's four-dimensional scoring rubric (elaborated reasoning, self-verification, exploratory approach, adaptive granularity) and explains why the "quality" in LIMO is a different construct from the "quality" in prior work that focused mainly on problem difficulty and answer correctness.

• The approach has sharp failure modes. If the model lacks the prerequisite knowledge (Premise I violated), no amount of cognitive template demonstration can help — the templates show how to think but cannot provide the concepts to think with. This is directly validated in the Qwen1.5 vs. Qwen2.5 comparison (Figure 5), where identical LIMO data produces 63.3% vs. 9.2% on AIME24. Prior work did not make or test this specific prediction because it didn't conceive of fine-tuning as bounded by pre-training knowledge completeness.

• The non-monotonic relationship between data quantity and reasoning quality becomes explicable. If large-scale fine-tuning on uncurated data introduces low-quality reasoning patterns (solutions that are correct but lack meta-cognitive richness, or worse, incorrect solutions with plausible-sounding reasoning), the training signal becomes a mixture of good and bad demonstrations of how to think. The model learns from the bad demonstrations too, potentially degrading the very capabilities it already possesses. This explains the striking result in Table 1: NuminaMath-100k fine-tuning drops the base model's average performance from 49.9% to 32.3% — more data actively damaged the model. The LIMO Hypothesis predicts this: uncurated data dilutes the quality of cognitive templates, and the dilution can be worse than no fine-tuning at all.

This reframing connects to, but fundamentally differs from, the LIMA insight (Zhou et al., 2023). LIMA showed that alignment (instruction-following, format adherence, helpfulness) can be achieved with ~1,000 high-quality examples. The LIMO Hypothesis extends this to reasoning but adds the critical conditional structure — it works if and only if pre-trained knowledge is sufficient — that LIMA did not require. Alignment tasks draw on general language understanding and social conventions that are universally present in pre-training; mathematical reasoning at the AIME level draws on specialized knowledge that may or may not be present depending on the pre-training data mixture. The conditional structure turns LIMA's "less is more" from a general observation into a testable, falsifiable theory about the relationship between pre-training, post-training, and task complexity.

This is a fundamental reframing, not an incremental refinement. It changes what researchers should optimize (from data volume to data character), how they should think about pre-training's role in downstream capabilities (as an enabler of data efficiency rather than merely a pretraining step to be overwritten), and what failure modes to expect (sharp cliffs when pre-training knowledge is insufficient, degradation from low-quality cognitive templates). The evidence supporting this reframing is distributed across the entire paper — the 4× efficiency improvement over baselines, the Qwen1.5 failure, the non-monotonic dataset size curves, the quality-level gradient in Figure 3 — but its significance is conceptual, not metric-driven.

Innovation 2: The Operationalization of Reasoning Chain Quality as a Multi-Dimensional, Computable Construct

Prior work on mathematical reasoning data curation has focused predominantly on problem quality — selecting challenging, diverse problems with verified correct answers. The dominant assumption was that any correct solution to a good problem constitutes adequate training material. The LIMO paper makes the case that this assumption is false, and in doing so, introduces a new object of optimization: reasoning chain quality as a construct separable from and independent of problem quality and answer correctness.

This matters because it explains a puzzle in the prior literature: why do some correctly-answered training examples produce better downstream models than others? The paper's L1–L5 quality gradient experiment (Figure 3) provides direct evidence. All solutions at all quality levels are correct — they arrive at the right answer. Yet models trained on L5 solutions dramatically outperform those trained on L1 solutions (the paper shows a consistent monotonic improvement with quality level on both AIME24 and MATH500). The difference is not in what answer was reached but in how the reasoning was conducted and communicated — the presence of verification steps, the exploration of alternatives, the appropriate granularity of explanation, the completeness of logical elaboration.

The paper's four-dimensional quality rubric — elaborated reasoning, self-verification, exploratory approach, adaptive granularity — operationalizes these previously qualitative dimensions into a computable score. This is not a trivial contribution. Prior to this work, there was no established framework for quantifying what makes a reasoning chain high-quality beyond surface features like length or human preference judgments that don't scale. The rubric, despite its limitations (it is keyword-based and therefore a proxy rather than a direct measurement of cognitive quality), provides a reproducible, scalable methodology for ranking reasoning chains — enabling systematic selection from large pools of candidate solutions without human annotation bottlenecks.

The innovation here is threefold:

1. It identifies specific, named dimensions of reasoning chain quality that are grounded in observation of actual expert reasoning (the authors' collaborative examination of model-generated solutions) rather than imposed from theory. These dimensions — particularly self-verification and exploratory approach — correspond to meta-cognitive processes that recent RL-based reasoning models (Guo et al., 2025; OpenAI, 2024) were found to develop spontaneously through reinforcement learning. The fact that the same dimensions emerge from both careful qualitative analysis of solution quality and from the behavior of RL-trained reasoning models suggests they capture something fundamental about expert mathematical reasoning, not merely stylistic preferences.

2. It demonstrates that these dimensions are trainable through SFT. Prior work showed that RL could induce self-verification and backtracking behaviors; LIMO shows that simply exposing the model to examples that exhibit these behaviors — through standard supervised fine-tuning — is sufficient to transfer them. This is significant because SFT is dramatically simpler and cheaper than RL-based reasoning training (which requires reward design, exploration strategies, and often millions of training episodes). If the key to unlocking sophisticated reasoning is showing the model high-quality cognitive templates rather than letting it discover them through trial and error, the practical barrier to building reasoning models drops substantially.

3. It provides a concrete answer to the field's quality-over-quantity question for the reasoning domain. LIMA showed that quality matters for alignment; LIMO specifies what "quality" means for reasoning and provides a measurement methodology. This transforms "use high-quality data" from a vague aspiration into an actionable engineering procedure with specific, computable criteria.

This is a methodological innovation with theoretical implications. It is incremental in the sense that it extends the quality-over-quantity philosophy from alignment to reasoning, but fundamental in that it identifies what quality means in this new domain — a non-obvious translation that prior work had not attempted. The evidence is Figure 3 and the consistent performance gradient it shows, establishing that the quality dimensions the rubric measures are causally related to downstream reasoning capability, not merely correlated stylistic features.

Innovation 3: Large-Scale SFT Data Can Actively Damage Reasoning — A Diagnostic Negative Result with Theoretical Implications

One of the paper's most striking results is the catastrophic degradation observed when fine-tuning Qwen2.5-32B-Instruct on NuminaMath-100k: the base model's average performance across all benchmarks drops from 49.9% to 32.3%, with AIME24 falling from 16.5% to 6.5% and MATH500 from 79.4% to 59.2% (Table 1). This is not a failure to improve — it is active regression below the untrained baseline. The model is worse at mathematics after seeing 100,000 additional mathematical training examples than it was before fine-tuning.

This result is significant not as a performance number but as a diagnostic finding that challenges core assumptions in the reasoning post-training literature. The standard mental model is that SFT on reasoning data is at worst neutral — if the data is noisy or low-quality, the model might not improve as much as it could, but it shouldn't get worse. The NuminaMath-100k result falsifies this assumption. Something about training on large quantities of uncurated or semi-curated mathematical solutions is actively destructive to the model's pre-existing reasoning capabilities.

The LIMO Hypothesis provides a mechanistic explanation: the model's pre-trained knowledge already enables competent mathematical reasoning (49.9% average). Fine-tuning on low-quality reasoning chains — solutions that are correct but lack meta-cognitive richness, or that contain errors masked by plausible-sounding reasoning — overwrites the model's internal reasoning patterns with inferior ones. The model learns from the bad demonstrations and degrades its own capabilities in the process. This is analogous to the catastrophic forgetting phenomenon in continual learning, but with a specific mechanism: the model is not forgetting facts but rather replacing effective reasoning strategies with less effective ones absorbed from the training data.

The OpenThoughts-114k baseline (58.3% average) provides a partial recovery — its solutions follow a structured reasoning format generated by DeepSeek-R1, so they contain some of the meta-cognitive quality that the LIMO rubric values — but still underperforms LIMO by ~20 percentage points despite having 140× more data. This suggests that the problem is not merely correctness but the density of high-quality cognitive templates in the training mixture. Even a dataset with decent average quality can underperform a tiny dataset with uniformly excellent quality if the large dataset contains enough low-quality examples to partially overwrite the beneficial patterns.

This finding has direct theoretical implications for the ongoing memorization-vs-generalization debate in LLM reasoning (Mirzadeh et al., 2024; Zhang et al., 2024; Chu et al., 2025). These prior works argued that SFT leads to memorization rather than generalization for reasoning tasks. The LIMO results suggest a more nuanced picture: SFT can produce models that either memorize or generalize, depending on the character of the training data. Training on large quantities of correct-but-cognitively-thin solutions may indeed produce memorizers — models that have pattern-matched problem templates without internalizing the underlying reasoning processes. Training on a small number of cognitively rich solutions may produce generalizers — models that have learned how to reason from exemplary demonstrations and can apply those reasoning patterns to novel problems. The out-of-distribution results (45.8% absolute improvement across diverse benchmarks, including cross-lingual transfer to Chinese exams) support this interpretation: LIMO's generalization is strong in settings where a pure memorizer would fail.

This reframes the debate: the problem may not be SFT as a training paradigm but rather the community's data curation practices. If SFT memorizes, it may be because we are feeding it memorizable data — large volumes of solution-answer pairs that encourage the model to learn surface-level associations rather than deep reasoning processes. The LIMO results demonstrate that SFT can produce strong generalization when the training data is selected to emphasize cognitive process over answer-production.

This is a fundamental diagnostic finding rather than an incremental improvement. It identifies a failure mode (SFT-induced reasoning degradation) that was not widely recognized in the reasoning community, proposes a mechanistic explanation grounded in the LIMO Hypothesis, and provides both negative evidence (NuminaMath-100k regression) and positive evidence (LIMO generalization) that the quality of cognitive templates in training data determines whether fine-tuning helps or harms pre-existing reasoning capabilities. The evidence is Table 1 and the consistent pattern across benchmarks.

Innovation 4: Pre-Training Knowledge Completeness as the Binding Constraint on Data-Efficient Reasoning — An Empirically Validated Boundary Condition

The paper's Qwen1.5 vs. Qwen2.5 comparison (Figure 5, Section 6.3.3) is methodologically simple — take two models with identical architecture (32B parameters, same model family) but different generations of pre-training data quality, fine-tune both on the identical 800 LIMO examples, and compare — but its implications are profound. Qwen2.5-32B-Instruct achieves 63.3% on AIME24; Qwen1.5-32B-Chat achieves 9.2%. The 54.1 percentage point gap isolates the contribution of pre-training knowledge to data-efficient reasoning elicitation.

This result does several things simultaneously:

It validates the conditional structure of the LIMO Hypothesis. The hypothesis does not claim that 800 high-quality examples can elicit reasoning from any model — it claims they can elicit reasoning from models where prerequisite knowledge is latent in the parameters. The Qwen1.5 failure is predicted by the hypothesis and provides strong negative evidence: if pre-training knowledge is insufficient, even the best cognitive templates cannot compensate. The model lacks the mathematical vocabulary to understand the templates, so the fine-tuning signal has nothing to activate.

It establishes a concrete empirical bound on data efficiency. The Qwen1.5 result shows what "insufficient pre-training knowledge" looks like quantitatively: 9.2% on AIME24 is marginally above random guessing for a 15-question multiple-choice competition. This is not a model that partially benefits from LIMO data but just needs more of it — it is a model for which the LIMO approach fundamentally fails. The sharpness of the boundary (54.1 points of AIME24 accuracy, 30.4 points of MATH500 accuracy between model generations of the same architecture) suggests that pre-training knowledge acts as a threshold — below some level of mathematical content in pre-training, the knowledge elicitation strategy is infeasible regardless of exemplar quality.

It reframes the relationship between pre-training and fine-tuning. Traditionally, pre-training and fine-tuning are treated as sequential but largely independent stages: pre-training provides a general language model, fine-tuning specializes it. The LIMO results suggest a more integrated relationship: the character of pre-training determines the feasibility regime of fine-tuning strategies. A model with rich mathematical pre-training can be fine-tuned with extreme data efficiency; a model without it requires a fundamentally different approach (likely much more data or a different training paradigm entirely). This has practical implications for model selection — given a target reasoning task, the choice of which pre-trained model to fine-tune is not merely about starting accuracy but about whether the model is in the regime where data-efficient activation is possible at all.

It connects to the inference-time computation narrative. The paper frames the synergy between pre-trained knowledge and inference-time computation as the enabling condition for data efficiency (Section 1). The Qwen1.5 result clarifies what happens when one half of the synergy is missing: without rich pre-trained mathematical knowledge, the model has nothing to "unpack" during extended deliberation. The cognitive templates in the LIMO data show how to think, but if the model lacks the concepts to think about, the templates are vacuous. This explains why the same approach produces such dramatically different results across model generations despite identical architecture — Qwen2.5's pre-training included substantially more mathematical content, providing the conceptual substrate that the LIMO data activates.

This is a fundamental finding about scaling dynamics rather than a methodological contribution. It identifies a critical variable (pre-training knowledge completeness) that prior data-efficiency work had not systematically varied, demonstrates that it acts as a binding constraint rather than a continuous moderator, and provides a clean experimental design (same architecture, same data, different pre-training generations) that isolates the effect. The evidence is Figure 5, with the 54.1-point AIME24 gap providing the quantitative anchor. The broader implication — that data-efficient reasoning elicitation is not a universal property of good fine-tuning data but an interaction between data quality and model capability — has significant consequences for how the field thinks about reproducing and extending these results.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation benchmarks are AIME24 (15 competition problems from the 2024 American Invitational Mathematics Examination), MATH500 (500 problems from Hendrycks et al., 2021), and AMC23 (problems from the 2023 American Mathematics Competitions). Out-of-distribution evaluation adds OlympiadBench, CHMath (2024 Chinese High School Mathematics League), Gaokao (China's 2024 College Entrance Exam), Kaoyan (Chinese Graduate School Entrance Exams), GradeSchool (a newly constructed elementary mathematics benchmark), Minerva (undergraduate-level STEM), and GPQA (graduate-level Q&A). The paper does not specify exact sizes for all OOD benchmarks, but notes that OlympiadBench, Gaokao, Kaoyan, GradeSchool, MinervaMath, and GPQA are "larger" (greedy decoding with single sample) while AIME24, AMC23, and CHMath have "less than 50 problems" each (4 samples with temperature=0.6). The training data is drawn from NuminaMath-CoT, DeepScaleR, AIME historical (before 2024), MATH, and Chinese examination papers, totaling "tens of millions" with exact deduplication against all test benchmarks via n-gram matching.

• Base model(s). Qwen2.5-32B-Instruct (Qwen et al., 2025) serves as the primary base model — a 32-billion-parameter instruction-tuned model from the Qwen2.5 family with enhanced mathematical and code-related pre-training content. For the pre-training knowledge ablation (Section 6.3.3), Qwen1.5-32B-Chat (Team, 2024a) is used — identical architecture but earlier-generation pre-training data with weaker mathematical coverage. For the model size scaling experiment (Section 6.3.4), the paper fine-tunes Qwen2.5-Instruct variants at 3B, 7B, 14B, 32B, and 72B parameter scales. The choice of Qwen2.5-32B-Instruct is motivated by the LIMO Hypothesis's requirement of comprehensive pre-trained domain knowledge (Premise I).

• Metrics. The primary metric across all benchmarks is pass@1 — the fraction of problems for which the model's first generated answer matches the ground truth. For larger benchmarks (MATH500, OlympiadBench, Gaokao, Kaoyan, GradeSchool, MinervaMath, GPQA), pass@1 is computed using greedy decoding (single sample, temperature=0). For smaller benchmarks with fewer than 50 problems (AIME24, AMC23, CHMath), the paper generates 4 samples per problem at temperature=0.6 and computes the unbiased pass@1 estimator following Chen et al. (2021). Evaluation uses "rule-based evaluations for numerical answers and an LLM-based evaluator for complex answer formats." All evaluations use a 32,768 token maximum output length.

• Baselines. The paper compares against:

  • OpenAI-o1-preview (OpenAI, 2024): a large-scale proprietary model with advanced mathematical reasoning from extended chain-of-thought training.
  • QwQ-32B-Preview (Team, 2024b): a 32B open-weight model specifically designed for mathematical problem-solving with strong reasoning capabilities.
  • Qwen2.5-32B-Instruct (the untrained base model): serves as the pre-fine-tuning baseline.
  • OpenThoughts-114k (Team, 2025a): a synthetic reasoning dataset of 114k examples covering mathematics, science, coding, and puzzles, with solutions following a structured reasoning format generated by DeepSeek-R1. Fine-tuned on Qwen2.5-32B-Instruct using identical training configuration.
  • NuminaMath-100k: a randomly selected 100k subset of NuminaMath-CoT (Li et al., 2024b), featuring mathematical problems from Chinese high school exercises to international olympiad competitions with Chain of Thought solutions. Fine-tuned on Qwen2.5-32B-Instruct using identical training configuration.

  The last two baselines are particularly important because they use the exact same base model and training recipe as LIMO, isolating the effect of training data characteristics (quality, curation, size) from model architecture and optimization choices.

• Generation budget / compute accounting. The paper does not report FLOPs or GPU-hours for training. Instead, it measures data efficiency through training set size — the primary axis of comparison is the number of SFT examples used: 800 for LIMO versus 100k–114k for baselines. Training configuration is held constant across all fine-tuned baselines (same DeepSpeed ZeRO-3, FlashAttention-2, learning rate 5.0e-6, cosine decay, no warmup, 15 epochs, batch size 64, sequence length 16,384). The paper is measuring data efficiency (performance per training example) rather than compute efficiency (performance per FLOP). For evaluation inference, the paper distinguishes between greedy decoding (1 sample per problem) for large benchmarks and 4-sample pass@1 estimation for small benchmarks, but does not provide total inference compute budgets, as the focus is on training data requirements.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. No confidence intervals, standard errors, or significance tests are reported for any of the main results in Table 1, Figures 3–8, or the ablation studies. The dataset deduplication protocol (n-gram matching against all evaluation benchmarks) is described but the specific threshold or n-gram length is not stated. The paper relies on the fact that most benchmarks have sufficient size (MATH500 at 500 problems, OlympiadBench and others at hundreds) to provide reasonable point estimates, though exact sizes for several OOD benchmarks are not provided. The AIME24 benchmark has only 15 problems, making its pass@1 estimates high-variance even with the 4-sample unbiased estimator.

Main Quantitative Results

In-Domain Performance (Table 1, top section)

The central result is that LIMO achieves state-of-the-art performance on in-domain mathematical reasoning benchmarks despite using only 800 training examples — 1% or less of the training data used by comparison methods.

AIME24: LIMO achieves 63.3% accuracy, compared to QwQ-32B-Preview at 50.0%, OpenAI-o1-preview at 44.6%, OpenThoughts-114k at 50.2%, NuminaMath-100k at 6.5%, and the untrained base model at 16.5%. LIMO's improvement over the base model is 46.8 absolute percentage points. Over the best prior open-weight model (QwQ-32B-Preview), LIMO improves by 13.3 points. Over the best SFT baseline with the same base model (OpenThoughts-114k), LIMO improves by 13.1 points despite using 1/142.5 the training data.

MATH500: LIMO achieves 95.6% accuracy, compared to QwQ-32B-Preview at 89.8%, OpenAI-o1-preview at 85.5%, OpenThoughts-114k at 80.6%, NuminaMath-100k at 59.2%, and the base model at 79.4%. LIMO's MATH500 score represents near-saturation of this benchmark — the remaining errors are on 22 of 500 problems.

AMC23: LIMO achieves 96.3% accuracy, compared to QwQ-32B-Preview at 83.6%, OpenAI-o1-preview at 81.8%, OpenThoughts-114k at 80.5%, NuminaMath-100k at 40.6%, and the base model at 64.0%. This is LIMO's highest benchmark score, indicating mastery of this competition format.

The NuminaMath-100k degradation pattern: Perhaps the most striking in-domain finding is not LIMO's absolute performance but the catastrophic degradation of the NuminaMath-100k baseline. On all three in-domain benchmarks, NuminaMath-100k performs worse than the untrained base model: AIME24 drops from 16.5% (base) to 6.5%, MATH500 from 79.4% to 59.2%, and AMC23 from 64.0% to 40.6%. This is a consistent 10–23 point degradation across all in-domain benchmarks. OpenThoughts-114k partially recovers — outperforming the base model on AIME24 (50.2% vs. 16.5%) and AMC23 (80.5% vs. 64.0%) but slightly worse on MATH500 (80.6% vs. 79.4%) — but remains substantially below LIMO on all three measures. These results demonstrate a non-monotonic relationship between data quantity and reasoning quality: more data can actively harm performance when the data contains low-quality reasoning chains.

Out-of-Distribution Generalization (Table 1, bottom section)

LIMO demonstrates strong generalization across seven out-of-distribution benchmarks spanning different competition formats, languages, and disciplines.

OlympiadBench: LIMO scores 67.6% versus QwQ-32B-Preview at 58.5%, OpenAI-o1-preview at 52.1%, OpenThoughts-114k at 56.3%, NuminaMath-100k at 36.7%, and the base model at 45.3%. The improvement over the base model is 22.3 percentage points; over the best prior model (QwQ-32B-Preview) it is 9.1 points.

Chinese-language benchmarks (CHMath, Gaokao, Kaoyan): These are particularly important for the generalization claim because they test cross-lingual transfer. LIMO achieves 84.2% on CHMath (vs. 74.1% for OpenThoughts-114k and 68.5% for QwQ-32B-Preview), 91.1% on Gaokao (vs. 80.1% for QwQ-32B-Preview), and 83.9% on Kaoyan (vs. 70.3% for QwQ-32B-Preview). On CHMath, the improvement over the untrained base model is 56.9 percentage points (27.3% → 84.2%) — the largest absolute gain on any benchmark. The NuminaMath-100k baseline again shows degradation: CHMath drops from 27.3% to 11.2%, Gaokao from 72.1% to 49.4%, Kaoyan from 48.2% to 32.7%.

GradeSchool: LIMO achieves 76.2% versus QwQ-32B-Preview at 63.8% and the base model at 56.7%. This benchmark, constructed by the authors from recent elementary mathematics problems, serves as an additional distribution shift test.

Minerva (undergraduate STEM): LIMO scores 52.2% versus OpenAI-o1-preview at 47.1% and QwQ-32B-Preview at 39.0%. The improvement over the base model (41.2%) is 11.0 points — a more modest gain than on competition math, consistent with the hypothesis that LIMO's reasoning patterns specialize toward the type of mathematical deduction emphasized in the training data rather than the broader STEM knowledge required by Minerva.

GPQA (graduate-level Q&A): LIMO achieves 70.7% versus OpenAI-o1-preview at 73.3% and QwQ-32B-Preview at 65.1%. This is the only benchmark where LIMO does not achieve the top score, trailing o1-preview by 2.6 points. The improvement over the base model (48.0%) is 22.7 points.

Overall out-of-distribution average: LIMO averages 78.1% across all 10 benchmarks (3 in-domain + 7 OOD), compared to QwQ-32B-Preview at 66.9%, OpenAI-o1-preview at 61.1%, OpenThoughts-114k at 58.3%, NuminaMath-100k at 32.3%, and the base model at 49.9%. The total improvement over the base model is 28.2 percentage points. The paper characterizes the OOD improvement as a "45.8% absolute improvement across diverse benchmarks" (Section 1), though the basis for this specific number is unclear — 28.2 points overall average improvement does not equal 45.8% for any obvious subset, suggesting this figure may reference the improvement on a specific subset of benchmarks or use a different aggregation method not detailed in the text.

The generalization pattern: LIMO's gains are largest on competition mathematics benchmarks (AIME24: +46.8 over base; CHMath: +56.9; AMC23: +32.3) and more modest on benchmarks requiring broader knowledge (Minerva: +11.0; GPQA: +22.7). This is consistent with the LIMO Hypothesis: the fine-tuning activates mathematical reasoning capabilities that were latent from pre-training, but does not inject new domain knowledge. Benchmarks requiring physics, chemistry, or biology knowledge (Minerva) benefit less because the LIMO data doesn't contain those domains.

Impact of Reasoning Chain Quality (Figure 3, Section 6.3.1)

To isolate the effect of solution quality from problem quality, the paper selects 500 problems from the LIMO dataset that each have multiple correct solutions generated by diverse models, categorizes solutions into five quality levels (L1–L5, with L5 highest) using the rule-based scoring system from Section 3.1.2, fine-tunes Qwen2.5-32B-Instruct on each quality level separately, and evaluates on AIME24 and MATH500.

AIME24 results by quality level: The paper's Figure 3 shows a "clear correlation between reasoning quality and model performance" with a consistent monotonic improvement from L1 to L5. Exact numbers are not provided in the text, but the figure description states that "L5-trained models achieve highest results on both AIME24 and MATH500, with performance decreasing consistently with each quality level." The "substantial gap between L5 and L1 solutions" demonstrates that even when problems are identical and all solutions are correct, solution quality dramatically affects downstream model capability.

MATH500 results by quality level: The same monotonic pattern holds, with L5 outperforming L4, L4 outperforming L3, and so on. The key insight: the scoring rubric's four dimensions (elaborated reasoning, self-verification, exploratory approach, adaptive granularity) successfully identify solutions that produce better downstream models, validating the rubric as a proxy for genuine cognitive quality rather than merely stylistic variation.

Impact of Question Difficulty (Figure 4, Section 6.3.2)

To test the hypothesis that "more challenging problems foster complex reasoning chains and enhanced knowledge integration," the paper constructs three datasets of 500 problems each with increasing difficulty: Simple-500 (MATH levels 1–2), Complex-500 (MATH levels 3–5), and Advanced-500 (AIME problems). DeepSeek-R1 generates high-quality solutions for each set, and Qwen2.5-32B-Instruct is fine-tuned on each separately. The difficulty gradient is verified by evaluating various LLMs and observing "declining accuracy and increasing solution length across these sets."

AIME24 results by training difficulty: Models trained on harder problems perform substantially better on AIME24. The paper states that "modifying problem selection alone leads to 16% accuracy improvement on AIME2024, reaching 51.5%" for the Advanced-500 model. The exact numbers for Simple-500 and Complex-500 are not reported in the text, but Figure 4 shows a clear upward trajectory: Simple-500 is lowest, Complex-500 is intermediate, and Advanced-500 is highest on AIME24. The 51.5% figure for Advanced-500 is notable because it's achieved using only 500 training examples — fewer than LIMO's 800 — yet reaches substantial AIME performance, suggesting problem difficulty is a powerful lever even independent of other quality dimensions.

MATH500 results by training difficulty: The Advanced-500 model achieves 91.2% on MATH500 "despite no in-domain training data" — a remarkable cross-benchmark transfer result. The Complex-500 model shows intermediate performance, and the Simple-500 model shows the lowest. The paper interprets the MATH500 performance of the Advanced-500 model as evidence that "reasoning improvements from increased problem difficulty generalize across datasets." The model trained only on AIME-level problems can solve MATH problems at 91.2% accuracy without ever having seen MATH-level training examples, suggesting transfer of general reasoning capability rather than problem-type memorization.

Pre-Training Knowledge Ablation (Figure 5, Section 6.3.3)

This experiment tests Premise I of the LIMO Hypothesis by comparing Qwen2.5-32B-Instruct and Qwen1.5-32B-Chat — identical architecture (32B parameters, same model family) but different pre-training generations with substantially different mathematical content — fine-tuned on the identical 800 LIMO examples.

AIME24: LIMO (Qwen2.5-based) achieves 63.3% versus Qwen1.5-based at 9.2% — a 54.1 percentage point gap. The Qwen1.5-based model's 9.2% is worse than the untrained Qwen2.5 base model's 16.5%, demonstrating that even the same high-quality LIMO data cannot compensate for insufficient pre-trained mathematical knowledge.

MATH500: LIMO achieves 95.6% versus Qwen1.5-based at 65.2% — a 30.4 percentage point gap. The Qwen1.5-based model's MATH500 performance (65.2%) is substantially below the Qwen2.5 base model's untrained performance (79.4%), again showing that the LIMO data cannot rescue a model lacking the prerequisite knowledge base.

These results validate the conditional structure of the LIMO Hypothesis: data-efficient reasoning elicitation is not a property of the training data alone but an interaction between data quality and model capability. The sharp boundary — 54.1 points of AIME24 accuracy between model generations of identical architecture — suggests that pre-training knowledge acts as a threshold condition rather than a continuous moderator.

Model Size Scaling (Figure 6, Section 6.3.4)

The paper fine-tunes Qwen2.5-Instruct models at 3B, 7B, 14B, 32B, and 72B parameter scales on the identical 800 LIMO examples to characterize how reasoning capability scales with model size under data-constrained fine-tuning.

AIME24 scaling: Performance rises from 2.5% (3B) to 7B (exact number not provided), to 14B (exact number not provided), to 63.3% (32B), to 68.3% (72B). The paper notes a "marked increase with model size," though the jump from 14B to 32B appears to be the steepest portion of the curve (the text does not provide the 7B and 14B exact values, but Figure 6 shows the trajectory).

MATH500 scaling: Performance rises from a value in the range of roughly 80–85% at 3B (exact number not provided) to 95.6% (32B) to 94.8% (72B). The paper observes that "gains are less dramatic" on MATH500 compared to AIME24 and notes a "potential saturation point, where increasing the number of parameters further may yield diminishing returns." The 72B model scores slightly below the 32B model on MATH500 (94.8% vs. 95.6%), though the difference is marginal and likely within noise for a 500-problem benchmark.

The paper interprets the AIME24 scaling pattern as evidence that "larger models are better able to handle the deep reasoning required by competition-level math problems," while the MATH500 saturation suggests that "even smaller models can achieve high accuracy on easier benchmarks." The non-monotonic behavior at 72B on MATH500 (slightly below 32B) is attributed to potential overfitting or optimization differences rather than a genuine reversal of scaling trends.

Sample Efficiency: How Much Data Is Actually Needed? (Figures 7–8, Section 6.3.5)

The paper explores the lower bound of data requirements by constructing datasets of varying sizes — 400, 800, 1,200, 1,600, and 2,000 questions — selected by ranking all 2,125 problems in the LIMO-Pool by solution quality score and taking the top-k subsets. Each dataset is used to fine-tune Qwen2.5-32B-Instruct with the identical training recipe.

AIME24 by dataset size (Figure 7):

• LIMO-400: 57.5% (versus base model's 16.5% and LIMO-800's 63.3%)
• LIMO-800: 63.3%
• LIMO-1200: 64.2% (+0.9 over LIMO-800)
• LIMO-1600: exact number not provided, above LIMO-800
• LIMO-2000: 69.6% (best overall)

The key insight: "Fine-tuning with just 400 samples yields dramatic improvement over the base model, increasing AIME24 accuracy from 16.5 to 57.5." Even at 400 examples — half of LIMO's already-tiny dataset — the model achieves near-competitive performance, suggesting that the critical mass of high-quality exemplars needed to activate reasoning is surprisingly small. The paper observes "diminishing returns beyond 800 samples, with only marginal increases between LIMO-800 and LIMO-1200 (+0.9 on AIME24, -0.2 on MATH500), suggesting an early plateau effect."

MATH500 by dataset size (Figure 8):

• LIMO-400: 94.8% (versus base model's 79.4%)
• LIMO-800: 95.6%
• LIMO-1200: 95.4% (-0.2 from LIMO-800, described as essentially flat)
• LIMO-1600: exact number not provided
• LIMO-2000: 95.8% (marginally best)

The paper interprets the near-flat MATH500 performance between 400 and 2,000 examples as evidence that "beyond a certain threshold, additional data contributes less significantly to fine-tuning gains." The difference between LIMO-400 (94.8%) and LIMO-2000 (95.8%) is only 1.0 percentage point on MATH500, versus 12.1 points on AIME24 (57.5% → 69.6%), suggesting that easier benchmarks saturate more quickly with respect to training data.

Joint interpretation: The early plateau is consistent with the LIMO Hypothesis — if the training data's role is to demonstrate cognitive templates (how to reason) rather than to teach mathematical content, then once the model has seen enough examples to internalize the core patterns (verification, exploration, structured elaboration), additional examples provide redundancy rather than new information. The fact that 400 examples capture most of the gain suggests that the "vocabulary" of meta-cognitive moves demonstrated in the LIMO data is relatively compact — perhaps a few hundred exemplars suffice to cover the space of useful reasoning strategies.

Ablation Studies and Robustness Checks

• Solution quality level (L1–L5) as a continuous moderator of downstream performance (Figure 3): Models trained on L5 (highest quality) solutions consistently outperform those trained on L1 (lowest quality) solutions on both AIME24 and MATH500, with a monotonic gradient across the five levels. This validates the paper's scoring rubric — the quality dimensions it measures (elaborated reasoning, self-verification, exploratory approach, adaptive granularity) are causally linked to downstream model capability, not merely correlational features. The experiment controls for problem difficulty by using the same 500 problems across all quality levels, isolating solution quality as the independent variable.

• Problem difficulty as an independent lever for reasoning improvement (Figure 4): Training on Advanced-500 (AIME problems) yields substantially higher AIME24 performance (51.5%) than training on Complex-500 or Simple-500, and achieves 91.2% on MATH500 despite zero in-domain MATH training data. This demonstrates that problem difficulty is a separable axis of data quality from solution quality — harder problems force the generation of richer reasoning chains, and training on those chains transfers to other benchmarks. The Simple-500 and Complex-500 models provide the comparison points, showing a consistent difficulty gradient.

• Pre-training knowledge completeness as a binary enabling condition (Figure 5): The 54.1-point AIME24 gap between Qwen2.5-32B-Instruct (63.3%) and Qwen1.5-32B-Chat (9.2%) — same architecture, same 800 LIMO examples, same training recipe — demonstrates that pre-training mathematical content acts as a threshold condition. The Qwen1.5-based model's AIME24 performance (9.2%) is below the Qwen2.5 base model's untrained performance (16.5%), indicating that the LIMO data cannot overcome fundamentally insufficient pre-trained knowledge. This is the paper's key negative result supporting the conditional structure of the LIMO Hypothesis.

• Model size scaling with fixed tiny training data (Figure 6): Performance on AIME24 increases monotonically with model size from 3B (2.5%) to 72B (68.3%), with the steepest gains between 14B and 32B. MATH500 shows diminishing returns and potential saturation at 32B (95.6%) with slight regression at 72B (94.8%). This ablation characterizes how the LIMO approach interacts with model scale — larger models extract more reasoning capability from the same 800 examples, but the gains are benchmark-dependent and may saturate.

• Dataset size and diminishing returns (Figures 7–8): Reducing the dataset from 2,125 (full LIMO-Pool) to 2,000, 1,600, 1,200, 800, and 400 examples shows that AIME24 performance increases from 57.5% (400) to 69.6% (2,000) with an early plateau around 800 examples, while MATH500 performance is nearly flat from 400 (94.8%) to 2,000 (95.8%). The key finding is that 400 examples already captures most of the reasoning improvement — the additional 1,600 examples add only ~12 points on AIME24 and ~1 point on MATH500, suggesting that the core reasoning patterns are demonstrated in the first few hundred highest-quality exemplars.

• Large-scale uncurated data causing catastrophic degradation (Table 1, NuminaMath-100k rows): The NuminaMath-100k baseline (a random 100k subset of NuminaMath-CoT) underperforms the untrained base model on all 10 benchmarks, with the average dropping from 49.9% to 32.3%. This is a negative result of critical importance: it demonstrates that large-scale fine-tuning on uncurated or insufficiently curated mathematical reasoning data can actively destroy pre-existing reasoning capabilities. The paper does not ablate which property of NuminaMath-100k causes the degradation (is it incorrect solutions? missing meta-cognitive features? domain mismatch?), leaving the mechanism unspecified.

• OpenThoughts-114k as a partial recovery (Table 1): The OpenThoughts-114k baseline (solutions generated by DeepSeek-R1 with structured reasoning format) recovers to 58.3% average — above the base model (49.9%) but substantially below LIMO (78.1%). This ablation demonstrates that solution format and generation source matter: DeepSeek-R1-generated solutions are higher quality than NuminaMath-CoT solutions, but training on 114k of them still underperforms training on 800 carefully selected exemplars, suggesting that even "good" data at scale contains sufficient noise to partially overwrite beneficial patterns.

Critical Assessment

Does the paper demonstrate that 800 examples can elicit competition-level reasoning? Yes, with strong evidence, but under specific conditions.

The performance numbers in Table 1 are genuinely impressive: 63.3% on AIME24 and 95.6% on MATH500 from 800 training examples. These are not cherry-picked results from a single run — they hold across 10 diverse benchmarks, with consistent patterns across difficulty levels, languages, and disciplines. The comparison against OpenThoughts-114k and NuminaMath-100k (same base model, same training recipe, 125–142× more data) isolates data curation as the causal variable, and the performance gaps are large: 20–46 absolute percentage points of average accuracy. The 400-example ablation (Figure 7) shows that even at 400 examples, AIME24 reaches 57.5% — still competitive — confirming that the effect holds at even smaller data scales.

However, the paper's framing as "1% of the training data required by prior approaches" (Section 1, 6.2) is somewhat imprecise. The comparison is against specific prior approaches (NuminaMath-100k, OpenThoughts-114k), not against all prior mathematical reasoning models. Models like DeepSeek-R1 (Guo et al., 2025) and o1 (OpenAI, 2024) used reinforcement learning rather than SFT, making direct data comparisons difficult. The "1%" claim holds against the specific SFT baselines tested, but the broader claim about the field's data requirements should be understood as "1% of what these specific SFT baselines used" rather than "1% of what the field universally requires."

Is the LIMO Hypothesis genuinely supported by the evidence, or merely consistent with it?

The paper's central theoretical claim is that data-efficient reasoning elicitation requires two conditions: sufficient pre-trained knowledge (Premise I) and high-quality cognitive templates (Premise II). The Qwen1.5 vs. Qwen2.5 comparison (Figure 5) provides strong evidence for Premise I — the 54.1-point AIME24 gap when pre-training quality is the only variable is a clean experimental result. The solution quality gradient (Figure 3) provides evidence for Premise II — the L1–L5 monotonic improvement when solution quality is the only variable demonstrates that reasoning chain quality matters independently.

However, the evidence does not fully establish the necessity of these conditions as framed by the hypothesis — only their sufficiency (when both are present, performance is high) and their individual contribution (when either is absent, performance degrades). The hypothesis claims that "sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes" — the word "can" is important. The paper demonstrates that they did emerge under these conditions for Qwen2.5-32B-Instruct. Whether they cannot emerge under other conditions (e.g., with 100k examples of moderately high quality, or with 800 examples of different design) is not fully tested.

The NuminaMath-100k degradation partially addresses this by showing a case where more data with lower quality produces worse results, supporting the claim that quality matters more than quantity. But it doesn't test the boundary: could a dataset of, say, 10,000 carefully curated examples (with intermediate quality between LIMO and NuminaMath) produce similar or better results? Could the same 800 LIMO examples work on a model with intermediate pre-training quality (between Qwen1.5 and Qwen2.5)? The hypothesis predicts specific answers to these questions, but they are not experimentally verified.

The rule-based quality scoring system is simultaneously a strength and a significant limitation.

The four-dimensional scoring rubric (Section 3.1.2) provides a transparent, reproducible method for ranking reasoning chains — this is a genuine methodological contribution, as prior work lacked any standardized approach to quantifying reasoning chain quality. The L1–L5 experiment (Figure 3) validates that the rubric's rankings correlate with downstream model performance, which is the right validation.

But the rubric has clear limitations that the paper does not fully address:

1. It is entirely lexical. The quality dimensions are operationalized through keyword frequencies normalized by length. A solution can score highly by being long and sprinkling in "therefore," "check," and "perhaps" without genuinely exhibiting elaborated reasoning, self-verification, or exploratory thinking. The paper provides no validation that high-scoring solutions actually exhibit these cognitive qualities beyond the keyword proxies. Human evaluation of a sample of solutions across score levels would substantially strengthen the claim that the rubric measures genuine quality rather than surface features.

2. The weighting (30/20/25/25) is arbitrary. No sensitivity analysis is provided for the weight choices. Would different weightings produce meaningfully different dataset compositions and downstream performance? The paper doesn't explore this. The weight choice implicitly prioritizes solution length (30%) over self-verification (20%), but this prioritization is not theoretically justified.

3. The scoring is applied to model-generated solutions. This creates an implicit dependency on the quality of the generating models (DeepSeek R1, DeepSeek-R1-Distill-Qwen-32B, QwQ-32B). If these models share systematic weaknesses (certain types of reasoning they don't exhibit, or stylistic patterns that inflate keyword scores without corresponding cognitive quality), the LIMO dataset inherits those weaknesses. The paper doesn't analyze whether the three generating models produce qualitatively different types of errors or whether certain mathematical domains are systematically underrepresented in their high-scoring solutions.

4. "Elaborated Reasoning" is proxied by solution length (30% weight). This is the most concerning proxy. Solution length correlates with reasoning elaboration, but the correlation is imperfect — a solution can be long because it's verbose, repetitive, or inefficient, not because it's thorough. The 30% weight on length means the rubric strongly favors longer solutions regardless of whether the additional length represents genuine cognitive elaboration. This could systematically bias the dataset toward verbose reasoning styles and potentially penalize concise-but-complete solutions that might actually be better training examples.

The benchmark evaluation has several methodological gaps.

No statistical significance or confidence intervals. For a paper making strong quantitative claims (63.3% vs. 50.0% on AIME24, etc.), the absence of any uncertainty quantification is a significant omission. AIME24 has only 15 problems, making the pass@1 estimate high-variance even with the 4-sample unbiased estimator — a few lucky or unlucky guesses could shift the result by several percentage points. MATH500 (500 problems) is more stable, but the Chinese-language benchmarks (CHMath, Gaokao, Kaoyan, GradeSchool) have unspecified sizes, making it impossible to assess result reliability. The fact that the 72B model slightly underperforms the 32B model on MATH500 (94.8% vs. 95.6%) — a negative scaling result — might simply be noise in a 500-problem benchmark, but without confidence intervals, this cannot be assessed.

Benchmark sizes for several OOD tests are not reported. The paper states that for "smaller benchmarks (less than 50 problems: AIME24, AMC23, and CHMath)," it uses 4-sample pass@1 estimation, and for "larger benchmarks" it uses greedy decoding. But the exact sizes of OlympiadBench, Gaokao, Kaoyan, GradeSchool, MinervaMath, and GPQA are never specified. This matters because some of these could be quite small — GradeSchool is described as "our new elementary mathematics benchmark" with no size given — and pass@1 on a small benchmark with greedy decoding is high-variance. The paper's strong OOD generalization claims depend on these numbers being reliable.

The "45.8% absolute improvement across diverse benchmarks" claim is ambiguous. Section 1 states that LIMO exhibits "a 45.8% absolute improvement across diverse benchmarks" in out-of-distribution generalization. The average performance in Table 1 shows LIMO at 78.1% versus the base model at 49.9% — a 28.2-point improvement. The 45.8% figure does not match any obvious subset of benchmarks in Table 1. The number may refer to a specific aggregation (e.g., improvement on the subset of OOD benchmarks only, or the average of per-benchmark improvements rather than the improvement of averages), but the paper does not specify. This ambiguity weakens what should be a headline result.

No comparison to RL-based reasoning models beyond o1-preview and QwQ-32B-Preview. The paper frames its contribution partly in relation to inference-time scaling approaches (DeepSeek-R1, o1) that use RL to induce reasoning behaviors. But the baselines don't include DeepSeek-R1 itself (only its distilled version, used for data generation), nor do they include any direct RL-based fine-tuning baseline on Qwen2.5-32B-Instruct. Such a comparison would test whether SFT on 800 high-quality examples can match what RL achieves with orders of magnitude more computation. The absence of this comparison means the paper demonstrates SFT efficiency relative to other SFT approaches, but does not establish SFT's competitiveness with RL-based reasoning training.

The data generation pipeline creates a dependency on frontier reasoning models.

The LIMO dataset's solutions are generated by DeepSeek R1, DeepSeek-R1-Distill-Qwen-32B, and QwQ-32B — all models that were themselves trained with substantial resources (RL for R1, distillation for the distill, separate development for QwQ). The LIMO approach is therefore not a fully independent method for eliciting reasoning from a base model — it depends on having access to other, more capable reasoning models to generate the high-quality exemplars. This is a form of distillation, even if the paper doesn't frame it that way: the reasoning patterns are transferred from frontier models to Qwen2.5-32B-Instruct through the medium of carefully selected solution traces.

This has implications for the paper's practical claims. The approach "works" in the sense that 800 examples from strong reasoning models significantly improve a weaker base model. But it doesn't demonstrate that the same quality of exemplars could be generated without access to frontier reasoning models — for instance, through human expert solutions, or through an iterative bootstrapping process starting from the base model's own outputs. The "less is more" philosophy applies to the quantity of training data but not to the quality of the source of that data — the source models (DeepSeek R1, QwQ-32B) themselves required massive compute and data investments.

This doesn't invalidate the paper's core claims about data efficiency in fine-tuning, but it does narrow their scope: the paper demonstrates that given access to frontier reasoning models for data generation, fine-tuning can be highly data-efficient. It does not demonstrate that the LIMO data could be produced without such access.

Missing ablations and experiments that would strengthen the paper.

Human evaluation of solution quality. The rule-based rubric's validity is established only through its correlation with downstream performance (Figure 3). A small-scale human evaluation — having expert mathematicians rate a sample of solutions on the four quality dimensions — would validate that the rubric measures what it claims to measure (genuine cognitive quality) rather than proxying it through lexical features.

Ablation of individual quality dimensions. The paper shows that L5 solutions outperform L1 solutions, but doesn't isolate which quality dimensions drive the improvement. An experiment that varies one dimension while holding others constant (e.g., solutions with high elaboration but low verification vs. high verification but low elaboration) would identify which aspects of reasoning chain quality are most critical for downstream transfer. This would have both theoretical and practical value — guiding future data curation efforts toward the highest-impact quality dimensions.

Comparison across multiple base model families. All experiments use Qwen models (Qwen2.5-Instruct at various sizes, Qwen1.5-Chat for the pre-training ablation, Qwen2.5-Math-7B-Instruct and DeepSeek-R1-Distill-Qwen-32B for data generation). The LIMO Hypothesis claims generality — "in foundation models where domain knowledge has been comprehensively encoded" — but only one model family is tested. Replicating the main result on, for example, Llama-3-70B-Instruct or DeepSeek-V2 would test whether the approach transfers across architectures and pre-training data distributions. The Qwen1.5 comparison partially addresses this by varying pre-training quality within a family, but doesn't test cross-family generalization.

Training data difficulty distribution analysis. The paper selects problems that DeepSeek-R1-Distill-Qwen-32B solves in 1–3 out of 32 attempts (~3–9% success), but doesn't report the actual difficulty distribution of the final LIMO-800 dataset. How many problems fall at 1/32 vs. 2/32 vs. 3/32? Is there a correlation between problem difficulty score and downstream impact? This would provide more granular insight into the difficulty threshold — is 1–3/32 uniformly optimal, or do problems at the harder end (1/32) contribute disproportionately?

Longer training or alternative optimization schedules. The paper trains for 15 epochs with no warmup and cosine decay. An ablation of epoch count (1, 3, 5, 10, 15, 20 epochs) would characterize whether the high epoch count is necessary for the tiny dataset or whether most of the gain occurs in the first few epochs. Similarly, comparing with and without the warmup omission would validate the paper's claim that omitting warmup facilitates rapid adaptation.

The memorization-vs-generalization debate is engaged but not resolved.

The paper positions its out-of-distribution results as evidence against the claim that SFT leads to memorization (Mirzadeh et al., 2024; Zhang et al., 2024; Chu et al., 2025). The OOD performance is genuinely strong — 84.2% on CHMath, 91.1% on Gaokao, 83.9% on Kaoyan — and the cross-lingual transfer to Chinese examinations is particularly compelling because it's unlikely that the model is simply pattern-matching English training examples to Chinese test problems.

However, the paper does not run the specific diagnostic tests that the memorization literature has proposed. Mirzadeh et al. (2024) tested generalization by altering numerical values in problems and observing performance drops. The LIMO paper doesn't apply this or similar tests to its fine-tuned model. It remains possible that LIMO's OOD performance reflects generalization across problem formats and languages but not necessarily across underlying mathematical structures in the way the memorization critique targets. The Chinese examination problems may share structural similarities with the English competition problems in the training data (both test algebra, geometry, number theory, combinatorics), meaning the model could be generalizing across surface features (language, problem framing) while still relying on pattern-matching for the underlying mathematical operations.

A stronger test would involve systematic perturbation of problem parameters (numerical values, geometric configurations, algebraic structures) in the evaluation benchmarks to distinguish genuine mathematical reasoning from template matching. The paper's current OOD evaluation demonstrates impressive breadth of transfer but doesn't fully address the depth question raised by the memorization literature.

The 800-example figure is empirically motivated but may not be a genuine minimum.

The dataset size ablation (Figures 7–8) shows that 400 examples achieve 57.5% on AIME24 — already competitive — and that 800 represents a point of diminishing returns. But the paper doesn't explore whether even fewer examples could work: 200? 100? 50? The monotonic improvement from 400 to 2,000 on AIME24 (57.5% to 69.6%) suggests that 400 is not the floor — the model might achieve meaningful AIME performance with even fewer examples if they were selected from the very top of the quality distribution. Characterizing the true minimum data requirement would require testing smaller increments below 400 and reporting where performance collapses. The paper's claim of "just hundreds of examples" is supported by the 400–800 range tested, but "just dozens" remains unexplored.

Similarly, the paper doesn't characterize the variance of results at each dataset size. With only 400 examples, the specific subset selected could substantially influence performance. Running multiple random seeds at each dataset size (different random subsets of the LIMO-Pool, different data orderings during training) would quantify the stability of the results and whether the observed diminishing returns pattern is robust to sampling variation.

Overall assessment of the paper's core empirical contributions.

The paper convincingly demonstrates that highly curated, quality-focused SFT data can elicit strong mathematical reasoning from a capable base model with far fewer examples than prior SFT-based approaches required. The 63.3% AIME24 from 800 examples, compared to 6.5% from NuminaMath-100k and 50.2% from OpenThoughts-114k, is a robust and striking result. The 400-example ablation showing 57.5% AIME24 further strengthens the data-efficiency claim. The Qwen1.5 comparison provides clean evidence that pre-training quality gates the approach's effectiveness. The solution quality gradient (L1–L5) validates the paper's quality rubric. These are solid, well-designed experiments that support the paper's central empirical narrative.

The paper is less successful at establishing the theoretical mechanisms proposed by the LIMO Hypothesis. The hypothesis claims that data efficiency arises from the synergy of pre-trained knowledge and inference-time computation space, and that exemplars serve as "cognitive templates" that teach meta-cognitive processes. While the results are consistent with this mechanism, they do not isolate it from alternative explanations. The performance improvements could also be explained by: (a) the LIMO data simply being in-distribution for the evaluation benchmarks (despite deduplication, the problems share mathematical domains and structures); (b) the generating models (DeepSeek R1, QwQ-32B) producing solutions in a reasoning style that Qwen2.5-32B-Instruct can easily imitate, regardless of the specific quality dimensions; (c) the small dataset size enabling more training epochs, which produces a different optimization trajectory than the 1–3 epoch SFT typical for larger datasets. The paper doesn't run the control experiments that would distinguish these mechanisms from the claimed cognitive template mechanism.

The paper's strongest contribution is empirical demonstration — showing that the conventional wisdom about data requirements for mathematical reasoning fine-tuning is wrong, and that careful data curation can dramatically reduce the needed training examples. The theoretical framework (LIMO Hypothesis) provides a useful organizing structure for these findings, but the causal claims it makes are only partially validated by the experiments. Future work will need to more precisely isolate the mechanism (is it solution quality? problem difficulty? training dynamics from small data? distillation from frontier models?) and test the hypothesis's predictions in settings where its conditional structure makes sharp, falsifiable claims.

6. Limitations and Trade-offs

6.1 The Approach Depends on Frontier Reasoning Models for Training Data Generation — It Is a Form of Distillation, Not Autonomous Elicitation

The assumption or constraint. The LIMO dataset's reasoning chains are generated by DeepSeek R1, DeepSeek-R1-Distill-Qwen-32B, and QwQ-32B — all models that were themselves trained with massive computational investments, including reinforcement learning with extended chain-of-thought (Guo et al., 2025) or independent large-scale development (Team, 2025b). The paper does not conceal this — Section 3.1.2 explicitly states that solutions are generated from "three state-of-the-art reasoning models" — but it also does not frame the approach as distillation. Instead, the paper's narrative positions LIMO as demonstrating that "sophisticated reasoning can emerge through minimal but strategically designed demonstrations" (Section 1, LIMO Hypothesis), implying that the demonstrations' quality — not their source — is what matters.

The consequence. This dependence fundamentally narrows the scope of the "less is more" claim. The paper demonstrates that fine-tuning can be highly data-efficient — but only when the training data is generated by other, more capable reasoning models that required orders of magnitude more resources to create. The data efficiency claim applies to the fine-tuning stage but not to the full pipeline: producing the LIMO dataset required access to DeepSeek R1 (trained with RL at scale), DeepSeek-R1-Distill-Qwen-32B (distilled from R1), and QwQ-32B (an independently developed reasoning model). A practitioner starting from only Qwen2.5-32B-Instruct and a corpus of math problems cannot replicate the LIMO approach without also having access to these frontier reasoning models — they cannot generate the high-quality reasoning chains that the approach depends on.

This has several downstream implications:

• The approach cannot bootstrap from the base model alone. The paper does not demonstrate that Qwen2.5-32B-Instruct — the very model being fine-tuned — can generate training data of sufficient quality for the LIMO approach, even through iterative self-improvement. In fact, the ReST^EM experiment in the related literature (discussed in the Snell et al. analysis, Appendix K) showed that on-policy self-generated revision data can degrade performance. The LIMO paper does not test whether its own base model can serve as a training data generator, leaving open the question of whether the approach can function without external frontier models.

• The capability ceiling of the fine-tuned model is bounded by the capability of the data-generating models. If DeepSeek R1 achieves, say, X% on AIME24, then LIMO — which learns from R1's reasoning traces — cannot exceed X% through this approach alone. The paper does not report DeepSeek R1's exact AIME24 performance, but the distilled version (DeepSeek-R1-Distill-Qwen-32B) was used for difficulty assessment (Section 3.1.1), and R1 itself is known to substantially outperform the distilled version. LIMO's 63.3% on AIME24 is strong, but it may be bounded by the quality of the R1-generated solutions in the training data.

• The approach inherits any systematic weaknesses or biases of the generating models. If DeepSeek R1, R1-Distill-Qwen, and QwQ-32B share common failure modes — certain types of mathematical reasoning they collectively do poorly, or stylistic patterns they all exhibit — the LIMO dataset will reflect those limitations. The paper does not analyze whether the three generating models produce qualitatively different errors or whether certain mathematical domains are systematically underrepresented in their high-scoring solutions.

What evidence exists in the paper. The paper provides no ablation that tests whether alternative data sources of comparable quality but different origin (e.g., human expert solutions, solutions from a different model family) produce comparable results. The L1–L5 quality gradient (Figure 3, Section 6.3.1) shows that solution quality matters, but all solutions at all quality levels were generated by the same set of frontier reasoning models — the experiment varies which of these model-generated solutions are selected, not who generates them. The paper does not report baseline experiments fine-tuning on, for example, expert-written solutions to the same problems, which would test whether the generating model's identity matters independently of solution quality.

Mitigation status. The paper does not acknowledge this as a limitation or frame the approach as distillation. No ablation addresses the question of whether the base model itself can generate adequate training data. No future work is suggested for reducing dependence on external reasoning models. A practitioner reading the paper might reasonably conclude that they need only "800 high-quality examples" without realizing that generating those examples required frontier models they may not have access to.

---

6.2 The Difficulty Estimation Cost Is Unaccounted for and Potentially Exceeds the Fine-Tuning Cost Itself

The assumption or constraint. The two-stage difficulty filtration pipeline (Section 3.1.1) requires running inference on tens of millions of candidate problems using Qwen2.5-Math-7B-Instruct (up to 4 attempts per problem) and then DeepSeek-R1-Distill-Qwen-32B (32 attempts per surviving problem). The Stage 2 filter alone — 32 solution attempts on each of the 2,125 problems in the LIMO-Pool — represents 68,000 inference calls to a 32B-parameter reasoning model generating long chain-of-thought solutions. The Stage 1 filter applies to an initial corpus of "tens of millions" of problems, each run up to 4 times through a 7B model — potentially hundreds of millions of inference calls. The paper states the initial corpus size qualitatively but provides no computation budget for the filtration process. The training cost (SFT on 800 examples for 15 epochs) is treated as the primary resource metric; the data curation cost is externalized.

The consequence. The total computational cost of producing the LIMO dataset — counting all inference calls for difficulty filtering, solution generation from three frontier models, and quality scoring — almost certainly exceeds the cost of fine-tuning itself by orders of magnitude. For the Stage 2 filter alone: 2,125 problems × 32 attempts × generating long reasoning chains from a 32B model represents approximately 68,000 long-sequence generations. DeepSeek-R1-Distill-Qwen-32B solutions for AIME-level problems can easily exceed 5,000–10,000 tokens each. At 68,000 generations with an average of, say, 5,000 output tokens, the output token count is roughly 340 million tokens — comparable to or exceeding the total tokens processed during SFT on 800 examples (800 examples × up to 16,384 tokens × 15 epochs ≈ 196 million tokens of training data, though training involves forward and backward passes).

Some of this curation cost is amortizable — the dataset, once constructed, can be reused for multiple fine-tuning runs or shared with the community. But for a practitioner seeking to apply the LIMO methodology to a new domain or a new base model, the full pipeline cost must be incurred. The paper's headline efficiency claim — "using only 1% of the training data" — accounts for SFT data quantity but not for the computation required to identify which 800 examples to use. A fair accounting of total resource requirements would include the filtration and generation costs, which would substantially reduce — though likely not eliminate — the efficiency advantage over large-scale SFT on less-curated data.

What evidence exists in the paper. The paper provides no FLOPs, GPU-hours, or inference cost estimates for any stage of the data curation pipeline. The size of the initial candidate pool is described qualitatively as "tens of millions of mathematical problems" (Section 3.1.1) without specification. The number of problems surviving Stage 1 (after Qwen2.5-Math-7B filtering) is not reported — only the LIMO-Pool size (2,125 after Stage 2) is specified. The number of candidate solutions generated per problem by the three frontier models (Section 3.1.2) is not specified, only that "multiple solutions" are sampled. The computational cost of the 4-dimensional quality scoring (keyword frequency computation) is negligible but the generation cost for candidate solutions is not.

Mitigation status. The paper does not acknowledge the unaccounted curation cost as a limitation. The abstract and introduction emphasize that LIMO "achieves 63.3% on AIME24... while using only 1% of the training data required by prior approaches" — a framing that treats data quantity as the sole resource metric. Section 8 (Future Work) does not mention reducing curation cost or developing cheaper difficulty estimation methods. A practitioner evaluating whether to adopt the LIMO methodology cannot assess the true total cost from the information provided.

---

6.3 The Rule-Based Quality Scoring System Uses Lexical Proxies That May Not Reliably Measure Genuine Cognitive Quality

The assumption or constraint. The four-dimensional quality rubric (Section 3.1.2) operationalizes complex cognitive constructs — elaborated reasoning, self-verification, exploratory approach, adaptive granularity — through simple lexical features: solution length (30% weight), frequency of verification-related keywords like "check" and "verify" (20% weight), frequency of tentative expressions like "perhaps" and "might" (25% weight), and frequency of connective phrases like "therefore" and "since" (25% weight). All keyword frequencies are normalized by text length.

This is an explicit design choice. The paper states that the scoring system uses "weighted metrics for each dimension" implemented through "a rule-based scoring system that calculated weighted metrics." The choice is motivated by scalability and reproducibility — keyword counting is transparent, fast, and does not require human annotation or learned quality estimators.

The consequence. The lexical proxies can be gamed — solutions can score highly on the rubric without genuinely exhibiting the corresponding cognitive qualities. A solution could achieve a high score by being long (inflating the 30% elaborated reasoning component), sprinkling in "therefore" and "check" frequently (inflating the logical connector and verification components), and including speculative language like "perhaps we could try" without actually exploring genuinely different solution strategies. The normalization by length partially addresses the advantage that longer solutions would otherwise have on all keyword-based dimensions, but it introduces a different issue: two solutions of equal length achieve the same "elaborated reasoning" score (30% of the total) regardless of whether the length represents genuine step-by-step elaboration or verbosity and repetition. The weight on solution length (30% — the single largest component) means the rubric systematically prefers longer solutions, potentially selecting verbose but cognitively mediocre reasoning chains over concise but genuinely insightful ones.

This matters because the LIMO dataset is constructed by selecting the highest-scoring solution per problem (Section 3.1.2) — the rubric is the sole arbiter of what constitutes a "high-quality" reasoning chain for training purposes. If the rubric preferentially selects solutions that are long and keyword-dense but only superficially exhibit the target cognitive patterns, the training data may teach the model to produce verbose, keyword-laden reasoning rather than genuinely deliberative reasoning. The resulting model might perform well on benchmarks (as LIMO does), but the improvement might be attributable to learning stylistic patterns of "good" reasoning traces rather than internalizing the underlying cognitive processes — a form of surface-level imitation rather than deep capability transfer.

The paper's L1–L5 quality gradient experiment (Figure 3, Section 6.3.1) provides evidence that the rubric's rankings correlate with downstream performance — models trained on L5 solutions outperform those trained on L1 solutions. This validates that the rubric measures something causally related to training effectiveness. But it does not validate that the rubric measures the specific cognitive constructs it claims to measure (elaborated reasoning, self-verification, etc.) rather than correlated surface features. A solution could score highly because it is long and includes many logical connectors, and this length+connector combination might produce better training data regardless of whether genuine verification or exploration is present.

What evidence exists in the paper. The L1–L5 gradient (Figure 3) shows a monotonic relationship between rubric score and downstream AIME24/MATH500 performance, establishing predictive validity. However, the paper provides no validation that high-scoring solutions actually contain more genuine verification, exploration, or adaptive granularity than low-scoring solutions. No human evaluation of solution quality is reported — for example, having expert mathematicians rate a sample of solutions on the four dimensions and correlating those ratings with the rubric scores. No ablation varies individual quality dimensions while holding others constant to test whether specific dimensions (e.g., verification density vs. length) independently contribute to downstream performance. No sensitivity analysis explores whether the 30/20/25/25 weighting is optimal or whether other weightings would produce meaningfully different dataset compositions and downstream results.

Mitigation status. The paper does not acknowledge the gap between the lexical proxies and the cognitive constructs they are intended to measure. The L1–L5 validation is presented as sufficient evidence that the rubric works, without discussion of what exactly it is measuring. The rule-based approach is presented as a feature (transparent, reproducible) rather than as a potential limitation (proxies may diverge from constructs). A practitioner applying the rubric to a new domain would not know whether the specific keywords chosen (English-language verification terms, tentative expressions, logical connectors) transfer to other languages, other reasoning domains, or other model output styles.

---

6.4 The Evaluation Is Conducted on a Single Model Family (Qwen), Limiting Evidence for the Generality of the LIMO Hypothesis

The assumption or constraint. All experiments in the paper use models from the Qwen family: Qwen2.5-32B-Instruct as the primary base model, Qwen2.5-Instruct variants at 3B/7B/14B/32B/72B for model size scaling (Section 6.3.4), Qwen1.5-32B-Chat for the pre-training knowledge ablation (Section 6.3.3), Qwen2.5-Math-7B-Instruct for Stage 1 difficulty filtering, and DeepSeek-R1-Distill-Qwen-32B for Stage 2 filtering. The LIMO Hypothesis is stated as a general claim:

"In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes." (Section 3)

The hypothesis does not specify that this applies only to models from a particular architecture family or with particular pre-training data characteristics beyond the general requirement of "comprehensively encoded" domain knowledge.

The consequence. The paper provides no evidence that the LIMO approach transfers to other model families — for example, Llama-3-70B-Instruct, DeepSeek-V2, Gemma, or Mistral — that also claim strong mathematical reasoning capabilities from pre-training. This matters because the LIMO Hypothesis's conditional structure (Premise I: latent prerequisite knowledge) makes a specific prediction: any model with sufficiently rich mathematical pre-training should benefit from the approach. But "sufficiently rich" is operationalized only as the difference between Qwen1.5 and Qwen2.5 — a within-family comparison. Different model families may encode mathematical knowledge differently (through different pre-training data mixtures, different tokenizers that affect mathematical notation handling, different architectural inductive biases for reasoning). The approach might fail on a model with strong mathematical pre-training if the model represents that knowledge in a way that doesn't interface well with the LIMO exemplars' reasoning patterns.

The Qwen1.5 vs. Qwen2.5 comparison (Figure 5) is a strong within-family test, but it varies only one dimension of pre-training (data quality/quantity within the same architecture and training pipeline). It does not test whether the approach transfers when architecture, tokenization, pre-training objectives, or data distribution differ. This limits the evidence for the hypothesis's claimed generality — the paper demonstrates that the approach works on one model family and fails on an earlier generation of the same family, but does not establish that it works on other contemporary model families with strong mathematical pre-training.

Additionally, the filtering pipeline's dependency on Qwen-family models for difficulty assessment (Qwen2.5-Math-7B-Instruct, DeepSeek-R1-Distill-Qwen-32B) means that the difficulty filtering is calibrated to Qwen-family capabilities. A problem that the Qwen-based filter deems "appropriately difficult" (1–3 successes in 32 attempts) might be substantially easier or harder for a non-Qwen base model. The LIMO dataset's difficulty profile is model-family-specific, and applying it to a different model family without recalibrating the filters could produce suboptimal results — either because the problems are too easy (not forcing extended deliberation) or too hard (beyond the model's latent capability).

What evidence exists in the paper. All experiments use Qwen-family models, with no cross-family baselines. The model size scaling experiment (Figure 6, Section 6.3.4) varies parameter count within Qwen2.5-Instruct but does not compare against, for example, a Llama-3-70B-Instruct fine-tuned on the same LIMO data. The pre-training knowledge ablation (Figure 5) compares Qwen1.5 vs. Qwen2.5 — a within-family generational comparison. The paper does not report any experiment that tests whether the approach generalizes to non-Qwen architectures with comparable or superior mathematical pre-training.

Mitigation status. The paper does not acknowledge the single-model-family limitation. The LIMO Hypothesis is presented as a general claim, and the Qwen2.5-32B-Instruct model is described as having "enhanced pre-training data quality, particularly in mathematical and code-related content" (Section 6.3.3) — implying that it satisfies Premise I — but no argument is made that the results should generalize to other models that also satisfy this premise. A practitioner considering applying the LIMO approach to a non-Qwen base model would be extrapolating from a sample of one model family with no cross-family validation evidence.

---

6.5 Hard Mathematical Problems Remain Unsolved — The Approach Cannot Elicit Capabilities That the Base Model Lacks

The assumption or constraint. The LIMO Hypothesis explicitly conditions its claims on the presence of latent prerequisite knowledge in the model's parameters. Premise I states:

"The latent presence of prerequisite knowledge within the model's parameter space." (Section 3)

The paper's difficulty filtering (Section 3.1.1) operationalizes this by excluding problems that DeepSeek-R1-Distill-Qwen-32B solves 0 times in 32 attempts — these are problems deemed beyond the capability frontier that the LIMO approach can reach. The paper acknowledges this filtering criterion but does not characterize what fraction of problems in the original corpus fell into this "too hard" category or what mathematical domains they represent.

The consequence. The LIMO approach has a hard capability ceiling: it can elicit reasoning capabilities that are latent in the base model's pre-trained knowledge, but it cannot create capabilities that are absent. This is the fundamental distinction between knowledge elicitation and knowledge acquisition that the paper itself draws (Section 1). The consequence is that for problems requiring mathematical concepts, techniques, or reasoning patterns that are genuinely outside the base model's pre-training distribution, the approach will fail regardless of how carefully the exemplars are curated.

The paper's experimental results are consistent with this limitation but do not directly characterize its boundaries. The difficulty filtering removes the hardest problems before training, so the approach's failure mode on those problems is never directly tested or quantified. The improved performance on AIME24 (63.3%) and MATH500 (95.6%) shows that the approach successfully activates latent capabilities on problems of moderate-to-high difficulty. But the paper does not report: (a) what fraction of the original candidate pool was excluded as "too hard" (0/32 successes by DeepSeek-R1-Distill-Qwen-32B); (b) whether the fine-tuned model shows any improvement on those excluded problems; (c) whether there is a sharp threshold — a difficulty level above which the approach provides zero benefit — analogous to what Snell et al. (2024) found for test-time compute scaling, where the hardest difficulty bin showed near-zero improvement regardless of budget.

The Qwen1.5 comparison (Figure 5) provides indirect evidence: on a model with insufficient pre-trained mathematical knowledge, the LIMO approach yields only 9.2% on AIME24 — barely above random for a model of this scale, and below the untrained Qwen2.5 base model's 16.5%. This demonstrates that the approach fails catastrophically when pre-training knowledge is globally insufficient. But it doesn't characterize the more nuanced case where a model has adequate pre-training knowledge overall but lacks knowledge of specific mathematical subdomains — would the approach selectively fail on problems from those subdomains while succeeding on others? The per-subject MATH breakdown (algebra, geometry, number theory, etc.) is not reported, so domain-specific failure patterns cannot be assessed.

This limitation has practical implications for deployment. A practitioner using LIMO-style fine-tuning for a mathematical reasoning application needs to know the approach's capability boundary: which types of problems will it help with, and which will remain unsolvable regardless of exemplar quality? The paper's difficulty filtering provides an operational definition of the boundary (problems solvable 1–3 times in 32 attempts by a frontier reasoning model), but this definition is relative to the filtering model, not absolute, and its correspondence to the fine-tuned model's actual capability ceiling is not directly tested.

What evidence exists in the paper. The paper reports that the initial candidate pool contained "tens of millions of mathematical problems" and that after two-stage filtering, 2,125 survived (the LIMO-Pool) — an approximately 99.99% reduction rate if the initial pool was, say, 10 million problems. Most of this reduction occurs at Stage 1 (removing problems that Qwen2.5-Math-7B-Instruct solves), and the paper does not break down what fraction was removed for being "too easy" vs. what fraction was removed at Stage 2 for being "too hard" (0/32 successes). The final dataset of 800 problems is selected from the 2,125 LIMO-Pool problems — all of which are in the narrow "edge of solvability" band. The paper provides no evaluation of LIMO's performance on problems outside this band — either easier problems (to test whether the approach degrades performance on simple problems, analogous to the verifier over-optimization phenomenon in Snell et al., 2024) or harder problems (to characterize the capability ceiling).

The model size scaling experiment (Figure 6, Section 6.3.4) shows that even the 72B model achieves only 68.3% on AIME24 — leaving 31.7% of AIME24 problems unsolved despite the largest model scale. Whether these 31.7% represent problems that are fundamentally beyond the Qwen2.5 pre-training knowledge distribution, or problems that could be solved with different exemplars or a different training approach, is not investigated.

Mitigation status. The paper acknowledges the conditional nature of the LIMO Hypothesis through Premise I — the approach requires latent prerequisite knowledge. But it does not characterize the failure mode when this condition is partially violated (adequate knowledge in some domains but not others), does not report what fraction of the original problem space falls into the "too hard" excluded category, and does not test whether the approach provides any benefit at all on those excluded problems. The Qwen1.5 comparison shows the extreme case (globally insufficient pre-training), but the more practically relevant case of selective capability gaps (a well-pre-trained model that nonetheless has blind spots) is unexplored. A practitioner would need to independently characterize which problem types their base model can and cannot handle to understand where LIMO-style fine-tuning will help versus where it will be ineffective.

---

6.6 No Confidence Intervals or Statistical Significance Testing — Key Quantitative Claims Are Point Estimates Without Uncertainty Quantification

The assumption or constraint. The paper reports all performance numbers as point estimates — single-number pass@1 accuracies for each benchmark. No confidence intervals, standard errors, or statistical significance tests are reported for any result in Table 1, Figures 3–8, or the ablation studies. The paper uses the unbiased pass@1 estimator from Chen et al. (2021) for small benchmarks (AIME24, AMC23, CHMath: 4 samples per problem at temperature=0.6) and greedy decoding (1 sample per problem) for larger benchmarks. The Chen et al. (2021) estimator accounts for the variance from using a finite number of samples per problem to estimate pass@1, but the paper reports only the point estimate, not the associated uncertainty.

The consequence. For several benchmarks, the sample sizes are small enough that the reported differences may not be statistically reliable:

• AIME24 has only 15 problems. Even with 4 samples per problem (60 total generations), the pass@1 estimate has substantial sampling variance. A difference of a few percentage points — for example, LIMO at 63.3% vs. OpenThoughts-114k at 50.2% on AIME24 — represents getting 1–2 more problems correct out of 15. A single correct answer gained or lost due to sampling variation could shift the pass@1 estimate by several percentage points. The 72B model's AIME24 performance (68.3%) vs. the 32B model (63.3%) — a 5.0 point difference that the paper interprets as evidence of scaling — corresponds to roughly 0.75 more problems correct out of 15, which could plausibly arise from sampling noise.

• AMC23 and CHMath also have fewer than 50 problems each (the paper's stated threshold for using 4-sample estimation), but their exact sizes are not reported. The reliability of comparisons involving these benchmarks depends on their actual sizes.

• The non-monotonic MATH500 result at 72B (Figure 6: 72B scores 94.8% vs. 32B's 95.6%) is a negative scaling result — larger model performs slightly worse on a 500-problem benchmark. This difference (0.8 percentage points, corresponding to 4 problems out of 500) could easily be noise. Without confidence intervals, the paper cannot distinguish a genuine reversal of scaling trends from sampling variation. The paper acknowledges this as a possible "saturation point" but does not quantify the uncertainty.

• The Gaokao, Kaoyan, GradeSchool, and other OOD benchmarks have unspecified sizes. Some of these — particularly GradeSchool, described as "our new elementary mathematics benchmark" — may be small. If any of these benchmarks contain fewer than 100 problems, the reported differences could have wide confidence intervals.

The absence of uncertainty quantification is particularly consequential for the paper's headline claims about relative performance: LIMO outperforms QwQ-32B-Preview by 13.3 points on AIME24, 5.8 points on MATH500, and 12.7 points on AMC23. On AIME24 (15 problems), the 13.3-point gap is approximately 2 problems. A significance test would indicate whether a 2-problem advantage on a 15-problem test is unlikely to occur by chance. On MATH500, the 5.8-point gap (~29 problems) is more likely to be significant, but the exact confidence interval depends on the pass@1 estimator variance.

The paper also reports cross-benchmark averages (the "AVG." row in Table 1: LIMO 78.1%, QwQ-32B-Preview 66.9%, etc.). Averaging across benchmarks of different sizes and characteristics without weighting by benchmark size or reporting the variance of the aggregate is a composite measure whose statistical properties are undefined. A small benchmark with high variance can disproportionately influence the average.

What evidence exists in the paper. The paper reports no confidence intervals, standard errors, p-values, or effect sizes anywhere in the main text or figures. The Chen et al. (2021) unbiased estimator is cited for the pass@1 metric on small benchmarks, but only the point estimates are reported, not the estimator's variance. The benchmark sizes for OlympiadBench, Gaokao, Kaoyan, GradeSchool, MinervaMath, and GPQA are not provided in Section 5 (Evaluation Framework) or Section 6 (Experiment), making it impossible for a reader to assess the reliability of those results.

Mitigation status. The paper does not acknowledge the absence of uncertainty quantification as a limitation. No future work is suggested regarding more rigorous statistical reporting. The field standard for mathematical reasoning benchmarks varies — some papers report confidence intervals or standard deviations, others report only point estimates — but given the small sizes of several key benchmarks (AIME24 at 15 problems, AMC23 and CHMath at <50 problems) and the paper's strong comparative claims, the absence of any uncertainty information weakens the reliability of the reported differences. A practitioner evaluating whether LIMO genuinely outperforms alternatives by the reported margins cannot assess whether those margins exceed what would be expected from sampling variation alone.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a reframing of what post-training for reasoning is trying to accomplish rather than a paradigm shift in how models are trained. The core intellectual move — recasting fine-tuning from knowledge acquisition to knowledge elicitation — changes what the field should optimize, how it should think about data, and where it should look for failures. This is a conceptual reorganization of existing pieces, not the invention of a new training algorithm. But reorganizations can be consequential: the Chinchilla scaling laws (Hoffmann et al., 2022) were a reframing of how to allocate pretraining compute between model size and data, and they reshaped pretraining practice. The LIMO Hypothesis has the potential to do something analogous for post-training — if the central claim holds across model families and domains, it implies that the community's massive investment in ever-larger reasoning datasets (NuminaMath's 860k examples, MetaMath's bootstrapped question generation, MAmmoTH2's web-scale curation) has been optimizing the wrong variable. The lever is not data volume but data character — specifically, the density of high-quality cognitive templates in the training mixture.

This reframing resolves a tension that has been building in the reasoning literature. On one side, the dominant post-training paradigm operated on the implicit assumption that more supervised examples monotonically improve reasoning (Yue et al., 2023; 2024; Yu et al., 2024; Li et al., 2024b). On the other side, a growing body of work challenged whether SFT-based reasoning models genuinely generalize or merely memorize problem templates (Mirzadeh et al., 2024; Zhang et al., 2024; Chu et al., 2025; Kambhampati, 2024). The LIMO results suggest a resolution: both sides can be right, depending on data quality. Large-scale SFT on uncurated data can produce memorizers — models that pattern-match problem templates without internalizing underlying reasoning processes. The NuminaMath-100k result (average accuracy dropping from 49.9% to 32.3%, Table 1) is a vivid demonstration. Small-scale SFT on cognitively rich exemplars can produce generalizers — models that have learned how to think from a few hundred exemplary demonstrations and transfer that capability across languages, formats, and disciplines. The 45.8% absolute improvement across diverse OOD benchmarks supports this interpretation. The debate was never about SFT versus RL; it was about what kind of data each approach uses and what that data teaches the model to do. LIMO makes this distinction explicit and operationalizable.

The paper also identifies a new binding constraint on data-efficient reasoning: pre-training knowledge completeness. Prior work on data efficiency (Zhou et al., 2023) treated quality as a property of the fine-tuning data alone. The LIMO Hypothesis adds a conditional: data-efficient elicitation works if and only if the model's pre-trained knowledge base contains the prerequisite concepts. The Qwen1.5 vs. Qwen2.5 comparison (Figure 5) — a 54.1-point AIME24 gap between models of identical architecture trained on identical LIMO data — provides unusually clean evidence for this conditional structure. This matters because it predicts where the approach will fail. A model family with weaker mathematical pre-training (older generation, smaller scale, different data mixture priorities) will not benefit from LIMO-style curation regardless of exemplar quality. The constraint is not continuous — it appears to act as a threshold. Qwen1.5-32B-Chat at 9.2% on AIME24 is not a model that partially benefits from LIMO data; it is a model for which the approach fundamentally fails. This reframing shifts attention from "how do we curate better fine-tuning data?" (the LIMA question) to "how do we assess whether a base model has sufficient latent knowledge for data-efficient elicitation, and what do we do when it doesn't?" — a new, empirically grounded research question.

The paper also elevates reasoning chain quality from an intuition to a measurable, actionable construct. The four-dimensional rubric (Section 3.1.2) — elaborated reasoning, self-verification, exploratory approach, adaptive granularity — provides a language and a methodology for discussing what makes a solution trace good for training. Before LIMO, "high-quality reasoning data" was a vague aspiration operationalized mainly through correctness filtering and problem difficulty binning. After LIMO, there is a concrete (if imperfect) rubric that researchers can adopt, critique, refine, or replace. The L1–L5 gradient experiment (Figure 3) validates that these dimensions are causally connected to downstream performance, establishing that reasoning chain quality is not merely an aesthetic preference but a training-relevant variable. This operationalization, even with its lexical-proxy limitations, lowers the barrier for the field to engage with data quality as a first-class object of study rather than an unexamined assumption.

Research directions that become more attractive:

• Pre-training data composition as a gating variable for fine-tuning strategies. The Qwen1.5 vs. Qwen2.5 result implies that the feasibility of data-efficient reasoning elicitation is determined at pre-training time. This makes pre-training data mixture analysis — quantifying how much mathematical content, of what type, and in what format, is needed for a model to cross the "elicitation threshold" — a high-priority research question. Pre-training curriculum design could be explicitly optimized to produce models in the elicitable regime.

• Quality-scoring rubrics for reasoning chains across domains. If the four-dimensional rubric captures general properties of good reasoning rather than mathematics-specific features, it should transfer (with appropriate keyword adaptation) to code generation, scientific reasoning, legal analysis, and other domains requiring structured multi-step cognition. Validating and extending the rubric across domains would test whether "reasoning chain quality" is a domain-general construct.

• Data quality as an alternative to RL for inducing meta-cognitive behaviors. DeepSeek-R1 (Guo et al., 2025) and o1 (OpenAI, 2024) showed that RL can induce self-verification, backtracking, and exploration. LIMO shows that SFT on examples exhibiting these behaviors can achieve similar transfers at dramatically lower computational cost. The tradeoff between RL (autonomous discovery of reasoning strategies, but expensive) and quality-focused SFT (cheap if high-quality exemplars exist, but bounded by exemplar quality) becomes a central design question for reasoning model development.

Research directions that become less attractive:

• Indiscriminate scaling of reasoning datasets without quality curation. The NuminaMath-100k result (catastrophic degradation across all 10 benchmarks, Table 1) is a strong negative signal. If large-scale SFT on uncurated data can destroy pre-existing reasoning capabilities, the default strategy of "collect more data, filter for correctness, train" is not just inefficient — it can be actively harmful. The field should shift from volume-maximization to quality-density-maximization in reasoning data curation.

• Purely algorithmic improvements to SFT training. The paper's SFT recipe is deliberately simple — standard full-parameter fine-tuning with cosine decay, no warmup, 15 epochs. The dramatic performance differences come from data curation, not from training algorithm innovations. This suggests that, for the current generation of foundation models, data quality dominates training procedure in determining reasoning outcomes. Research effort is better spent on understanding what makes data effective than on marginal improvements to optimization schedules or architectures.

Follow-Up Research This Work Enables

1. Systematically characterize the pre-training knowledge threshold for data-efficient reasoning elicitation. The Qwen1.5 vs. Qwen2.5 comparison (Figure 5) shows that pre-training generation matters enormously, but it is a coarse binary comparison — old generation vs. new generation. A more granular study would train a series of models with controlled variations in mathematical pre-training data quantity and composition (e.g., 1%, 5%, 10%, 50%, 100% of a fixed mathematical corpus, or systematic exclusion of specific mathematical domains like geometry or number theory), fine-tune each on identical LIMO data, and map the relationship between pre-training mathematical content and post-training reasoning capability. This would identify whether the threshold is sharp (a specific minimum corpus size below which elicitation fails entirely) or continuous (diminishing returns as pre-training math content increases) and would reveal which mathematical subdomains are most critical for the transfer. The paper's finding that Qwen1.5 at 9.2% AIME24 is far below Qwen2.5 at 63.3% — a gap that cannot be explained by architecture differences — makes this a high-priority extension.

2. Test whether LIMO-style curation transfers to non-mathematical reasoning domains with comparable structure. The LIMO Hypothesis claims generality: "in foundation models where domain knowledge has been comprehensively encoded during pre-training." Mathematical reasoning is a clean testbed because of its verifiable answers, well-defined difficulty hierarchies, and existing benchmarks. But the hypothesis's scope extends to any domain where pre-trained knowledge is rich and reasoning chains can be evaluated for quality. A direct replication in code generation (using benchmarks like LiveCodeBench or SWE-bench, with unit tests providing correctness signals and problem difficulty operationalized through solve rates) would test domain generality. A more ambitious extension would target scientific reasoning (using benchmarks like GPQA or ScienceQA with domain-specific quality rubrics adapted from the four LIMO dimensions), legal reasoning, or medical diagnosis. The key design choice is whether the four quality dimensions — elaborated reasoning, self-verification, exploratory approach, adaptive granularity — are domain-general properties of good reasoning or mathematics-specific features. A failure to transfer (e.g., LIMO-style curation producing no improvement on code benchmarks despite strong pre-trained code capabilities) would refine the hypothesis's scope.

3. Develop and validate non-lexical quality assessment methods that directly measure cognitive process quality rather than surface features. The paper's rule-based scoring rubric (Section 3.1.2) uses keyword frequencies and solution length as proxies for cognitive quality. The L1–L5 gradient (Figure 3) validates that the rubric identifies solutions that produce better downstream models, but it does not validate that the rubric measures genuine cognitive quality as opposed to correlated surface features. A follow-up study would: (a) have expert mathematicians rate a sample of solutions on the four dimensions, establishing human-judgment ground truth; (b) compare those ratings against the rubric's keyword-based scores to quantify the proxy's fidelity; (c) train a learned quality estimator (fine-tuning a small LLM as a reasoning chain quality classifier on the human-labeled data) and compare its training effectiveness against the rule-based rubric; (d) ablate individual quality dimensions by constructing datasets that vary one dimension while holding others constant (e.g., solutions matched for length but varying in verification density) and measuring the independent contribution of each dimension to downstream performance. This would transform the rubric from a heuristic validated only by downstream correlation to a measurement instrument with known psychometric properties. A finding that the rubric's keyword proxies correlate weakly with human quality judgments would motivate investment in learned quality estimators; a finding of strong correlation would validate the paper's scalable approach.

4. Investigate whether the LIMO approach can be bootstrapped — generating high-quality training data from the base model itself through iterative self-improvement. The LIMO dataset's reasoning chains come from three frontier reasoning models (DeepSeek R1, DeepSeek-R1-Distill-Qwen-32B, QwQ-32B) that required massive RL or distillation investments. This makes the "less is more" claim apply to fine-tuning data quantity but not to the full pipeline's resource requirements. A critical extension would test whether Qwen2.5-32B-Instruct — the model being fine-tuned — can serve as its own data generator through iterative refinement: (1) fine-tune on a small seed dataset of high-quality exemplars (generated from frontier models, as in the current paper); (2) use the fine-tuned model to generate 32 solutions per problem for a new set of challenging problems; (3) score solutions using the quality rubric; (4) select the top solutions to form an expanded dataset; (5) fine-tune again; (6) repeat. The key measurement is whether iteration improves performance beyond what the seed dataset alone achieves, and whether performance approaches what direct distillation from frontier models achieves. The ReST^EM negative result discussed in the related Snell et al. analysis (Appendix K) — where on-policy data collection degraded revision model performance — suggests that bootstrapping is non-trivial and may require careful quality filtering. A positive result would dramatically expand the approach's applicability; a negative result would clarify that the LIMO effect depends on access to models that are qualitatively better than the fine-tuned model, not merely trained with more data.

5. Characterize the failure modes: on which types of mathematical problems does LIMO-style fine-tuning provide zero or negative benefit? The paper's difficulty filtering (Section 3.1.1) deliberately excludes problems that DeepSeek-R1-Distill-Qwen-32B solves 0 times in 32 attempts — these are deemed "too hard." But the paper never evaluates LIMO on these excluded problems to confirm that the approach truly provides no benefit. A targeted follow-up would: (a) construct a "beyond-capability" benchmark from problems that fell into the 0/32 category during filtering; (b) evaluate LIMO (and the base model) on these problems to measure whether any improvement occurs; (c) break down results by mathematical subdomain (algebra, geometry, number theory, combinatorics) to identify whether the capability ceiling is uniform or domain-specific; (d) measure whether increasing model scale (72B, or even larger models if available) pushes the capability boundary outward — that is, do problems that are beyond the 32B model's latent knowledge become within reach for the 72B model? This would provide direct evidence for the LIMO Hypothesis's most important practical prediction: that the approach cannot create capabilities the model lacks. The paper's model size scaling experiment (Figure 6) shows the 72B model reaching 68.3% on AIME24 — leaving 31.7% unsolved. Understanding whether those remaining problems are fundamentally beyond the Qwen2.5 knowledge distribution or solvable with different exemplars would guide whether future investment should go toward better curation or larger pre-training.

6. Compare LIMO-style quality-focused SFT against RL-based reasoning training on the same base model. The paper benchmarks against o1-preview and QwQ-32B-Preview (Table 1), both of which use methodologies substantially different from standard SFT. But the paper does not compare LIMO against an RL-based reasoning training run on the same base model (Qwen2.5-32B-Instruct) with comparable compute investment. Such a comparison would directly test the paper's implicit claim — that SFT on high-quality exemplars can match or approach what RL achieves — by holding the base model constant and varying only the training paradigm. The experiment design would: (a) run an RL-based mathematical reasoning training pipeline (e.g., GRPO with outcome-based reward on MATH or competition problems) on Qwen2.5-32B-Instruct, recording performance as a function of training compute; (b) compare against LIMO (SFT on 800 examples) at matched total FLOPs (including the cost of generating the LIMO dataset); (c) measure both in-domain and OOD generalization. A finding that quality-focused SFT approaches RL performance at substantially lower compute cost would strengthen the paper's practical claims. A finding that RL substantially outperforms SFT even with high-quality data would clarify that the LIMO approach is a strong SFT baseline but not an RL replacement, refining the paper's positioning relative to the inference-time scaling literature.

Practical Applications and Downstream Use Cases

1. Low-cost specialization of general-purpose foundation models for competitive mathematics. An organization that has already trained or licensed a general-purpose foundation model with strong mathematical pre-training (e.g., Qwen2.5-32B-Instruct or a comparable model from another family) can achieve competition-level mathematical reasoning — 63.3% on AIME24, 95.6% on MATH500 — using only 800 carefully curated SFT examples. The total training cost is negligible compared to pretraining or RL-based reasoning training: 15 epochs on 800 examples with a 32B model can be completed on a small GPU cluster in hours. The main cost is generating the high-quality training data, which requires access to frontier reasoning models (currently available via API for models like DeepSeek-R1 or through open-weight releases). For a mathematics education company building an automated tutoring system, a competition platform developing problem-solving tools, or a research lab needing a strong mathematical reasoner for downstream experiments, the LIMO recipe provides a concrete, replicable path to state-of-the-art performance at minimal fine-tuning cost — provided the base model is in the elicitable regime and frontier model access is available for data generation.

2. Efficient domain adaptation for multilingual mathematical reasoning. LIMO's strong cross-lingual transfer — 84.2% on CHMath, 91.1% on Gaokao, 83.9% on Kaoyan, all Chinese-language benchmarks, despite the LIMO training data being predominantly English — demonstrates that the fine-tuned reasoning patterns transfer across languages. This has immediate practical value for organizations serving non-English-speaking educational markets. Rather than curating large mathematics datasets in each target language (which requires language-specific problem sources and solution verification), a single small dataset of high-quality English exemplars can elicit mathematical reasoning that generalizes to Chinese, and plausibly to other languages with substantial pre-training representation. The consistent pattern — LIMO outperforming the base model by 56.9 points on CHMath, 19.0 on Gaokao, 35.7 on Kaoyan — suggests that the transfer is robust, not accidental. A deployment for, say, an Indian educational platform covering Hindi, Tamil, and English could potentially use a single LIMO dataset rather than language-specific fine-tuning pipelines, dramatically reducing data curation costs.

3. Data quality auditing for existing reasoning datasets. The paper's quality rubric and the L1–L5 gradient experiment (Figure 3) provide a methodology for evaluating the training effectiveness of existing reasoning datasets without running full fine-tuning experiments. A practitioner with a large reasoning dataset (hundreds of thousands of question–solution pairs) can: (a) score all solutions using the four-dimensional rule-based rubric; (b) bin the dataset by solution quality score; (c) fine-tune models on stratified subsets at each quality level using a small, fast experimental setup (e.g., a smaller model from the target family, fewer epochs); (d) measure how downstream performance varies with quality bin. This reveals whether the dataset's quality distribution is skewed toward high or low training effectiveness, and whether the dataset contains a "quality core" — a subset of high-scoring solutions that achieves most of the benefit — analogous to the LIMO-400 result (Figure 7) where 400 examples achieved 57.5% of the full 2,000-example AIME24 performance. For organizations that have already invested in large-scale reasoning data curation (e.g., NuminaMath's 860k examples), this auditing methodology could identify that the dataset's effective size is much smaller than its nominal size because most solutions fall into low-quality bins that contribute little or negative training signal. The 100k-example NuminaMath baseline's 32.3% average (versus the base model's 49.9%) suggests this auditing would reveal severe quality skew in some widely-used datasets.

4. Rapid prototyping of reasoning capabilities for new model releases. When a new foundation model is released with claims of strong mathematical pre-training, the LIMO approach provides a fast, low-cost test of whether the model is in the "elicitable regime." Rather than running large-scale fine-tuning experiments to assess the model's reasoning potential, a practitioner can: (a) apply the LIMO curation pipeline (difficulty filtering using a strong reference model, solution generation from frontier reasoning models, quality scoring, top-800 selection) to produce a dataset; (b) fine-tune the new model on this dataset with the LIMO recipe (15 epochs, no warmup, cosine decay); (c) evaluate on AIME24 and MATH500. The performance gap between the base model and the LIMO-fine-tuned model quantifies how much latent reasoning capability the model possesses — a large gap (like Qwen2.5-32B-Instruct's 16.5% → 63.3% on AIME24) indicates rich pre-trained mathematical knowledge; a small gap (like Qwen1.5-32B-Chat's trajectory, which the paper suggests would be poor) indicates insufficient pre-training. This provides a rapid diagnostic for model selection (which base model to build a reasoning system on) and for pre-training quality assessment (did the pre-training data mixture adequately cover mathematical reasoning?). The entire pipeline — data curation plus fine-tuning — could be executed in days, providing actionable guidance for downstream investment decisions.

When to Prefer This Method

The paper does not position LIMO against a clearly articulated set of named alternatives with explicit tradeoff conditions. It benchmarks against specific SFT baselines (NuminaMath-100k, OpenThoughts-114k) and against general reasoning models (o1-preview, QwQ-32B-Preview), but it does not develop a decision framework — "prefer LIMO when X, prefer RL-based reasoning training when Y, prefer large-scale SFT when Z" — grounded in empirical comparisons that isolate the relevant variables. The paper's contribution is primarily a demonstration that data-efficient SFT is possible, not a systematic comparison of when it should be chosen over alternatives.

The closest the paper comes to a tradeoff framework is the implicit guidance provided by the LIMO Hypothesis's two premises. A practitioner can infer:

• Prefer quality-focused, data-efficient SFT (LIMO-style) when: (a) the base model has strong domain-specific pre-training (validated by, e.g., non-trivial pass@1 on target benchmarks before fine-tuning — the paper's Qwen2.5-32B-Instruct starts at 16.5% on AIME24, 79.4% on MATH500); (b) access to frontier reasoning models is available for generating high-quality exemplars (the LIMO dataset required DeepSeek R1, R1-Distill-Qwen, and QwQ-32B); (c) the goal is broad generalization across related domains and languages (LIMO shows 45.8% OOD improvement); (d) computational budget for fine-tuning is extremely limited (15 epochs on 800 examples is trivially cheap). The paper's evidence strongly supports these conditions.

• Avoid this approach when: (a) the base model's pre-training lacks the target domain knowledge (Qwen1.5-32B-Chat at 9.2% AIME24, Figure 5, demonstrates failure); (b) frontier reasoning models are unavailable for data generation, and self-generated data from the base model has not been validated (the paper provides no evidence that self-bootstrapping works); (c) the target problems require capabilities beyond what any accessible reasoning model can demonstrate (the difficulty filtering explicitly excludes these, and the approach's ceiling is bounded by the generating model's capability).

However, the paper does not provide the experiments that would turn these inferences into an empirically grounded decision framework. It does not compare LIMO against RL-based training on the same base model at matched compute. It does not test whether larger-scale SFT with comparably rigorous quality filtering (e.g., 100k examples curated to the same quality standards as the 800 LIMO examples) would outperform or underperform the tiny-dataset approach. It does not characterize the tradeoff between exemplar quality and exemplar diversity — 800 examples at the very top of the quality distribution versus, say, 5,000 examples from a slightly wider quality band. Without these comparisons, a practitioner choosing between LIMO-style curation and alternatives is operating on extrapolation rather than evidence. The paper's contribution is establishing that one point in the design space (800 highest-quality exemplars, Qwen2.5-32B-Instruct) works dramatically better than the field's default approach (large-scale, loosely curated SFT). Mapping the full tradeoff surface — quality vs. quantity, SFT vs. RL, single-model vs. cross-family transfer — remains future work.