OpenThoughts: Data Recipes for Reasoning Models

🎯 Pitch

A systematic study of 1,000+ experiments reveals that a weaker teacher model (QwQ-32B) consistently outperforms a stronger one (DeepSeek-R1) when distilling reasoning traces, producing a 7B model that beats the best open-data competitors by up to 20 points on GPQA Diamond. Surprisingly, answer correctness filtering and deduplication provide no benefit once question quality is high, and mixing diverse problem sources actually hurts performance.

1. Executive Summary

This paper systematically investigates the design space for curating supervised fine-tuning (SFT) datasets to build reasoning models, conducting over 1,000 controlled ablation experiments across math, code, and science domains. Using Qwen-2.5-7B-Instruct as the student model and distilling from teacher models such as DeepSeek-R1 and QwQ-32B, the authors develop OpenThoughts3, a data pipeline spanning question sourcing — filtering — deduplication — repeated teacher sampling — teacher model selection. The resulting OpenThinker3-7B achieves 53% on AIME 2025, 51% on LiveCodeBench 06/24-01/25, and 54% on GPQA Diamond — improvements of 15.3, 17.2, and 20.5 percentage points over DeepSeek-R1-Distill-Qwen-7B, establishing it as the state-of-the-art open-data 7B reasoning model. Key counterintuitive findings include that a weaker teacher (QwQ-32B) consistently outperforms a stronger one (DeepSeek-R1) as the distillation source, that sampling multiple answers per question provides a 16× effective data scaling axis while mixing diverse question sources degrades performance, and that neither answer verification nor deduplication strategies yield meaningful gains over simply training on all available data — a finding that holds only when starting from already high-quality question sources selected through LLM-based difficulty and response-length filtering.

2. Context and Motivation

The Core Problem: We Don't Know How to Build Reasoning Dataset Recipes

The fundamental question this paper tackles is deceptively simple: what is the best recipe for creating a dataset that teaches a language model to reason? This matters because reasoning models—those that can produce long chains of thought before arriving at an answer—have emerged as one of the most significant advances in language model capabilities, achieving breakthrough performance on math, coding, and science benchmarks (Guo et al., 2025; OpenAI, 2024). However, the gap between what frontier labs can build and what the open research community can reproduce is exceptionally wide.

The practical manifestation of this gap is in the numbers: DeepSeek-R1-Distill-Qwen-7B achieves 38% on AIME 2025 and 33% on GPQA Diamond (Table 1), but the recipe for producing it is partially proprietary. Open-source efforts attempting to replicate these results through SFT on distilled reasoning traces have emerged (OpenR1, SkyT1, OpenThoughts-114K, LIMO), yet each project makes different choices about what questions to use, which teacher model to distill from, how to filter the resulting data, and whether to verify answer correctness. The field lacks a systematic understanding of which of these choices matters, which are interchangeable, and which are actively harmful.

This gap is significant for several reasons that the authors highlight throughout their exposition:

Barrier to entry for reasoning research: If building a competitive reasoning model requires guessing the right combination of question sources, teacher models, and filtering strategies, the exploration cost is prohibitively high. The paper notes that "this exploration is prohibitively expensive for many researchers due to the high costs of teacher inference and model training" (Section 1). A single training run on a million-scale dataset with Qwen-2.5-7B-Instruct consumed 25,000 A100 GPU hours on 512 GPUs (Appendix O), and the full pipeline represents an order of magnitude more computation. Without systematic guidance on what design choices matter, resource-constrained researchers cannot efficiently allocate their experimental budgets.
Opaque replication: When DeepSeek releases a distilled model, the community can inspect the outputs but not the curation decisions that produced them. This creates a situation where open-source models are "distillations of distillations"—each generation potentially inheriting and amplifying undocumented biases from upstream curation choices. By systematically ablating each pipeline step and releasing all artifacts (models, datasets, code at openthoughts.ai), this paper aims to make the full reasoning SFT pipeline transparent.
The SFT-only path to reasoning: One of the most striking findings from prior work is that supervised fine-tuning alone—without any reinforcement learning—can produce strong reasoning models. The DeepSeek-R1-Distill models (Section 1) demonstrate that distillation from a large reasoning model into a smaller one via SFT achieves performance competitive with RL-trained models. This makes SFT data curation the critical path for building open reasoning models, since SFT is far more accessible than RL training pipelines. Yet the design space for SFT reasoning data is enormous: the authors identify at least six distinct axes (question sourcing, mixing, filtering, deduplication, answer sampling, teacher selection), each with dozens of candidate choices.

Conflicting Intuitions and Practices in Prior Work

The paper is motivated by observing that different projects make contradictory design choices, often based on pre-existing heuristics rather than empirical validation:

Small curated datasets vs. massive diverse collections. At one extreme, projects like s1 and LIMO (Muennighoff et al., 2025; Ye et al., 2025) argue that a small number of carefully selected, high-quality prompts (~1,000 examples) is sufficient to induce reasoning behavior. At the other extreme, datasets like OpenR1-Math (Face, 2025) and Amazon-Nova-1.4M collect hundreds of thousands of examples from diverse question sources under the assumption that scale and diversity are primary drivers of performance. Figure 1 (left panel) provides the direct comparison: small curated datasets like s1.1 and LIMO underperform large-scale SFT datasets at every data scale, and the gap widens as scale increases. Yet both approaches have produced viable models, leaving the field without guidance on when curation beats scale.

Diversity vs. quality in question sources. A standard intuition in machine learning is that diverse training data leads to better generalization. Many reasoning dataset projects mix questions from many sources to cover a broad range of problem types. The authors explicitly test this assumption in Section 4.2 and find it to be wrong: mixing the top 2 code question sources outperforms mixing the top 16 by 5 percentage points on average (Table 4). The intuition is that lower-ranked sources introduce lower-quality questions that dilute the training signal, and the benefits of "diversity" conferred by including them do not compensate. This result directly challenges the prevalent practice of aggregating many question sources.

Verification as a necessary step. A common practice in reasoning data pipelines—including the authors' own prior releases OpenThoughts-114K and OpenThoughts2—is to filter out training examples where the teacher model's final answer is incorrect, using either programmatic verification (unit tests, symbolic math checking) or LLM-based judgment (Bespoke-Labs, 2025). The intuition is compelling: training on wrong answers should degrade performance. The paper systematically tests this assumption across all three domains (Section 4.5, Tables 7 and 44–46) and finds that no verification or answer filtering strategy outperforms simply training on all samples, even when the all-samples baseline contains known incorrect answers. This is one of the paper's most counterintuitive findings, and it directly contradicts the design of many prior datasets.

Teacher model strength. The standard assumption—manifest in most distillation work—is that a stronger teacher produces a stronger student. DeepSeek-R1 is empirically the strongest model on the evaluation benchmarks, outperforming QwQ-32B by 9% on CodeElo, 8% on GPQA Diamond, and 23% on JEEBench (Table 29). Yet the paper finds that QwQ-32B consistently produces better training data across all three domains (Table 8, with full results in Tables 47–49): QwQ-32B yields average improvements of 1.9 and 2.6 percentage points over DeepSeek-R1 for code and math, respectively. The mechanism is not fully understood—the authors speculate it may relate to the distribution of reasoning traces (length, style, error patterns) rather than final answer accuracy—but the finding undermines the simple "better teacher = better data" assumption.

Where Existing Approaches Fall Short

The authors identify specific limitations in prior work that their systematic study addresses:

No controlled comparisons. Prior reasoning datasets are developed as monolithic products: a team makes a series of design choices, releases the resulting dataset, and reports final model performance. There is no way to attribute performance gains to individual decisions. OpenThoughts3 provides the first large-scale factorial ablation of the reasoning SFT pipeline. Each of the 1,000+ experiments (Section 4) fixes a dataset size of 31,600 examples (chosen as a log-scale midpoint between 10K and 100K) and varies one pipeline component while holding others constant. This experimental design enables causal attribution: when response-length filtering improves math scores by 4% over random filtering (Table 5), the reader knows the gain is attributable to that filtering choice, not to concurrently varying question sources or teacher models.

No scaling analysis of individual components. Figure 3 shows the scaling behavior of the pipeline as each successive stage is applied—from "Question Source" alone through "Question Filter" to the final "Teacher Model" recipe. The gains from each stage compound, and the scaling curves shift upward. This demonstrates that the design choices are not merely local optima at the 31.6K scale but produce consistent improvements across orders of magnitude. Prior work could not make such claims because individual components were never isolated and scaled independently.

Inference heuristics without empirical backing. The paper identifies several widely-used heuristics in reasoning data curation that lack empirical validation: that deduplication improves dataset quality, that answer verification is essential, that diverse question sources outperform focused ones, and that stronger teachers produce better training data. By testing each of these systematically, the paper provides the field with evidence to replace heuristics with data-driven design choices.

No recipe that achieves parity with closed distillation. Prior to OpenThinker3-7B, the best open-data 7B reasoning model (Nemotron-Nano-8B) achieved an average benchmark score of 53.2 (Table 1), while the proprietary R1-Distill-Qwen-7B achieved 42.9. This 10+ point gap represented the cost of not having access to DeepSeek's internal data curation pipeline. OpenThinker3-7B achieves 55.3, surpassing not only R1-Distill-Qwen-7B but also Nemotron-Nano-8B, AceReason-7B, and Skywork-7B. This establishes that open-source data curation—with systematic experimentation—can exceed the performance of proprietary distillation.

How This Paper Positions Itself

The paper positions itself as both a systematic empirical investigation and an open infrastructure project. It is not proposing a single novel technique but rather a methodology for discovering effective data recipes, and it releases the resulting artifacts as public goods.

The framing is deliberately recipe-oriented: the paper walks through a pipeline of independent stages (Figure 2), ablates each one, selects the top-performing approach based on average benchmark score, and proceeds to the next stage with that selection fixed. This sequential greedy optimization reflects a practical engineering approach rather than a theoretical framework. The authors acknowledge this limitation explicitly (Section 6): they select strategies that maximize overall average performance rather than optimizing per-domain, and they assume some cross-domain transfer (e.g., training on math data helps science performance). Whether joint optimization would yield further gains remains an open question.

The paper also positions itself within the broader landscape of open-source reasoning model development by explicitly comparing against existing datasets (AM, Nemotron-Nano, s1.1, LIMO) at matched data scales in Figure 1, and against existing models (R1-Distill-Qwen-7B, Nemotron-Nano-8B, AceReason-7B, Skywork-7B) at matched model scales in Table 1. The consistent finding—OpenThoughts3 dominates across scales and benchmarks—establishes the paper's pipeline as a new strong baseline for open reasoning SFT research.

Notably, the paper does not explore reinforcement learning as a training regime, despite RL being a standard component of frontier reasoning models like o3 and DeepSeek-R1. The authors are explicit about this scope limitation (Section 1): "we did not explore datasets for reinforcement learning." The contribution is thus bounded to the SFT-only distillation paradigm, which the results demonstrate is already sufficient to achieve state-of-the-art open-data performance at the 7B scale.

3. Technical Approach

3.1 Reader Orientation

This paper builds a data generation pipeline — a systematic, multi-stage recipe — for curating supervised fine-tuning (SFT) datasets that teach a base language model to produce long chains of reasoning ("thinking tokens") before giving final answers. The system solves the problem of how to efficiently allocate a fixed budget of teacher model inference and training compute across many competing design choices (which questions to use, how to filter them, which teacher to distill from, whether to verify answers) to maximize downstream reasoning accuracy on math, code, and science benchmarks. The shape of the solution is a greedy sequential optimization: at each of six pipeline stages, the authors run controlled ablation experiments holding a dataset size of 31,600 examples, select the top-performing strategy based on average benchmark score across eight evaluation tasks, and proceed to the next stage with that choice locked in — producing a final "recipe" that is then scaled to 1.2 million examples.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six sequential stages, illustrated in Figure 2 and the detailed pipeline diagram in Figure 4:

Question Sourcing — Select sources of raw questions for each domain (math, code, science). These can be human-written (e.g., StackExchange forums, competition problems), fully synthetic (LLM-generated from scratch), or semi-synthetic (LLM-generated using existing text corpora as seeds). The output is a large pool of candidate questions per domain.
Question Mixing — Decide how many distinct question sources to combine. The ablation sweeps from 1 to 16 top-ranked sources per domain, measuring whether diversity from additional sources compensates for the lower average quality they introduce.
Question Filtering — From each source (which may contain millions of candidate questions), select a subset of approximately 31,600 high-quality questions using LLM-based methods (difficulty scoring, response-length heuristics, similarity to exemplar questions) or classical methods (fastText classifiers, embedding similarity).
Deduplication and Repeated Teacher Sampling — Remove near-duplicate questions (exact or fuzzy string matching) to avoid wasting teacher compute on redundant prompts. Then, for each retained question, query the teacher model multiple times (1×, 4×, or 16×) to generate diverse reasoning traces, trading off question diversity against answer diversity.
Answer Filtering — (Later determined to be unnecessary). Apply verification strategies (LLM-based judgment, majority consensus, length-based heuristics, fastText quality classifiers) to filter out low-quality or incorrect teacher responses before training.
Teacher Model Selection — Compare different teacher models (DeepSeek-R1, QwQ-32B, Phi-4-Reasoning-Plus) as the source of reasoning traces, measuring downstream student performance to determine which teacher produces the most effective training data.

Information flows linearly: candidate questions enter at Stage 1 → sources are ranked and mixed at Stage 2 → questions are filtered by quality at Stage 3 → near-duplicates are removed and each question is annotated multiple times at Stage 4 → generated answers are optionally filtered at Stage 5 (this step is ultimately skipped) → the teacher model is fixed at Stage 6 → the final dataset is scaled up to target size.

3.3 Roadmap for the Deep Dive

First, the experimental design — training setup, evaluation suite, dataset size choices, and cross-validation strategy — since all downstream ablations depend on this fixed methodology for measuring "what works."
Second, the question sourcing stage (Section 4.1), which defines the raw material for the entire pipeline. Understanding how questions are generated (synthetic, semi-synthetic, non-synthetic) and how their quality is measured is prerequisite to subsequent filtering and mixing decisions.
Third, the question mixing stage (Section 4.2), which tests the diversity-vs-quality tradeoff by comparing models trained on 1, 2, 4, 8, or 16 top-ranked sources. This establishes the principle that narrower, higher-quality sources dominate.
Fourth, the question filtering stage (Section 4.3), which addresses the scale problem: since source datasets can contain millions of questions, how do you select a high-quality subset? This introduces LLM-based filtering (difficulty scoring, response-length heuristics, AskLLM similarity) and classical filtering (fastText, embeddings).
Fifth, the deduplication and repeated sampling stage (Section 4.4), which explores a second axis of data scaling: instead of finding more unique questions, annotate the same questions multiple times with different teacher responses.
Sixth, the answer filtering stage (Section 4.5), which tests whether removing low-quality teacher responses improves student performance — and produces the surprising null result that it does not.
Seventh, the teacher model selection stage (Section 4.6), which tests the counterintuitive finding that a "weaker" teacher produces better training data.
Eighth, the scaling and final dataset construction (Section 5), which shows how the winning strategies from each stage are combined and scaled to 1.2M examples, and validates that the pipeline improvements compound across scales.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical pipeline optimization paper whose core idea is that the optimal SFT reasoning dataset recipe is discoverable through systematic, controlled ablation of each pipeline stage independently, and that the resulting recipe produces a state-of-the-art open-data reasoning model when scaled.

Experimental Design: Controlled Ablations at 31,600 Examples

The authors operationalize "better dataset" as "produces a higher-performing downstream model," and they measure downstream performance using a fixed evaluation protocol. This is not a paper that measures intrinsic dataset properties (e.g., perplexity, diversity metrics) — it measures the end-to-end causal effect of each curation choice on model accuracy.

Fixed dataset size. Every ablation experiment uses exactly 31,600 examples for training. The authors choose this number because it is the log-scale midpoint between 10,000 and 100,000 (since $\sqrt{10} \approx 3.16$ , so $10,000 \times 3.16 = 31,600$ ). The rationale: this scale is "small enough to be cost-effective yet large enough to provide a meaningful signal" (Section 4, Training paragraph). Each ablation run thus requires annotating 31,600 questions with the teacher model and fine-tuning the student model on the resulting dataset.

Student model. All ablation experiments fine-tune Qwen-2.5-7B-Instruct (Qwen2.5-Team, 2024), a 7-billion parameter instruction-tuned model. This choice is justified by the model's strong baseline performance and wide adoption in the open-source community. The base model achieves 24.0 average score across the evaluation benchmarks (Table 1), representing a non-trivial but far from saturated starting point that leaves room for reasoning data to make a difference.

Training hyperparameters. The authors define four hyperparameter sets based on dataset size, shown in Table 9:

Hyperparameter Set	Dataset Size	Learning Rate	Batch Size	Epochs	Packing
Micro	< 1K	1e-5	32	13	No
Small	1K–3.16K	2e-5	96	7	No
Medium	3.16K–31.6K	4e-5	128	5	No
Large	> 31.6K	8e-5	512	5	Yes

All ablation experiments fall into the Medium set: learning rate $4 \times 10^{-5}$ , batch size 128, 5 epochs, no packing. The optimizer is AdamW with betas $(0.9, 0.999)$ , weight decay 0, cosine learning rate schedule with warmup ratio 0.1. Training uses DeepSpeed v3 without memory offloading, with persistent dataloader workers (num_workers=4).

Why these hyperparameters? The authors performed hyperparameter sweeps to tune learning rate, batch size, and epochs at each scale. The key constraint is that smaller datasets require more epochs and smaller batch sizes to ensure sufficient optimizer steps. At 31,600 examples with batch size 128 and 5 epochs, there are $31600 / 128 \times 5 \approx 1234$ training steps. The authors deliberately do not use example packing at this scale because packing reduces the number of effective training steps (multiple examples share one sequence), and they want to ensure adequate optimization. (At the Large scale used for the final 1.2M model, packing is enabled to save compute time.)

Chat template and system prompt. The authors use the DeepSeek-R1 chat template: reasoning traces are enclosed in <think> and response tokens (replacing the SkyT1 template's <|begin_of_thought|> and <|end_of_thought|> tokens used in earlier OpenThoughts releases). Table 11 shows that this switch has negligible effect on performance (e.g., AIME24: 31.3% with SkyT1 template vs. 32.7% with R1 template). An ablation in Appendix D.5 tests removing the chat template entirely and finds that while explicitly enabling reasoning helps dramatically (AIME25: 45.3% reasoning-on vs. 2.0% reasoning-off with a Llama-Nemotron model), even when reasoning is "disabled," many responses still begin with thinking tokens (1681 out of 3127 samples), suggesting the model has internalized the reasoning pattern from training.

Evaluation protocol. The primary metric is average accuracy across eight benchmarks spanning three domains:

Math: AIME24 (30 questions, integer answers 0–999), AMC23 (40 questions, numerical answers), MATH500 (500 diverse problems). Reported as mean accuracy with standard error over 10 iterations (AIME24, AMC23) or single pass (MATH500).
Code: CodeElo (391 problems, unit-test-based execution accuracy with Elo-calibrated difficulty), CodeForces (453 problems, unit-test accuracy), LiveCodeBench 05/23-05/24 (511 problems with iterative repair). Reported as mean accuracy with standard error over 3 iterations (CodeElo, CodeForces) or 6 iterations (LiveCodeBench).
Science: GPQA Diamond (198 multiple-choice questions, graduate-level), JEEBench (515 questions spanning Physics, Chemistry, Mathematics from the IIT JEE-Advanced exam). Reported as mean accuracy with standard error over 3 iterations.

All evaluations use the Evalchemy framework (Raoof et al., 2025) with unified generation hyperparameters: temperature 0.7, top_p = 1.0, max_new_tokens = 32,768. Evalchemy partitions each benchmark into independent shards, runs them in parallel across GPUs, and aggregates per-shard metrics in real time.

Why these benchmarks? The authors choose benchmarks that require multi-step reasoning with verifiable answers. Math and code have unambiguous correctness criteria (exact answer matching, unit test passing). Science is included because prior work showed reasoning transfers across domains, and GPQA Diamond is specifically designed to be resistant to shallow pattern matching ("Google-proof"). The inclusion of both "in-distribution" benchmarks (used during pipeline optimization) and "held-out" benchmarks (AIME25, HMMT 02/25, LiveCodeBench 06/24-01/25, HLE MCQ — only evaluated after all pipeline decisions are frozen) provides a test of generalization and prevents overfitting the pipeline to specific evaluation sets.

Contamination control. The authors apply a rigorous decontamination protocol (Appendix F) before any training. Each training sample is compared against each evaluation sample using two criteria: (1) normalized InDel similarity computed via Longest Common Subsequence: $\text{indelsim} = 100 \times \text{LCSlength}(s_1, s_2) / \max(|s_1|, |s_2|)$ , with a threshold of 75% (samples exceeding this similarity are considered contaminated), and (2) 13-gram overlap using the Qwen2-7B-Instruct tokenizer (if any 13-gram is shared between training and evaluation samples, the training sample is discarded). A sample is removed if either criterion flags it.

Why these thresholds? The authors tuned them on a manually constructed testbed of 3,092 contaminated samples (including exact matches, embedding-in-context, synonym-substituted, and reformatted variants) and 3,000 uncontaminated samples (Appendix F, Figure 7). The chosen thresholds achieve a 99.6% true negative rate (missing only 12 contaminated samples out of 3,092) while falsely flagging only 1.4% of uncontaminated samples as contaminated.

Cross-validation strategy for strategy selection. A subtle but important design choice: the authors must select the "best" strategy at each pipeline stage without overfitting to the test set. Their approach is greedy sequential selection: at each stage, they run ablations on all candidate strategies using the fixed 8-benchmark evaluation suite, select the strategy with the highest average accuracy, and proceed to the next stage with that choice locked in. This is not cross-validated per stage — the same 500-question MATH500 test set and the same AIME24/AMC23 problems are used for strategy selection across all stages. The held-out benchmarks (AIME25, HMMT, LCB 06/24-01/25, HLE) serve as the cross-validation: if the pipeline were overfit to the evaluation benchmarks, held-out performance would degrade relative to in-distribution baselines. Table 1 shows this is not the case: OpenThinker3-7B achieves the best held-out scores across all four benchmarks, suggesting the pipeline selections generalize.

Why greedy sequential and not joint optimization? The authors acknowledge this limitation explicitly (Section 6, Open Direction 1): "we selected strategies that maximized overall average benchmark performance, rather than optimizing for each domain individually." Joint optimization over all six stages would require an exponentially larger number of experiments (product of choices rather than sum). The greedy approach is a practical compromise that, based on the scaling results in Figure 3, appears to produce compounding gains without destructive interactions between stages.

Question Sourcing (Section 4.1): Finding Raw Questions

The first pipeline stage determines where the raw questions come from. The authors categorize question sources into three types:

Fully synthetic — an LLM generates questions from scratch using template prompts, with little-to-no seed material. Examples: CodeAlpaca (Chaudhary, 2023), CamelChemistry (Li et al., 2023a).
Semi-synthetic — an LLM uses existing text corpora as seeds to formulate questions. Examples: AutoMathText (Zhang et al., 2024) takes unstructured math text from the web and prompts GPT-4o-mini to convert each snippet into a competition-style problem; TigerLabMath (Yue et al., 2024) applies similar conversion to web-crawled math content.
Non-synthetic — humans wrote the questions. Examples: StackExchange forums (CodeGolf, Physics, Biology), competition problems (AIME, CodeForces), chatbot interaction logs (ShareGPT).

Why this taxonomy matters: it represents a spectrum of cost and quality. Non-synthetic questions are high-quality but limited in quantity and potentially subject to copyright or contamination concerns. Synthetic questions are unlimited in quantity but may lack the nuance, internal consistency, or difficulty distribution of human-written problems. The ablation tests which extreme — or which blend — produces the best training signal.

Experimental procedure. For each candidate source, the authors generate 31,600 questions. If a source produces fewer than 31,600 questions, they repeat or resample until reaching the target count. For sources requiring LLM generation, they use GPT-4o-mini as the generator (chosen for low cost and high throughput). All questions, regardless of source, are then annotated with DeepSeek-R1 as the teacher model (this teacher choice is later ablated in Section 4.6). This ensures that any performance differences are attributable to question quality, not teacher quality.

Results summary (Table 3 and full results in Tables 32–34). For code, the top-performing sources are:

StackExchange CodeGolf (85.9K available questions, 38.8 average score): human-written puzzles focused on minimal-character solutions.
OpenCodeReasoning (459K available, 38.4 average): a large synthetic dataset spanning 28,319 unique competitive programming problems across 735,255 Python samples.
KodCode-V1 (384K available, 37.7 average): fully synthetic, diverse coding questions ranging from algorithmic to package-specific knowledge.

The gap between top and bottom code sources is 17.2 points (38.8 vs. 21.6 for bugdaryan/sql-create-context-instruction), demonstrating that question source quality dwarfs other pipeline choices in importance.

For math, the top sources are:

OpenMath-2-Math (>1M available, 38.1 average): the MATH subset of OpenMathInstruct2, containing synthetic questions generated from MATH and GSM8K seed problems.
NuminaMath-1.5 (853K available, 37.4 average): scanned competition math problems from sources like AoPS.
MathPile (99.5K available, 36.2 average): a new dataset the authors created by taking unstructured math text from GAIR/MathPile and using GPT-4o-mini to convert each snippet into a competition-style question (prompt in Figure 18).

For science, the top sources are:

StackExchange Physics (547K available, 34.3 average): human-written physics questions from the Physics StackExchange forum.
Organic Chemistry PDF Pipeline (46.2K available, 34.0 average): a new dataset the authors created through a multi-stage extraction pipeline (detailed in Appendix R.1.3): first, use Gemini-2.0-Flash-Lite to OCR text from organic chemistry PDF textbooks and SCP-116K papers (prompt in Figure 24); then use GPT-4o-mini to extract question-answer-solution triplets from the OCR'd text (prompt in Figure 23); then refine the extracted questions to be self-contained and exam-like (prompt in Figure 22); then filter to retain only questions related to math, science, or code (prompt in Figure 21); then further filter for organic chemistry specifically (prompt in Figure 20). This 5-stage synthetic extraction pipeline is the most complex question-generation strategy in the paper.

Key finding: no clear pattern favors synthetic over non-synthetic. StackExchange CodeGolf (human-written) outscores KodCode-V1 (fully synthetic) for code; OpenMath-2-Math (semi-synthetic) outscores NuminaMath-1.5 (human-written) for math; StackExchange Physics (human-written) and the Organic Chemistry PDF Pipeline (semi-synthetic extraction) are essentially tied for science. This suggests that the relevant variable is not provenance but some latent quality dimension that correlates imperfectly with both human authorship and synthetic generation methodology.

Why this finding matters: it justifies the subsequent filtering stage. If the best sources were unambiguously a single type, one could simply select that type and scale it. Since quality varies within types, systematic filtering is necessary to extract the best subset from any source.

Question Mixing (Section 4.2): Diversity vs. Quality

Given the ranked list of question sources from Section 4.1, the mixing stage asks: how many of the top-ranked sources should be combined? The intuition for mixing many sources is that diverse question types, styles, and difficulty levels should improve generalization. The counter-intuition is that lower-ranked sources contain lower-quality questions (ambiguous phrasing, incorrect answers in the original source, inappropriate difficulty), and adding them dilutes the training signal.

Experimental setup. For each domain separately, the authors take the top- $N$ ranked sources from Section 4.1, sample $31,600 / N$ questions from each, and concatenate them. Values of $N$ tested: 1, 2, 4, 8, 16.

Results (Table 4 and Tables 35–37). The pattern is consistent across all three domains. For code: mixing the top 2 sources achieves an average score of 41.3, while mixing the top 16 achieves 36.4 — a 5-point degradation. For math: using only the top 1 source (OpenMath-2-Math) achieves 37.6, while mixing 16 sources achieves 33.8 — a 3.8-point degradation. For science: top 2 sources achieve 33.7 vs. 30.8 for top 16 — a 2.9-point degradation.

Design choice for final pipeline. The authors select:

1 source for math: OpenMath-2-Math alone
2 sources for code: StackExchange CodeGolf and OpenCodeReasoning
2 sources for science: StackExchange Physics and OrganicChemistry-PDFs

Why this matters. This result directly contradicts the practice in many prior reasoning datasets (OpenR1, OpenThoughts2, Amazon-Nova) that aggregate questions from many sources. The finding suggests that reasoning capability is not primarily driven by surface-form diversity (different question phrasing styles, different problem domains within a subject) but by some deeper quality property that is concentrated in the best sources and diluted when weaker sources are included. The mechanism may be that lower-quality questions contain subtle errors, unrealistic problem structures, or inconsistent difficulty that confuse the student model during training, and the negative impact of these flawed examples outweighs any benefit from seeing a wider variety of problem types.

A subtle limitation. The mixing experiment tests uniform mixing (equal numbers from each source). A weighted mixing strategy that includes more questions from top sources and fewer from lower-ranked ones might recover some diversity benefits without the full dilution cost. The authors do not explore this — they treat mixing as a binary source-selection problem (include or exclude each source entirely) rather than a continuous allocation problem.

Question Filtering (Section 4.3): Selecting High-Quality Subsets

Since each source dataset can contain millions of candidate questions, answering and training on all of them is infeasible. The filtering stage selects approximately 31,600 high-quality questions from the chosen sources. The authors test a wide range of filtering strategies, spanning two categories: LLM-based and classical.

LLM-based filtering strategies:

Difficulty-based selection (code domain winner): Ask GPT-4o-mini to rate each question on a 1–10 difficulty scale using a rubric specific to the domain. For code, the rubric is based on the ICPC problem difficulty scale (Level 1: basic implementation needing only simple loops; Level 10: problems requiring novel approaches not covered in standard competitive programming material — full rubric in Figure 27). For math, the rubric uses the AoPS difficulty scale (Level 1: AMC 8 1–20; Level 10: historically hard IMO problems — full rubric in Figure 29). For science, the rubric uses international science olympiad standards (Level 1: basic recall; Level 10: legendary difficulty — full rubric in Figure 30). The strategy retains only the questions rated as most difficult (highest numerical scores).
Response-length selection (math and science domain winners): Use an LLM to respond to each question, then select questions where the LLM's response was longest. The intuition is that questions eliciting longer responses from a capable model are more complex, open-ended, or require more reasoning steps — and thus are better training examples. The authors test multiple LLMs for response generation: GPT-4o-mini, GPT-4.1-mini, and GPT-4.1-nano. The prompt simply asks the LLM to answer the question; the response length in characters is used as the filtering criterion.
AskLLM selection: Ask GPT-4o-mini to rate each question on a 1–100 scale based on similarity to a provided set of positive exemplar questions (high-quality competition problems from S1.1, NuminaMath, CodeForces) and dissimilarity to negative exemplar questions (simple arithmetic, trivial coding tasks). The prompt (Figure 28) provides 5 positive and 5 negative examples and asks the model to return a score where "100 means exactly like the positive questions whereas 1 is exactly like the negative questions." Structured decoding extracts a numerical response.

Classical filtering strategies:

fastText classification: Train a fastText classifier (Joulin et al., 2017) to distinguish "high-quality" from "low-quality" questions based on n-gram features. The training data consists of: positive examples from sources that empirically produce strong models (e.g., CodeForces, NuminaMath, S1.1), and negative examples from sources that empirically produce weak models (e.g., Lap1official/Math, bugdaryan/sql-create-context-instruction). The classifier uses bigrams (n=2), hidden dimension 256, learning rate 0.1, trained for 3 epochs with minimum n-gram count of 3. The score for a question is the classifier's confidence that it belongs to the positive class.
Embedding-based selection: Embed a set of positive and negative exemplar questions using OpenAI's text-embedding-3-large model. For each candidate question, compute its mean cosine similarity to positives minus its mean cosine similarity to negatives. Select questions with the highest score.
Random selection: The baseline — uniformly sample 31,600 questions without any filtering.

Experimental setup. For each filtering strategy, the authors start with the full pool of questions from the chosen sources (Section 4.2) — which can be millions of candidates — apply the filter to select approximately 31,600 questions, generate teacher responses using DeepSeek-R1, and train a student model.

Results (Table 5 and Tables 38–40). The key findings are:

For code, difficulty-based filtering achieves the highest average score (43.0), followed by response-length selection with GPT-4.1-nano (42.2) and AskLLM (41.6). Random selection achieves 39.7. Embedding-based selection performs worst (36.9). The gain from the best filter over random is approximately 3.3 points (8.3% relative improvement).

For math, response-length selection with GPT-4.1-mini achieves the highest score (41.9), an improvement of 6.7 points over random (35.2). AskLLM achieves 36.3. Notably, the response-length selection with GPT-4o-mini produces an anomalous result of 6.8 average score — essentially random performance. The authors do not explain this outlier, but it may indicate that GPT-4o-mini's responses for math questions are so variable in quality that length becomes an anti-signal.

For science, response-length selection with GPT-4.1-mini again wins (35.9), though the margin over random (33.8) is only 2.1 points. The best fastText filter (positives: SciQ; negatives: Wikipedia + Arxiv) achieves 35.1.

Important patterns across domains. LLM-based filtering consistently outperforms classical filtering (fastText, embeddings). This is not because classical filters are inherently worse — fastText and embedding-based filters have been highly successful in pretraining data curation (Soldaini et al., 2024; Li et al., 2024) — but because the quality signal for reasoning questions is likely more nuanced than surface-level n-gram or embedding similarity can capture. A fastText classifier trained on CodeForces vs. SQL questions can distinguish competitive programming from database queries, but cannot distinguish a well-posed competitive programming problem from a poorly-posed one. LLM-based filters can assess internal logical consistency, appropriate difficulty, and answerability — dimensions that n-gram overlap misses.

Why stronger LLMs produce better filters. For response-length filtering, stronger models (GPT-4.1-mini > GPT-4.1-nano > GPT-4o-mini) produce better filtering results for math and science. The mechanism: a stronger model's response length is a better proxy for question complexity because the model engages more deeply with genuinely difficult questions rather than rambling on simple ones. A weaker model might produce long responses for trivial questions (due to verbosity or confusion) and short responses for hard questions (due to giving up), making length an unreliable quality signal.

Design choices for final pipeline. Based on these results, the authors select:

Difficulty-based filtering with GPT-4o-mini for code questions
Response-length filtering with GPT-4.1-mini for math and science questions

Why difficulty for code but response-length for math? The paper does not provide a mechanistic explanation, but one plausible account is that code difficulty is more objectively rateable by an LLM (the ICPC rubric maps clearly to data structure and algorithm complexity) while math difficulty is harder to judge from the question text alone because solution difficulty often depends on non-obvious mathematical insights. For math, response length — a behavioral measure of how much reasoning another model deploys — may be a more reliable proxy for the question's instructional value.

Deduplication and Repeated Teacher Sampling (Section 4.4): A Second Axis of Scale

Given a set of filtered questions, the next stage explores two competing strategies for increasing dataset size: (1) find more unique questions, or (2) annotate the same questions multiple times with different teacher responses.

Deduplication levels. The authors test three degrees of deduplication strictness:

No deduplication: Keep all questions as-is, including any that appear multiple times in the source (e.g., the same competition problem scraped from multiple forums).
Exact match deduplication: Remove questions that are character-for-character identical to another question in the pool.
Fuzzy deduplication: Remove questions whose normalized InDel similarity exceeds a threshold. Following the same formula from decontamination (Appendix F): $\text{indelsim} = 100 \times \text{LCSlength}(q_1, q_2) / \max(|q_1|, |q_2|)$ . The threshold is set empirically (Appendix R.3).

Repeated sampling levels. For each retained question, the teacher model is queried multiple times to generate distinct reasoning traces: 1× (one response per question, maximizing question diversity for a fixed budget), 4× (moderate repetition), and 16× (many responses per question, maximizing answer diversity at the cost of question diversity).

Experimental design. The authors sweep all 9 combinations: 3 deduplication levels × 3 sampling multipliers. For each combination, they generate 31,600 total question-answer pairs. For example, with exact dedup and 16× sampling, they select 31,600/16 = 1,975 unique questions and annotate each 16 times.

Results (Table 6 and Tables 41–43). The key finding is that multiple sampling compensates for fewer unique questions. For code (Table 41), the combination "No Dedup + 16× sampling" achieves an average score of 41.3, essentially tied with "Exact Dedup + 4× sampling" (41.3) and "Fuzzy Dedup + 4× sampling" (41.2), while "No Dedup + 1× sampling" achieves 39.8. The spread is only ~1.5 points across all combinations.

For math (Table 42), the pattern is slightly different: "Exact Dedup + 1× sampling" achieves the highest score (41.7), followed by "Exact Dedup + 16× sampling" (40.1) and "Exact Dedup + 4× sampling" (39.2). The no-deduplication variants tend to perform worse (36.5–38.3). This suggests that for math, deduplication is mildly beneficial, likely because math questions are more prone to near-duplication (the same competition problem with minor rewording appearing across sources).

For science (Table 6, reproduced in the main text), "Exact Dedup + 16× sampling" achieves the highest score (36.2), with "Fuzzy Dedup + 16× sampling" close behind (36.1). The baseline "No Dedup + 1× sampling" achieves 35.5. Again, the spread is small (~1 point).

Critical finding: deduplication does not matter much given high-quality source questions. This is different from the pretraining data literature, where deduplication is a critical step for improving model performance and reducing memorization (Lee et al., 2022). The authors' interpretation is that for reasoning SFT, question diversity may be less important than answer diversity — seeing multiple different solution paths for the same problem teaches the model about the solution space, which is more valuable than seeing many different problems each solved once. This is consistent with the question mixing results (Section 4.2), where diversity of lower-quality sources hurt performance.

Why 16× sampling matters practically. Multiple sampling provides a scalable axis for dataset expansion that does not require finding new high-quality questions. If the best question source (e.g., OpenMath-2-Math) has a finite pool of high-quality problems, 16× sampling lets you multiply the effective dataset size by 16 without needing to include lower-quality questions from other sources. This is a key insight for building large SFT datasets: maximize answer diversity for a fixed set of high-quality questions rather than maximizing question diversity by including lower-quality questions.

Design choices for final pipeline. The authors select:

Math and science: Exact deduplication + 16× sampling
Code: No deduplication + 16× sampling (the second-best strategy for code after "No Dedup + 4×," chosen for better scalability since 16× provides more room for dataset expansion)

Why different deduplication choices per domain? For code, deduplication was not beneficial — likely because the code question sources (CodeGolf, OpenCodeReasoning) have low natural duplication rates since coding problems tend to be unique in their specifications. For math and science, exact deduplication provided a small benefit, likely removing duplicate competition problems that appear across multiple scraping sources.

Answer Filtering (Section 4.5): The Surprising Null Result

A natural step in any data pipeline is to filter out low-quality annotations — in this context, teacher responses that are incorrect, incoherent, or otherwise poor. The authors test a comprehensive set of answer filtering strategies:

Filtering strategies tested:

GPT Verification: Ask GPT-4o-mini whether the teacher's answer is correct for the given question (prompt in Figure 32 for math/science, Figure 33 for code). Use structured decoding (temperature 0.0, top_p = 1.0, presence penalty 1.0) to extract a boolean judgment. Filter out responses judged incorrect.
Majority Consensus Selection: For each question with 16 teacher responses, provide all responses to GPT-4o-mini and ask it to identify which responses agree with the majority final answer (prompts in Figures 31, 36, 37 for math, science, code respectively). For math and science, only the last 1,000 characters of each response are shown to focus on the final answer. Filter out responses that disagree with the majority.
fastText Selection: Train a fastText classifier to distinguish high-quality from low-quality question-answer pairs. For math/science, positives come from S1.1 (DeepSeek-R1 responses) and negatives from the same questions answered by GPT-4o-mini. For code, positives are CodeForces questions with DeepSeek-R1 responses, negatives are CodeForces with GPT-4o-mini responses. The query format is "Question: {question} \nAnswer: {answer_column}". Classifier hyperparameters match those in Section 4.3 (n=2, dim=256, lr=0.1, 3 epochs).
Length-based Selection: For each question, retain only the 8 shortest or 8 longest responses out of 16 (reducing dataset size by half). The intuition for "shortest" is that concise reasoning traces are likely correct, as verbose reasoning often indicates uncertainty (supported by analysis in Appendix I.3 showing incorrect responses are significantly longer than correct ones — e.g., for DeepSeek-R1-Distill-Qwen-7B on AIME24, incorrect responses average 18,198 tokens vs. 7,817 for correct ones). The intuition for "longest" is that longer reasoning traces provide richer training signal.
Non-English Removal: Use GPT-4o-mini to detect responses containing non-English content (prompt in Figure 34) and filter them out.
Long Paragraph Removal: Use GPT-4o-mini to detect responses containing paragraphs with >750 words (prompt in Figure 35) and filter them out.
Python Tag Filtering (code only): Remove responses that lack Python code tags ("'python).
Random Filtering: A compute-controlled baseline — randomly subsample to 31,600 examples from the initial pool.
No Filtering (compute-uncontrolled baseline): Keep all generated responses, resulting in a larger dataset (up to 63,200 examples, since the starting pool is doubled to provide headroom for filtering).

Experimental setup. To ensure filtered datasets still reach 31,600 examples, the authors first generate 63,200 total question-answer pairs (twice the target size), apply each filtering strategy to remove some fraction, then sample 31,600 from the survivors. The "No Filtering" baseline has an unfair advantage: it trains on all 63,200 examples, providing more total training signal. The "Random Filtering" baseline is the fair comparison: same dataset size, no quality-based selection.

Results (Table 7 and Tables 44–46). The headline result is that no filtering strategy meaningfully outperforms training on all available data:

For math (Table 45): "No Filtering" achieves 41.9, while the best filtering strategy (Random Filtering) achieves 41.6 — a 0.3-point difference well within standard error. GPT Verification achieves 40.0. Removing long paragraphs degrades performance to 38.0.

For code (Table 44): "No Filtering" achieves 42.2, essentially tied with fastText Selection (42.3) and Shortest Answers (42.0). All filtering strategies cluster within a 2.3-point range.

For science (Table 46): "No Filtering" achieves 38.3, with Longest Answers (37.5) and Removing Non-English (37.4) close behind. GPT Verification achieves 35.6.

Why does answer filtering not help? The authors do not provide a definitive mechanism, but the evidence points to two contributing factors. First, the teacher models (DeepSeek-R1 in these ablations) are sufficiently capable that their answers, even when incorrect, contain useful partial reasoning. Incorrect solutions may demonstrate important reasoning patterns — attempting a valid approach, identifying a dead end, and pivoting — that are absent from correct-only trajectories. Filtering them out removes this instructional content. Second, the verification methods themselves introduce errors: GPT-4o-mini may incorrectly classify correct answers as wrong (false positives in filtering) or miss incorrect answers (false negatives), and majority consensus may eliminate correct but unconventional solutions that happen to be outvoted.

Design choice for final pipeline. Based on these results, the authors skip answer filtering entirely. This is a significant departure from their own prior work (OpenThoughts-114K used LLM-based verification, OpenThoughts2 used unit test verification) and from many other reasoning datasets.

Why this matters. This finding implies that the computational cost of answer verification — which can be substantial when using LLM judges or executing unit tests on large-scale datasets — is not justified by downstream performance gains. It also simplifies the pipeline considerably: the final OpenThoughts3-1.2M is built without any answer filtering step.

A crucial caveat. This result holds only when starting from already high-quality questions filtered through the earlier pipeline stages. The authors are not claiming that answer filtering is useless in general — if the question pool included many trivial or ill-posed questions, answer verification might provide a stronger signal. The result is specific to the regime where questions have already been selected for quality via LLM-based filtering (Section 4.3).

Teacher Model Selection (Section 4.6): When Weaker Teachers Produce Better Students

The final pipeline stage ablates which teacher model's reasoning traces produce the best training data. The authors test three candidate teachers:

DeepSeek-R1 (Guo et al., 2025): The strongest model overall on the evaluation benchmarks. A 671B-parameter mixture-of-experts model trained with reinforcement learning. Achieves 76.0 on AIME24, 98.8 on AMC23, 73.7 on GPQA Diamond, 53.7 on CodeElo (Table 29). This is the default teacher used in all previous pipeline stages.
QwQ-32B (Qwen-Team, 2025): A 32B-parameter model that emphasizes "questioning" in its reasoning process. Empirically weaker than DeepSeek-R1 on most benchmarks: 78.3 on AIME24 (slightly better), but 44.3 on CodeElo (9.4 points worse), 65.0 on GPQA Diamond (8.7 points worse), and 69.9 on JEEBench (22.4 points worse). Overall average: 64.2 vs. DeepSeek-R1's 65.3.
Phi-4-Reasoning-Plus (Abdin et al., 2025): A 14B-parameter model. Performs poorly on code benchmarks (0.8 on LiveCodeBench, 2.4 on CodeElo, 3.5 on CodeForces) because it often fails to output code with proper markdown tags that Evalchemy can parse. Overall average: 45.2.

Experimental setup. For each domain separately, the authors generate 31,600 question-answer pairs using each teacher model (keeping all other pipeline choices — question sources, filtering, deduplication, sampling — fixed at the best settings determined in previous stages). Sampling hyperparameters are kept constant across teachers. The student (Qwen-2.5-7B-Instruct) is fine-tuned on each dataset and evaluated on the 8-benchmark suite.

Results (Table 8 and Tables 47–49). The counterintuitive finding: QwQ-32B consistently outperforms DeepSeek-R1 as a teacher across all three domains, despite being empirically weaker.

For math: QwQ-32B as teacher achieves average score 44.2, vs. 41.6 for DeepSeek-R1 — a 2.6-point improvement. This is driven by dramatically better math scores (71.6 vs. 64.8), partially offset by slightly worse code scores (10.9 vs. 14.9). Phi-4-Reasoning-Plus achieves only 30.6.

For code: QwQ-32B achieves 44.2, vs. 42.3 for DeepSeek-R1 — a 1.9-point improvement. The gain comes from code (29.5 vs. 27.2) and math (58.7 vs. 54.7). Phi-4-Reasoning-Plus achieves only 29.0.

For science: QwQ-32B achieves 39.1, vs. 35.9 for DeepSeek-R1 — a 3.2-point improvement. Math scores (62.1 vs. 55.9) drive the gain. Phi-4-Reasoning-Plus achieves 21.7.

Why does the weaker teacher produce better students? The paper does not provide a controlled mechanistic analysis, but the authors note several plausible explanations:

Reasoning trace quality vs. answer accuracy. DeepSeek-R1 may produce more factually correct final answers, but QwQ-32B's reasoning traces may be more pedagogically valuable — perhaps longer, more explicit about false starts and corrections, or more closely matching the reasoning patterns that a 7B model can learn to emulate. Appendix I.3's finding that shorter responses tend to be more correct suggests a tension: a teacher that is too competent may produce reasoning traces that are too compressed or "expert-blind" for a smaller student to learn from.
Distribution shift between teacher and student capabilities. A 671B model's reasoning patterns may be qualitatively different from what a 7B model can execute. QwQ-32B, being closer in scale (32B vs. 7B), may produce traces that are more "learnable" by the student. This is a form of capability matching: the optimal teacher is not the strongest model but the model whose reasoning style best matches what the student can internalize.
Overthinking and verbosity. The strong performance of response-length filtering (Section 4.3) and the finding that incorrect responses are longer than correct ones (Appendix I.3, Table 27) suggest that verbosity is a double-edged signal. QwQ-32B may produce reasoning traces that are appropriately detailed without being excessively long, while DeepSeek-R1 may overthink or self-correct excessively, producing traces that are long but not proportionally more informative.

Design choice for final pipeline. Based on these results, the authors select QwQ-32B as the teacher model for all domains. This is a critical decision because it means the final 1.2M dataset is annotated using QwQ-32B, not DeepSeek-R1, despite DeepSeek-R1 being the standard choice in most prior reasoning datasets (OpenR1, SkyT1, OpenThoughts 1 and 2).

A note on Phi-4-Reasoning-Plus. Its poor performance is partly an evaluation artifact: the model outputs code without markdown code fences (```python), which Evalchemy's code extraction fails to parse. This highlights a practical challenge in reasoning model evaluation — the model may be capable but unable to follow the output formatting expected by the evaluation harness. The authors could have post-processed the outputs to add code fences, but doing so would have been an unfair advantage for this teacher relative to the others.

Scaling the Pipeline: From 31,600 to 1.2M Examples (Section 5)

Having determined the optimal strategy at each pipeline stage through ablation at the 31,600 scale, the authors scale up to the final OpenThoughts3-1.2M dataset. This section validates that the stage-wise improvements compound and that the recipe scales effectively.

Scaling validation (Figure 3). The authors train models at dataset sizes of 316, 1K, 3.16K, 10K, and 31.6K examples, for each successive pipeline stage. The curves in Figure 3 show:

The "Question Source" curve (using only the best question source, no filtering, DeepSeek-R1 as teacher) establishes a baseline scaling trajectory.
The "Question Filter" curve (adding LLM-based filtering) shifts the entire scaling curve upward.
The "Teacher Model" curve (switching from DeepSeek-R1 to QwQ-32B) provides a further upward shift, particularly at larger scales.

The curves are roughly parallel and monotonically increasing with dataset size on log-scale, suggesting log-linear scaling: $\text{accuracy} \propto \log(\text{dataset\_size})$ within the observed range. This pattern holds most clearly for math and code; science shows more variance but still trends upward.

Why does this matter? Parallel upward-shifted scaling curves indicate that the pipeline improvements are multiplicative with scale — the gain from filtering is roughly the same at 1K as at 31.6K examples, meaning it's not merely a small-data regularization effect that washes out at scale. It also suggests that further scaling beyond 1.2M would yield additional gains, a direction the authors identify as future work.

Final dataset composition (Figure 4). The full pipeline to produce OpenThoughts3-1.2M operates as follows:

Step 1: Source questions. Collect raw questions from the winning sources:

Math: OpenMath-2-Math (>1M available)
Code: StackExchange CodeGolf (85.9K) and OpenCodeReasoning (459K)
Science: StackExchange Physics (547K) and OrganicChemistry PDFs (46.2K)

Step 2: Filter questions. Apply the winning filtering strategies:

Code: Difficulty-based selection with GPT-4o-mini, keeping hardest-rated questions
Math: Response-length selection with GPT-4.1-mini, keeping questions with longest responses
Science: Response-length selection with GPT-4.1-mini, keeping questions with longest responses

The filtering stage processes a much larger pool than the final dataset to provide enough headroom. For math, the authors increase the input to the filtering stage from 1M to 3M questions.

Step 3: Deduplicate. Apply exact deduplication to math and science questions; no deduplication for code.

Step 4: Randomly sample questions. To reach the target of 1.2M question-answer pairs with 16× sampling, the pipeline requires 1.2M / 16 = 75,000 unique questions distributed as:

Math: 53,125 questions (producing 850,000 pairs at 16×)
Code: 15,625 questions (producing 250,000 pairs)
Science: 6,250 questions (producing 100,000 pairs)

The domain ratio (850K math : 250K code : 100K science) follows the OpenThoughts2-1M mixture, which the authors found produced strong, balanced performance.

Step 5: Generate multiple answers. Query QwQ-32B 16 times per question, producing 1.2M total question-answer pairs. The authors note that annotating the full 1.2M dataset with QwQ-32B required 22,000 H100 GPU hours on 16 single-GPU GH200 nodes (Appendix O).

Step 6: Train the final model. Fine-tune Qwen-2.5-7B-Instruct on OpenThoughts3-1.2M using the "Large" hyperparameter set (Table 9): learning rate $8 \times 10^{-5}$ , batch size 512, 5 epochs, with example packing enabled. Training consumed 25,000 A100 GPU hours on 128 nodes, each with 4 A100 GPUs (512 GPUs total). The resulting model is OpenThinker3-7B.

Why this ratio and scale? The 1.2M total size was chosen to push beyond OpenThoughts2-1M while remaining computationally feasible. The 16× sampling multiplier was selected because it provided the best scalability (Section 4.4) — it allows reaching 1.2M pairs from only 75K unique questions, which is important because the filtering stage is applied to unique questions, not question-answer pairs. If the pipeline used 1× sampling, it would need 1.2M unique questions filtered from the sources, which would require processing a much larger pool (likely >10M raw questions) through LLM-based filtering — computationally expensive and potentially forcing inclusion of lower-quality questions that survive filtering.

Scaling behavior on evaluation benchmarks (Figure 8). The full scaling analysis across all 12 benchmarks shows that OpenThoughts3-1.2M dominates existing SFT reasoning datasets at every scale tested (1K, 10K, 100K, 1M) and across every benchmark. The log-linear scaling trend continues beyond the 31.6K ablation scale: on AIME24, accuracy rises from ~20% at 1K to ~69% at 1M; on GPQA Diamond, from ~30% to ~54%. Some benchmarks show signs of saturation (AMC23 reaches 93.5%, MATH500 reaches 90.0%), while others continue to improve steeply at 1M (CodeElo, CodeForces), suggesting further scaling would be beneficial.

Cross-domain transfer and the mixing decision. Figure 9 isolates scaling within each domain. The math-only curve at 250K examples achieves a math average of ~80%, while the full mixed dataset achieves lower math but higher overall average due to code and science contributions. This validates the mixing strategy: including code and science data improves cross-domain performance without catastrophically degrading math, though there is a tradeoff. The "No Pipeline" baseline — a naive union of all candidate question sources without any filtering, using DeepSeek-R1 as teacher, no multiple sampling — dramatically underperforms the full pipeline, confirming that the stage-wise optimization produces a dataset that is qualitatively better, not just larger.

Generalization to a different base model. Appendix G.1 tests whether the OpenThoughts3 recipe transfers to a different student architecture by fine-tuning Llama-3.1-8B-Instruct on a 100K subset. The results (Table 14) show that while Qwen-2.5-7B-Instruct achieves higher absolute performance (e.g., 54.3 vs. 37.0 on AIME24), Llama-3.1-8B-Instruct shows larger absolute improvements over its base model on several benchmarks (+59.4 on AMC23 vs. +33.3 for Qwen). This suggests the dataset recipe generalizes across model families, though the absolute ceiling depends on the base model's initial capabilities.

Summary of Design Choices and Their Justifications

31,600-example ablation scale: chosen as the log-scale midpoint (\sqrt{10} \approx 3.16) between 10K and 100K, balancing experimental cost against statistical signal. The scaling curves (Figure 3) validate that this scale produces rankings consistent with larger scales.
Greedy sequential optimization: each pipeline stage is optimized independently, with the winner locked in for subsequent stages. This is a practical necessity given the combinatorial explosion of joint optimization, and the compounding gains in Figure 3 suggest destructive interactions between stages are limited.
Qwen-2.5-7B-Instruct as student: a widely-used, strong base model that leaves significant headroom for reasoning improvement (24.0 average base score vs. 55.3 final). Using a too-weak base model would produce noisy ablation signals; using a too-strong base model would leave insufficient room for dataset quality differences to manifest.
8-benchmark evaluation suite with held-out sets: provides domain coverage (math, code, science) while reserving 4 held-out benchmarks (AIME25, HMMT, LCB 06/24-01/25, HLE) for final validation, preventing overfitting the pipeline decisions to specific evaluation sets.
Average score as selection criterion rather than per-domain optimization: simplifies decision-making and assumes cross-domain transfer (e.g., math data helps science). This is a practical choice, not a claim that domain-specific optimization would underperform.
No answer filtering: the most surprising pipeline decision, justified by the consistent null result across all filtering strategies and all three domains. The mechanism is likely that teacher responses, even when incorrect, contain pedagogically valuable partial reasoning, and filtering methods introduce their own errors.
QwQ-32B over DeepSeek-R1 as teacher: justified by consistent 2–3 point improvements across all domains despite QwQ-32B's lower benchmark scores. The likely mechanism involves better capability matching between a 32B teacher and a 7B student, producing more learnable reasoning traces.
16× repeated sampling: enables dataset scaling without requiring more unique questions, avoiding the quality dilution that comes from including lower-ranked question sources. This is a key architectural insight: for reasoning SFT, answer diversity dominates question diversity.
Exact deduplication for math and science, none for code: domain-specific choice based on ablation results. Math and science questions are more likely to appear in near-identical form across sources (same competition problems, same textbook exercises); code problems are more naturally unique due to specification details.
LLM-based filtering over classical methods: fastText and embedding filters are cheaper but capture surface-level similarity; LLM-based filters assess internal question quality (difficulty, coherence, answerability) that n-gram overlap misses. The cost difference is acceptable at pipeline construction time (one-time cost) even if it would be prohibitive at pretraining scale.

4. Key Insights and Innovations

Innovation 1: The Systematic Recipe as a Scientific Contribution in Itself

The paper's most distinctive intellectual move is its framing: the process of discovering a data recipe is treated as a first-class research contribution, not merely a means to an end. Prior work on reasoning SFT datasets—SkyT1 (NovaSky-Team, 2025b), OpenR1 (Face, 2025), LIMO (Ye et al., 2025)—released artifacts (datasets and models) but treated the pipeline design as an implementation detail. The dominant assumption was that a good dataset is the output of a good pipeline, and the community's role is to use or improve the artifact. This paper argues the opposite: the methodology for discovering what works is the enduring contribution, and the specific recipe is a snapshot of that methodology applied at a particular scale.

What makes this framing novel is that it transforms pipeline design from a craft guided by intuition and precedent into an empirical science. Each of the 1,000+ experiments in Section 4 isolates exactly one pipeline variable, holding dataset size constant at 31,600 examples, training the same student model (Qwen-2.5-7B-Instruct), and evaluating on the same 8-benchmark suite. This design enables causal attribution: when response-length filtering improves math scores by 6.7 points over random filtering (Table 5), the reader knows the gain is attributable to that filtering choice and not to concurrently varying question sources or teacher models. No prior reasoning dataset project provides this level of controlled evidence for its design decisions.

The significance extends beyond the specific recipe. By releasing all experimental artifacts—the ablation results, the intermediate models, the curation code—the paper establishes a template for how the open-source community can systematically explore the reasoning data design space. Future researchers can swap in new question sources, new teacher models, or new filtering strategies into the same experimental harness and contribute evidence rather than opinion. This is a fundamental shift from the prior norm of "we built a good dataset, here it is" to "here is a methodology for building good datasets, along with the evidence it generates."

The compounding scaling curves in Figure 3 provide the key evidence for this framing. Each stage of the pipeline shifts the entire scaling curve upward, and the gains are roughly multiplicative with scale. This means the ablation results at 31,600 examples—a cost-effective experimental scale—generalize to much larger datasets, validating the methodology itself as reliable. If the stage-wise improvements had been local optima that washed out at scale, the entire approach would be called into question. The fact that they compound is what elevates this from a one-off engineering success to a reproducible scientific methodology.

Innovation 2: Answer Diversity as the Dominant Scaling Axis, Not Question Diversity

The paper's most counterintuitive empirical finding reorganizes how the field should think about data scaling for reasoning SFT. The standard intuition—inherited from both pretraining (where data diversity is a primary driver of generalization) and instruction tuning (where task diversity is critical)—is that more diverse training examples produce better models. In the reasoning SFT context, this intuition manifests as the practice of aggregating questions from many sources to maximize problem-type coverage.

The paper systematically tests and refutes this intuition across two independent experiments. First, the question mixing ablation (Section 4.2, Table 4) shows that mixing the top 2 code question sources achieves a 41.3 average score, while mixing the top 16 achieves 36.4—a 5-point degradation that can only be attributed to quality dilution from lower-ranked sources. Second, the repeated sampling experiment (Section 4.4, Table 6) shows that annotating fewer unique questions 16 times produces performance comparable to or better than annotating more unique questions once—the 16× sampling variants achieve average scores of 40.6–41.3 for code vs. 39.8 for 1× sampling, a pattern that holds across all three domains.

These results jointly establish a new principle: for reasoning SFT, answer diversity is more valuable than question diversity. Seeing 16 different solution paths to the same hard problem teaches the student model about the structure of the solution space—how different approaches converge on the same answer, where common errors occur, what intermediate reasoning steps are robust. This is a fundamentally different kind of learning signal than seeing 16 different problems each solved once. The prior implicit assumption—that each new question contributes roughly equal marginal value—turns out to be quantitatively wrong when question quality varies across sources and when answer diversity for high-quality questions is under-exploited.

This finding is significant beyond the specific dataset because it reframes the resource allocation question for any reasoning dataset project. If answer diversity dominates, the optimal strategy is not "find more unique high-quality questions" (which becomes increasingly expensive as you exhaust the best sources) but "find a core set of excellent questions and invest heavily in answer diversity for those questions." The 16× sampling multiplier used in OpenThoughts3-1.2M achieves a 1.2M dataset from only 75K unique questions, avoiding the quality dilution that would come from including questions from sources ranked 3–16. For practitioners, this insight directly translates to a computational budget allocation: spend more teacher inference compute on generating multiple responses to high-quality questions rather than on sourcing and filtering a larger number of unique questions.

Innovation 3: Weaker Teachers as Stronger Distillation Sources

The paper provides a clear empirical demonstration that challenges a foundational assumption in knowledge distillation: that a stronger teacher produces a stronger student. DeepSeek-R1 is unequivocally the stronger model on the evaluation benchmarks, achieving a 65.3 overall average vs. QwQ-32B's 64.2 (Table 29), with particularly large advantages on CodeElo (53.7 vs. 44.3), GPQA Diamond (73.7 vs. 65.0), and JEEBench (92.3 vs. 69.9). Yet across all three domains, QwQ-32B produces training data that yields better student models—by 2.6 points for math, 1.9 points for code, and 3.2 points for science (Table 8).

What makes this finding intellectually distinctive is not merely the observation (which has precedents in the distillation literature), but the diagnostic implication it carries: the quality of a teacher's reasoning traces for pedagogical purposes is not measured by the teacher's final-answer accuracy. This is a conceptual shift. Prior work selecting teacher models implicitly assumed that answer correctness was the relevant quality signal—if model A gets more answers right than model B, model A's reasoning traces are better training data. The paper's evidence shows this assumption is false, and not by a small margin.

The likely mechanism—which the paper identifies but does not mechanistically validate—is capability matching: a 32B model's reasoning patterns may be more structurally similar to what a 7B model can learn to reproduce than a 671B model's reasoning patterns, even if the 671B model's patterns lead to more correct answers. This connects to a broader principle in machine teaching: the optimal teacher is not the most knowledgeable entity but the entity whose knowledge representation best aligns with the learner's capacity. A Nobel laureate is not necessarily the best calculus teacher—someone who learned calculus more recently may better understand where students struggle and how to explain concepts at the appropriate level of abstraction.

The practical significance is immediate: selecting a teacher model for reasoning distillation should not default to the strongest available model. A systematic evaluation of candidate teachers—generating training data from each and measuring downstream student performance—is necessary because the ranking of teachers by their own benchmark scores does not predict the ranking by their effectiveness as distillation sources. The paper also provides a suggestive boundary condition: Phi-4-Reasoning-Plus, the weakest teacher tested, produces dramatically worse students (30.6 average for math, 29.0 for code), indicating a lower bound below which teacher quality degradation overwhelms any capability-matching benefit. The optimum lies somewhere between "strongest" and "too weak," and the paper's contribution is demonstrating that this optimum is not at the "strongest" endpoint.

Innovation 4: The Null Result on Answer Verification as a Diagnostic of Pipeline Quality

Perhaps the paper's most surprising empirical contribution is what doesn't work. Across all three domains and across all tested filtering strategies—LLM-based verification (GPT-4o-mini judging answer correctness), majority consensus filtering, fastText quality classification, length-based heuristics, and language/format checks—no answer filtering strategy meaningfully outperforms training on all available data (Section 4.5, Tables 7 and 44–46). The best filtering method for math (random filtering) achieves 41.6 vs. 41.9 for no filtering; for code, fastText filtering achieves 42.3 vs. 42.2; for science, Longest Answers achieves 37.5 vs. 38.3. These differences are within experimental noise.

This null result is intellectually significant because it challenges a deeply held intuition that the authors themselves previously operationalized in OpenThoughts-114K and OpenThoughts2, both of which employed verification. The intuition is straightforward: training on incorrect answers should degrade performance, so removing them should help. The evidence says this intuition is wrong under the conditions of this pipeline—specifically, when questions have already been filtered for quality via LLM-based methods (Section 4.3) and when the teacher model (DeepSeek-R1 or QwQ-32B in these ablations) is sufficiently capable.

The implication is that teacher responses contain instructional value beyond their final-answer correctness. An incorrect solution that demonstrates a valid approach, encounters a subtle error, and pivots to an alternative strategy may teach the student more about the problem-solving process than a flawless but compressed correct solution. The verification filter removes these "productive failure" examples, and whatever benefit comes from eliminating genuinely misleading responses is offset by the loss of pedagogically valuable incorrect ones. Additionally, the verification methods themselves introduce noise—GPT-4o-mini may incorrectly classify correct answers as wrong or vice versa—and this noise may further erode any potential benefit.

This finding is best understood as a diagnostic of the rest of the pipeline. The fact that answer filtering doesn't help means the earlier stages—question sourcing from top-ranked sources, LLM-based question filtering for difficulty/length—are already selecting questions for which teacher responses are sufficiently high-quality that post-hoc verification adds no value. If the question pool included many trivial or ill-posed questions, answer verification might provide a stronger signal. The null result thus validates the effectiveness of the earlier pipeline stages: they have already removed the kinds of questions whose teacher responses would be harmful to train on. For the field, this provides a concrete threshold: if your question filtering is good enough, you can skip answer verification entirely, simplifying the pipeline and saving the compute cost of verification.

Innovation 5: LLM-Based Question Filtering as a Superior Quality Signal to Classical Methods

The paper provides a clear empirical ranking of filtering methodologies that has implications beyond reasoning datasets. In the question filtering experiments (Section 4.3, Table 5), LLM-based methods—difficulty scoring with GPT-4o-mini, response-length selection with GPT-4.1 models, and AskLLM similarity scoring—consistently outperform classical methods (fastText classifiers and embedding-based similarity). For code, the best LLM method (difficulty-based) achieves 43.0 vs. 39.7 for random selection and 36.9 for embedding-based selection. For math, the gap is 41.9 (LLM length-based) vs. 35.2 (random) vs. 32.6 (fastText). For science, it's 35.9 (LLM length-based) vs. 33.8 (random) vs. 31.5 (fastText).

What makes this conceptually interesting is why the gap exists. Classical filtering methods capture surface-level similarity: a fastText classifier trained on CodeForces vs. SQL questions can distinguish competitive programming from database queries, but cannot distinguish a well-posed competitive programming problem from a poorly-posed one. LLM-based methods capture a different signal: they assess the question's internal quality—its difficulty, coherence, answerability, and instructional value—by engaging with the question as a problem-solver would. When GPT-4o-mini rates a question's difficulty, it is simulating the cognitive challenge the question would pose. When GPT-4.1-mini produces a long response, it is demonstrating that the question elicits sustained reasoning. These are behavioral measures of question quality, not merely statistical measures of surface similarity to known-good examples.

This finding parallels and extends work in pretraining data curation (e.g., Wettig et al., 2025; Shum et al., 2025) that uses LLMs to assess data quality, but applies it in the more demanding context of reasoning SFT where the quality signal must capture logical coherence and appropriate difficulty rather than merely factual accuracy or fluency. It also provides a practical guideline: for reasoning dataset construction, the cost of LLM-based filtering (which is incurred once at pipeline construction time) is justified by the downstream performance gains, even though it would be too expensive for pretraining-scale data curation. The strong performance of response-length filtering with GPT-4.1-mini—a model that did not exist when most prior reasoning datasets were constructed—also demonstrates that pipeline designs should be re-evaluated as stronger and cheaper LLMs become available for curation tasks.

5. Experimental Analysis

Evaluation Methodology

Dataset. All pipeline experiments use benchmark evaluation sets, not a training corpus. The primary evaluation suite consists of eight benchmarks across three domains: math (AIME24 with 30 questions, AMC23 with 40 questions, MATH500 with 500 questions), code (CodeElo with 391 problems, CodeForces with 453 problems, LiveCodeBench 05/23-05/24 with 511 problems), and science (GPQA Diamond with 198 questions, JEEBench with 515 questions). Four additional benchmarks are held out until all pipeline decisions are finalized: AIME 2025 (30 questions), HMMT 02/25 (30 questions), LiveCodeBench 06/24-01/25 (369 problems), and Humanity's Last Exam MCQs (512 questions). These benchmarks were chosen because they require multi-step reasoning with verifiable answers — exact integer matching for math competitions, unit test execution for code, and multiple-choice selection for science. The authors apply rigorous decontamination using both normalized InDel similarity at a 75% threshold and 13-gram overlap against the Qwen2-7B-Instruct tokenizer; any training sample flagged by either criterion is discarded. Tuning on a manually contaminated testbed of ~3,000 samples achieved a 99.6% true negative rate while falsely flagging only 1.4% of clean samples (Appendix F, Figure 7).
Base model(s). All pipeline ablation experiments fine-tune Qwen-2.5-7B-Instruct (Qwen2.5-Team, 2024), a 7-billion parameter instruction-tuned model. This model was chosen because it provides a strong but unsaturated baseline: it achieves 24.0 average score across the evaluation benchmarks prior to any reasoning SFT (Table 1, "Base Model" row), representing a substantial gap from the ~55 point ceiling achieved by the best models that leaves room for dataset quality differences to manifest. Appendix G.1 additionally tests Llama-3.1-8B-Instruct to validate that the dataset recipe generalizes across model families; results (Table 14) show the recipe transfers — Llama achieves larger absolute improvements over its base on several benchmarks (+59.4 on AMC23 vs. base, compared to Qwen's +33.3) — though Qwen's higher starting point yields better absolute scores (e.g., AIME24 54.3 vs. 37.0). For the final FLOPs-matched comparison, the pretraining-scaled baseline is a model with approximately 14× more parameters (exact model not specified in the text, but implied to be a larger PaLM 2 variant).
Metrics. The primary metric throughout is average accuracy across the eight in-distribution benchmarks, computed as the unweighted mean of per-benchmark accuracies. Each benchmark uses a different scoring protocol: AIME24, AIME25, AMC23, and HMMT report mean accuracy with standard error over 10 iterations; LiveCodeBench uses 6 iterations; CodeElo, CodeForces, GPQA Diamond, JEEBench, and HLE use 3 iterations; MATH500 is evaluated in a single pass over all 500 problems. Standard errors are reported as subscripts in all tables (e.g., 41.9(0.3)). All evaluations use the Evalchemy framework (Raoof et al., 2025) with unified generation hyperparameters: temperature 0.7, top_p = 1.0, max_new_tokens = 32,768. Evalchemy partitions each benchmark into independent shards, runs them in parallel across GPUs, and streams per-shard metrics back to a central coordinator for real-time aggregation. The authors cache full model completions for long chain-of-thought tasks to reduce redundant inference and enable failure case analysis. Final answers are extracted from the model's output using the DeepSeek-R1 chat template: reasoning traces appear within thinking and response tokens, with the final answer following the response block. For code, answers are expected in markdown-fenced code blocks with a language tag.
Baselines. The paper compares against multiple existing SFT reasoning datasets and models:
- Existing datasets at matched scale (Figure 1 and Figure 8): OpenThoughts3-1.2M is compared to AM (1.4M examples, Amazon-Nova), Nemotron Nano (1M examples, Nvidia), s1.1 (~1K examples, Muennighoff et al., 2025), and LIMO (~1K examples, Ye et al., 2025). All models are fine-tuned from Qwen-2.5-7B-Instruct using identical hyperparameters, isolating the effect of dataset quality.
- Existing models at matched model scale (Table 1): OpenThinker3-7B is compared to DeepSeek-R1-Distill-Qwen-7B (800K examples, proprietary distillation), Nemotron-Nano-8B (3.9M examples, SFT + RL), AceReason-7B (57K examples, RL only), Skywork-7B (119K examples, RL only), OpenR1-Distill-7B (350K examples, SFT only), and the base Qwen-2.5-7B-Instruct. This comparison spans different optimization algorithms (SFT, RL, combined) and dataset scales.
- Within the pipeline experiments, baselines include: random filtering at each filtering stage; the "No Pipeline" baseline (Section 5) constructed by taking the union of 31.6K samples from all candidate question sources without any filtering, using DeepSeek-R1 as teacher, and no multiple sampling; and the Qwen-2.5-7B-Instruct base model without any reasoning SFT.
Generation budget / compute accounting. The unit of data scale is number of training examples (question-reasoning trace-answer triplets). All pipeline ablation experiments use a fixed budget of 31,600 examples, chosen as the log-scale midpoint between 10K and 100K (√10 ≈ 3.16, so 10,000 × 3.16 = 31,600). This scale was selected to be cost-effective while providing meaningful signal. Training compute is measured in GPU hours: a single 31,600-example run requires approximately 25,000 A100 GPU hours on 512 GPUs (Section 4, Training paragraph). The final OpenThoughts3-1.2M training consumed 22,000 H100 GPU hours for annotation with QwQ-32B (across 16 GH200 nodes) and 25,000 A100 GPU hours for student training (across 128 nodes, each with 4 A100 GPUs). For teacher model comparisons, sampling hyperparameters are kept constant across all teachers. For answer filtering experiments, to ensure filtered datasets still reach 31,600 examples, the authors first generate 63,200 pairs, apply each filter, then sample 31,600 from survivors — the "No Filtering" baseline uses all 63,200 pairs (not compute-controlled), while "Random Filtering" provides a fair compute-controlled baseline.
Cross-validation / statistical protocol. The pipeline optimization uses a greedy sequential design: at each stage, all candidate strategies are ablated independently (while holding other stages at previously-selected best settings), the strategy with the highest average benchmark score is selected, and the pipeline proceeds to the next stage. This is not cross-validated within the 8-benchmark suite — the same evaluation questions are used for strategy selection across all stages. However, four held-out benchmarks (AIME25, HMMT 02/25, LiveCodeBench 06/24-01/25, HLE MCQ) are reserved exclusively for final evaluation after all pipeline decisions are frozen. The fact that OpenThinker3-7B achieves the best held-out scores across all four benchmarks (Table 1) provides evidence against overfitting the pipeline to the evaluation sets. Standard errors are reported throughout using repeated evaluation runs (3–10 iterations per benchmark, depending on the benchmark), enabling statistical comparison: values within two standard errors of the highest-scoring strategy are bolded in tables. For hyperparameter selection at different data scales, the authors conducted sweeps to find appropriate learning rates, batch sizes, and epoch counts (Table 9). The base model generalization experiment (Appendix G.1) serves as an additional robustness check, testing whether the recipe transfers to a different model family (Llama-3.1-8B-Instruct).

Main Quantitative Results

Question Sourcing: Question Quality Dominates Pipeline Performance

The first pipeline stage evaluates which question sources produce the best training data when all questions are annotated by DeepSeek-R1 and trained into a student model at the 31,600-example scale. The headline result (Tables 3, 32–34) is that question source quality alone drives a 17.2-point gap between the best and worst code sources, making it the single largest lever in the entire pipeline.

For code (Table 32), the top performers are StackExchange CodeGolf (38.8 average), OpenCodeReasoning (38.4), and KodCode-V1 (37.7). The worst performer, bugdaryan/sql-create-context-instruction, achieves only 21.6 — a 17.2-point gap. Even among the top tier, the gap between 1st (38.8) and 10th (35.0 for glaiveai/glaive-code-assistant-v3) is 3.8 points. This indicates substantial variance even among seemingly plausible code question sources, and suggests that source selection is not merely about choosing a domain but about identifying specific datasets with appropriate difficulty, well-posedness, and reasoning structure.

For math (Table 33), the top sources are OpenMath-2-Math (38.1), NuminaMath-1.5 (37.4), and a synthetic variant the authors created by applying the OpenMathInstruct pipeline to AIME sections (37.2). The gap between 1st and last (Lap1official/Math at 24.4) is 13.7 points. Notably, three of the top five math sources are synthetic or semi-synthetic (OpenMath-2-Math, OpenMathInstruct-AIME, MathPile), while NuminaMath (human-written) also performs well — there is no clear provenance-based pattern.

For science (Table 34), StackExchange Physics (34.3), the Organic Chemistry PDF Pipeline (34.0), and CQADupStack-Physics (33.3) form a tight top tier. The gap to the bottom (AdapterOcean/biology_dataset at 21.9) is 12.4 points. The Organic Chemistry PDF Pipeline — a multi-stage extraction involving Gemini OCR followed by GPT-4o-mini refinement — performs comparably to human-written StackExchange questions, demonstrating that sophisticated synthetic extraction can match human-curated quality.

A critical auxiliary finding: no clear pattern favors synthetic over non-synthetic sources. StackExchange CodeGolf (human-written) outscores KodCode-V1 (fully synthetic) for code; OpenMath-2-Math (semi-synthetic) outscores NuminaMath-1.5 (human-written) for math; and StackExchange Physics (human-written) ties Organic Chemistry PDFs (semi-synthetic extraction) for science. The relevant quality dimension is not provenance but some latent property — likely a combination of appropriate difficulty, well-posedness, internal consistency, and reasoning depth — that cross-cuts the human/synthetic distinction.

Design choice for final pipeline: OpenMath-2-Math alone for math; CodeGolf + OpenCodeReasoning for code; StackExchange Physics + OrganicChemistry-PDFs for science.

Question Mixing: Fewer High-Quality Sources Outperform Diverse Collections

The mixing experiment (Section 4.2) tests a widely-held intuition: that combining diverse question sources improves generalization. The authors sweep from 1 to 16 top-ranked sources per domain, sampling 31,600/N questions from each.

The results (Table 4 and Tables 35–37) consistently refute the diversity hypothesis. For code (Table 4): Top 2 sources achieve 41.3 average score, while Top 16 sources achieve 36.4 — a 4.9-point degradation. The monotonic pattern is clear: Top 1 (39.9) < Top 2 (41.3) > Top 4 (38.6) > Top 8 (37.0) > Top 16 (36.4). The peak at 2 sources and subsequent decline indicates that there is some benefit to including a second high-quality source (perhaps covering different problem types), but beyond that, quality dilution from lower-ranked sources dominates any diversity benefit.

For math (Table 36): Top 1 source (37.6) outperforms all mixtures, with Top 16 achieving 33.8 — a 3.8-point degradation. Unlike code, there is no benefit from a second source; a single top source is optimal.

For science (Table 37): Top 2 sources (33.7) slightly outperform Top 1 (33.6), but Top 16 drops to 30.8 — a 2.9-point degradation.

The key mechanism is quality dilution, not diversity: lower-ranked question sources (identified as such in Section 4.1) introduce questions with worse properties — ambiguous phrasing, flawed problem structures, inappropriate difficulty levels — that confuse the student model during training. The negative impact of these bad examples outweighs any benefit from seeing a wider variety of problem types within a domain. This directly challenges the practice in prior reasoning datasets (OpenR1, OpenThoughts2, Amazon-Nova) that aggregate questions from many sources.

Design choice for final pipeline: 1 source for math (OpenMath-2-Math), 2 sources for code (CodeGolf + OpenCodeReasoning), 2 sources for science (StackExchange Physics + OrganicChemistry-PDFs).

Question Filtering: LLM-Based Methods Outperform Classical Filters

The filtering stage (Section 4.3) tests methods for selecting high-quality subsets from the chosen sources. The absolute gains are substantial: for math, the best filter improves over random selection by 6.7 points (41.9 vs. 35.2, Table 39); for code, by 3.3 points (43.0 vs. 39.7, Table 38); for science, by 2.1 points (35.9 vs. 33.8, Table 40). These gains, while smaller than the source selection effect (17.2 points), are still practically meaningful.

The consistent pattern across all three domains is that LLM-based filtering outperforms classical methods (fastText, embeddings). For code (Table 38): difficulty-based selection with GPT-4o-mini leads (43.0), with response-length filtering via GPT-4.1-nano close behind (42.2). The best fastText filter (positives: CodeForces; negatives: CodeReview) achieves 40.5, a full 2.5 points behind the LLM winner. Embedding-based selection performs worst at 36.9.

For math (Table 39): response-length filtering with GPT-4.1-mini wins decisively at 41.9, with the next-best LLM method (AskLLM) at 36.3. The best fastText filter (positives: Numina; negatives: Lap1official) achieves 35.6, indistinguishable from random (35.2). A striking anomaly: response-length filtering with GPT-4o-mini collapses to 6.8 average — essentially random performance — suggesting that GPT-4o-mini's responses to math questions are so unreliable that response length becomes an anti-signal.

For science (Table 40): response-length filtering with GPT-4.1-mini leads (35.9) while fastText variants cluster between 31.5 and 35.1. The margin is smaller than for math or code, likely because science question quality variance is lower given the already-narrow top-tier sources.

Why LLM-based filtering wins. fastText and embedding-based methods capture surface-level similarity to known-good questions. An n-gram model can distinguish competitive programming from SQL but cannot assess whether a given competitive programming problem is well-posed or appropriately difficult. LLM-based methods — difficulty scoring via a detailed rubric (Figures 27, 29, 30), response-length as a behavioral measure of question complexity, AskLLM similarity scoring — engage with the question's internal quality. They are more expensive at curation time but produce better training data.

A positive feedback loop with stronger filtering models. For response-length filtering, stronger models produce better quality signals: GPT-4.1-mini > GPT-4.1-nano > GPT-4o-mini for both math and science. This suggests that as more capable and cheaper LLMs become available, the quality of LLM-based filtering will continue to improve, further widening the gap over classical methods.

Design choice for final pipeline: difficulty-based filtering with GPT-4o-mini for code; response-length filtering with GPT-4.1-mini for math and science.

Deduplication and Repeated Teacher Sampling: Answer Diversity > Question Diversity

The deduplication and sampling experiment (Section 4.4) tests 9 combinations (3 deduplication levels × 3 sampling multipliers) at the 31,600 scale. The headline result (Tables 6, 41–43) is that multiple sampling per question compensates for having fewer unique questions, and deduplication provides at most a marginal benefit when starting from high-quality filtered sources.

For code (Table 41): the highest-scoring combinations are Exact Dedup + 4× sampling (41.3), No Dedup + 16× sampling (41.3), and No Dedup + 4× sampling (41.2). The lowest is No Dedup + 1× sampling (39.8), a gap of only 1.5 points across all 9 combinations. The takeaway is that for code, the choice of deduplication and sampling ratio barely matters within reasonable ranges — the earlier stages (source selection, filtering) have already determined the quality ceiling.

For math (Table 42): Exact Dedup + 1× sampling achieves the highest score (41.7), Exact Dedup + 16× is second (40.1), and the no-deduplication variants are lower (36.5–38.3). The pattern suggests that for math, deduplication provides a small but consistent benefit (about 2–5 points), likely because math competition problems are more frequently duplicated across sources with minor rewording — removing these duplicates prevents wasted teacher compute and ensures answer diversity is genuine rather than redundant.

For science (Table 6, main text): Exact Dedup + 16× sampling leads (36.2), Fuzzy Dedup + 16× is second (36.1), and the 1× sampling variants cluster lower (34.2–35.5). The spread is ~2 points.

Why multiple sampling is the key practical insight. The 16× multiplier means 1.2M question-answer pairs can be produced from only 75K unique questions. This avoids the quality dilution from including lower-ranked sources (Section 4.2) while still achieving the scale needed for strong performance (Figure 8 shows log-linear improvements up to 1M). The finding is that seeing 16 different reasoning paths for the same hard problem teaches the student more than seeing 16 different hard problems each solved once — answer diversity dominates question diversity for reasoning SFT.

Design choice for final pipeline: Exact deduplication + 16× sampling for math and science; No deduplication + 16× sampling for code (the second-best strategy for code, chosen over the tied first-place Exact Dedup + 4× because 16× provides better scalability for the final 1.2M dataset).

Answer Filtering: The Universal Null Result

The answer filtering experiment (Section 4.5) tests whether removing low-quality teacher responses improves the student. This is the most counterintuitive result in the paper: no answer filtering strategy outperforms training on all available data across any domain.

For math (Table 45): No Filtering achieves 41.9 (with 63,200 examples — an unfair comparison since it's not compute-controlled). The best fair comparison is Random Filtering at 41.6 — a 0.3-point difference within standard error. Majority Consensus achieves 41.0, GPT Verification achieves 40.0, Removing Long Paragraphs degrades to 38.0. The inference is clear: removing answers identified as low-quality by any automated method either removes pedagogically useful training signal or introduces filtering errors, and neither effect is offset by removing genuinely harmful examples.

For code (Table 44): fastText Selection achieves a statistically insignificant 42.3 vs. 42.2 for No Filtering. Shortest Answers achieves 42.0. GPT Verification achieves 40.7. All strategies cluster within ~2 points.

For science (Table 46): No Filtering achieves 38.3, Longest Answers achieves 37.5, Majority Consensus achieves 35.7. The spread is ~2.6 points.

Why does verification fail? The paper's implicit argument — derived from this consistent null result combined with the earlier pipeline findings — is that when questions are already high-quality (selected from top sources and filtered via LLM-based methods), the teacher model's responses, even when incorrect, contain valuable partial reasoning. An incorrect solution that demonstrates valid reasoning up to a subtle error point may teach the student more about problem-solving than a compressed correct solution that skips steps. Verification methods also introduce noise: GPT-4o-mini may incorrectly classify correct answers as wrong (false positives), and majority consensus may discard correct but unconventional solutions. The net effect is that whatever benefit comes from removing genuinely misleading responses is offset by the loss of pedagogically valuable "productive failure" examples and the introduction of filtering errors.

Boundary condition. This result is specific to the regime where questions have already been filtered for quality. If the question pool included many trivial or poorly-posed questions, teacher responses might be genuinely harmful and verification would provide a stronger signal. The null result is thus partly a validation of the upstream pipeline: it has already removed the kinds of questions for which teacher responses would be bad enough to need filtering.

Design choice for final pipeline: Skip answer filtering entirely.

Teacher Model Selection: The Weaker Teacher Paradox

The teacher model experiment (Section 4.6) compares DeepSeek-R1, QwQ-32B, and Phi-4-Reasoning-Plus as distillation sources. The headline result (Tables 8, 47–49) is that QwQ-32B consistently outperforms DeepSeek-R1 as a teacher despite being empirically weaker on the evaluation benchmarks.

For math (Table 48): QwQ-32B as teacher achieves 44.2 average vs. DeepSeek-R1's 40.6 — a 3.6-point improvement. The gain is driven by the math sub-score (71.6 vs. 62.5), offset slightly by worse code (10.9 vs. 13.3). Phi-4-Reasoning-Plus achieves only 30.6, confirming it is too weak to serve as an effective teacher.

For code (Table 47): QwQ-32B achieves 44.2 vs. DeepSeek-R1's 38.0 (note: this DeepSeek-R1 baseline differs from earlier stages because the question sources and filtering are now fixed at the pipeline-optimal settings, and the teacher is the only variable). The gain is a 6.2-point improvement, driven by both code (29.5 vs. 19.2) and math (58.7 vs. 54.3). Phi-4-Reasoning-Plus achieves 29.0, with a catastrophic code score of 0.5 (attributable to its failure to produce parseable code tags).

For science (Table 49): QwQ-32B achieves 39.1 vs. DeepSeek-R1's 35.9 — a 3.2-point improvement. Again, the gain is math-driven (62.1 vs. 55.9), with science roughly tied (48.0 vs. 49.0).

Table 29 contextualizes the paradox: DeepSeek-R1 achieves 65.3 average on the benchmarks vs. QwQ-32B's 64.2. DeepSeek-R1 is better or equal on 8 of 12 benchmarks, with particularly large leads on CodeElo (53.7 vs. 44.3), GPQA Diamond (73.7 vs. 65.0), and JEEBench (92.3 vs. 69.9). Yet QwQ-32B is the better teacher across all three domains. The likely mechanism is capability matching: a 32B model's reasoning patterns — the length, structure, error-recovery behavior, level of abstraction — are more learnable by a 7B student than a 671B model's patterns, even if the 671B model arrives at more correct answers. This is reinforced by the finding in Appendix I.3 that incorrect responses tend to be significantly longer than correct ones (e.g., for DeepSeek-R1 on AIME24: incorrect responses average 18,198 tokens vs. 7,817 for correct ones; Table 27), suggesting that stronger models may produce overly compressed reasoning that is harder to learn from.

Design choice for final pipeline: QwQ-32B as teacher for all domains.

Scaling: Compounding Gains Across Pipeline Stages

Figure 3 validates that pipeline improvements compound across scales. At each successive stage, the scaling curve shifts upward in a roughly parallel fashion, indicating multiplicative gains. At 31.6K examples: the "Question Source" baseline (best source, DeepSeek-R1 teacher, no filtering) achieves approximately the lowest curve; adding "Question Filter" shifts the curve up; adding "Question Mix" provides a further shift; switching to "Teacher Model" (QwQ-32B) provides the final upward shift. The curves are approximately log-linear, suggesting that accuracy ∝ log(dataset_size) within the observed range.

Figure 8 provides the full scaling analysis on all 12 benchmarks, comparing OpenThoughts3 against existing SFT datasets (Nemotron Nano, AM, s1.1, LIMO) at matched data scales from 1K to 1M examples. OpenThoughts3 dominates at every scale: on AIME24, OpenThoughts3 at 1K (~20%) already exceeds AM at 1K (~5%) and LIMO at 1K (~10%); at 1M, OpenThoughts3 reaches ~69% vs. AM's ~29% and Nemotron Nano's ~55%. On GPQA Diamond, the pattern is similar: ~54% at 1M vs. ~48% for Nemotron Nano and ~38% for AM. On LiveCodeBench 06/24-01/25, OpenThoughts3 achieves ~52% at 1M vs. ~42% for Nemotron Nano and ~40% for AM. The log-linear trends continue upward at 1M for most benchmarks, suggesting further scaling would yield additional gains.

Figure 9 isolates scaling within each domain and confirms that the full pipeline recipe ("OpenThoughts3") dramatically outperforms the "No Pipeline" baseline (naive union of all candidate sources, DeepSeek-R1 teacher, no filtering, no multiple sampling) by increasing margins as scale increases. For math: at 31.6K, the gap is ~15 points; at 250K, the gap grows to ~20 points. For code: at 31.6K, the gap is ~5 points; at 250K, ~8 points. For science: at 10K, the gap is ~2 points; at 100K, ~6 points. The widening gap indicates that the pipeline improvements are not merely a small-data effect but compound with scale.

Final model performance (Table 1). OpenThinker3-7B, trained on the full 1.2M dataset, achieves a 55.3 average across all 12 benchmarks, establishing it as the state-of-the-art open-data 7B model regardless of optimization algorithm. This compares to: 53.2 for Nemotron-Nano-8B (SFT + RL, 3.9M examples), 52.9 for AceReason-7B (RL only, 57K examples), 51.6 for Skywork-7B (RL only, 119K examples), 47.2 for OpenR1-Distill-7B (SFT only, 350K examples), and 42.9 for DeepSeek-R1-Distill-Qwen-7B (SFT only, 800K examples). On held-out benchmarks: 53.3 on AIME25 (vs. 48.0 for Nemotron-Nano, 38.0 for R1-Distill-7B); 51.7 on LiveCodeBench 06/24-01/25 (vs. 50.9 for Nemotron-Nano, 34.5 for R1-Distill-7B); 42.7 on HMMT 02/25 (vs. 26.7 for Nemotron-Nano, 25.0 for R1-Distill-7B); 10.2 on HLE MCQs (vs. 12.4 for R1-Distill-7B — the only benchmark where OpenThinker3-7B is not best, potentially reflecting the difficulty of HLE questions that exceed the teacher model's capability range).

Ablation Studies and Robustness Checks

Chat template choice has negligible impact (Appendix D.4, Table 11): Switching from the SkyT1 template (<|begin_of_thought|>, <|end_of_thought|>) to the R1 template ( think, response) produces near-identical results. OpenThinker-7B with the R1 template achieves 32.7% on AIME24 vs. 31.3% with the SkyT1 template. The choice of token delimiters matters less than their presence — what matters is that the model learns to separate thinking from final answer.

System prompt has task-dependent effects but reasoning can be implicitly learned (Appendix D.5, Table 12): Testing with Llama-3.1-Nemotron-Nano-8B shows that explicitly enabling reasoning dramatically helps on math (AIME25: 45.3% reasoning-on vs. 2.0% reasoning-off), but using no system prompt at all sometimes outperforms explicit "reasoning on" (AIME24: 70.0% no prompt vs. 61.3% reasoning-on; GPQA Diamond: 55.9% vs. 23.2%). Additionally, when "reasoning off" is specified, the model still produces think tokens in 1,681 out of 3,127 responses — the model has internalized the reasoning pattern from training and sometimes overrides the system prompt. This suggests the training data's reasoning structure is more important than the specific chat template used at inference.

Sequence packing does not harm performance (Appendix D.3, Table 10): Training with or without packing produces comparable results (e.g., AIME24: 18.7% with packing vs. 22.0% without — within noise for a single evaluation). This contrasts with findings from OpenR1 (Face, 2025), where packing was reported to degrade performance. The authors attribute the difference to LlamaFactory's greedy packing implementation, which only packs shorter sequences together and avoids the truncation issues that affected OpenR1.

Compressing reasoning traces severely degrades performance (Appendix H.3, Table 22): Two methods are tested for reducing training-time reasoning trace length: (1) removing self-reflection keywords ("wait," "but wait," "but the question") from traces, which reduces average length from 11,593 tokens to 328 tokens, and (2) filtering out traces longer than 2,048, 4,096, or 8,192 tokens. Removing self-reflection causes an average relative performance drop of 49.1% across benchmarks (e.g., AIME24: 34.0% → 5.0%; MATH500: 84.0% → 61.8%). The filtering approach shows monotonic degradation: higher thresholds (keeping longer traces) preserve more performance — "Filter > 8192" achieves 42.3 average vs. 51.4 for the baseline, while "Filter > 2048" achieves only 34.2. This demonstrates that long reasoning traces with self-reflection are not merely verbose but contain essential instructional content for the student model.

Verification helped at larger model scales in prior OpenThoughts releases (Appendix H.1.1, Table 15): When the authors revisit their earlier OpenThoughts-114K dataset, they find that verification (removing teacher responses with incorrect final answers) hurt the 7B model (41.9 with verification vs. 45.0 without) but helped the 32B model (64.5 with verification vs. 62.1 without). This reversal by model scale is intriguing: larger models may be better able to distinguish useful partial reasoning from genuinely misleading content in the training data, making them more robust to incorrect examples but also better able to benefit from the cleaner signal that verification provides. For smaller models, the "productive failure" value of incorrect traces may outweigh the noise reduction from verification.

Removing proof-based math questions hurts performance (Appendix H.1.2, Table 16): When proof-based math questions are removed from OpenThoughts-114K (because their answers cannot be verified via numerical matching), downstream performance drops by 5.6 points on average. This suggests that proof-based questions provide valuable reasoning training signal even if their correctness cannot be automatically verified, supporting the decision to skip answer verification in the final pipeline.

LLM-generated unit tests for code verification do not improve downstream performance (Appendix H.1.4, Table 18): Using GPT-4o-mini to generate unit tests and filter code responses to only those whose tests pass produces worse results than no filtering (CodeForces: 10.4% vs. 13.5% for unfiltered). This is additional evidence that verification introduces more harm (through filtering errors and removal of useful training signal) than benefit.

Claude 3.7 with extended thinking underperforms R1 and QwQ-32B as a teacher (Appendix H.2, Tables 19–20): When Claude 3.7 (with thinking mode, up to 62,976 thinking tokens) is substituted as the code or science annotator in OpenThoughts-114K, downstream performance is slightly worse than using R1 (e.g., code annotator swap: AIME24 23.3% vs. 26.6% for R1; science annotator swap: LiveCodeBench 42.1% vs. 45.9% for R1). Claude's longer thinking budget does improve its own performance (Figure 10: AIME24 rises from 18.0% to 51.3% with 62,976 thinking tokens), but this doesn't translate to better teaching. For math, R1 annotations of the s1K dataset dramatically outperform Claude annotations (AIME25: 60.0% vs. 40.0%; MATH500: 95.4% vs. 87.0%), while even Gemini annotations outperform Claude (Table 21).

FastText as an answer filter is competitive only for code (Tables 44–46): For code, fastText filtering based on CodeForces-as-positive/CodeForces-with-GPT4o-mini-as-negative achieves 42.3 (slightly beating No Filtering's 42.2, within noise). For math and science, fastText filtering on answers is consistently mid-table or worse, never exceeding the no-filtering baseline. This suggests that the n-gram quality signal is weaker for math/science reasoning traces than for code, where syntactic patterns (proper use of data structures, algorithmic idioms) may be more predictive of response quality.

Decontamination triage is effective (Appendix F, Figure 7): The chosen thresholds (75% InDel similarity, 13-gram overlap) catch 3,080 of 3,092 manually contaminated samples (99.6% true negative rate) while falsely flagging only 1.4% of clean samples. The manual construction of the contaminated testbed — including exact matches, reworded questions, synonym-substituted variants, and format-altered versions — provides confidence that the decontamination generalizes to realistic duplication patterns. The near-zero false negative rate is critical because even a handful of leaked evaluation questions could explain the 1–3 point improvements observed in some pipeline stages, making the ablation ranking unreliable.

OpenThinker3-7B sometimes surpasses its teacher on specific benchmarks (Table 29): On JEEBench, OpenThinker3-7B achieves 72.4, while its teacher QwQ-32B achieves 69.9. This is one of the few examples in the paper of weak-to-strong generalization — the student exceeds the teacher on a specific benchmark even though the teacher is better on average. However, OpenThinker3-7B remains far below QwQ-32B on most benchmarks (e.g., CodeElo: 31.0 vs. 44.3; GPQA Diamond: 53.7 vs. 65.0), indicating this is an exception rather than a general phenomenon.

Safety alignment degrades with successive OpenThoughts releases (Appendix L, Table 30): As reasoning capability improves across OpenThinker generations, safety alignment degrades. Harmbench harmfulness rate increases: 36.8 (OpenThinker-7B) → 42.8 (OpenThinker2-7B) → 55.5 (OpenThinker3-7B). Over-refusal on XSTEST remains relatively stable (2.4–5.6). The paper acknowledges that OpenThinker3-7B was trained without any safety-tuning data. The finding illustrates a tradeoff between reasoning capability and safety that mirrors observations in prior work (Qi et al., 2023).

Reasoning-invariant generalization remains fragile (Appendix N, Figures 13–14): Evaluated on the "Alice in Wonderland" problems (Nezhurina et al., 2024), which test robust reasoning through structure-preserving variations of simple problem templates, distilled reasoning models show substantial performance fluctuations across variants. OpenThinker3-7B performs worse than larger 32B distilled models on average, while larger 32B models show strong fluctuations — e.g., OpenThinker-32B goes from near 1.0 correct response rate on AIW variations 2–3 down to near 0.0 on variation 1 (Figure 13). For reference, o1-preview and o3-mini show much smaller fluctuations and higher overall rates. However, all distilled reasoning models dramatically outperform conventional LLMs at all scales (Figure 14: 32B reasoning models populate the upper performance region above 0.7; conventional LLMs including Llama 3.1 405B and DeepSeek v3 671B remain confined below 0.2). This shows that while SFT distillation dramatically improves average reasoning capability, it does not confer the robust zero-shot generalization that larger RL-trained models exhibit.

Critical Assessment

Claim: "OpenThinker3-7B is the state-of-the-art open-data 7B reasoning model"

Supported, with important nuance. Table 1 provides consistent evidence: OpenThinker3-7B achieves 55.3 average across 12 benchmarks, exceeding Nemotron-Nano-8B (53.2), AceReason-7B (52.9), Skywork-7B (51.6), OpenR1-Distill-7B (47.2), and DeepSeek-R1-Distill-Qwen-7B (42.9). The margin over the next-best open-data model (Nemotron-Nano-8B, which is actually 8B and uses both SFT and RL) is 2.1 points. On held-out benchmarks, OpenThinker3-7B leads on 3 of 4 (AIME25, LCB 06/24-01/25, HMMT), while trailing on HLE MCQs (10.2 vs. 12.4 for R1-Distill-Qwen-7B). However, the SOTA claim is bounded to the 7B scale and to open-data models — Table 31 shows that frontier API models (Gemini-2.5-Pro at 69.6 average, o4-mini at 67.2, o3 at 67.0) remain far ahead, with gaps of 10+ points persisting primarily on CodeElo and CodeForces. The claim also evaluates only Qwen-2.5-7B-Instruct as the base; Appendix G.1 shows that using Llama-3.1-8B-Instruct produces a weaker absolute model (e.g., AIME24: 37.0 vs. 54.3), so the recipe's optimality may depend on the base model's inherent capabilities.

Claim: "Sampling multiple answers per question is an effective technique to increase dataset size by at least 16×"

Supported, with data-dependent boundary conditions. Table 6 and Tables 41–43 show that 16× sampling variants achieve performance comparable to or better than 1× sampling at the same budget. For code, the gap between No Dedup + 16× (41.3) and No Dedup + 1× (39.8) is 1.5 points. For math, Exact Dedup + 16× (40.1) vs. Exact Dedup + 1× (41.7) actually slightly favors 1× — but the 16× variant allows 16× more total pairs from the same question pool. Figure 8's scaling curves validate that this increased scale translates to improved performance at the 1M level. The boundary condition: this effectiveness relies on starting from high-quality questions (the top 1–2 sources, LLM-filtered). If the underlying questions are poor, 16× sampling would amplify bad training signal. The paper does not test this boundary condition explicitly — the 16× result is only demonstrated within the optimized pipeline, not with unfiltered questions.

Claim: "Models with better performance are not necessarily better teachers"

Strongly supported, but mechanism remains speculative. Tables 8 and 47–49 provide consistent, large-magnitude evidence: QwQ-32B (64.2 average) produces better students than DeepSeek-R1 (65.3) across all three domains, by margins of 1.9–3.6 points. Table 48 is particularly striking: QwQ-32B as math teacher achieves 44.2 vs. DeepSeek-R1's 40.6, a 3.6-point gap that is well outside standard error. The Phi-4-Reasoning-Plus failure (30.6 average for math) establishes that the effect is not monotonic — there is a lower bound beyond which teacher quality matters. However, the paper does not provide a mechanistic explanation. Plausible hypotheses (capability matching, reasoning trace style, length distributions) are mentioned but not tested through controlled experiments manipulating teacher properties while measuring student outcomes. A clean experiment would systematically vary teacher model scale (e.g., 7B, 14B, 32B, 72B, 405B versions of the same model family) while measuring student performance — this would isolate the capability-matching hypothesis from confounds of training recipe, tokenizer, and reasoning style that differ between QwQ-32B and DeepSeek-R1.

Claim: "We experimented with numerous verification and answer filtering methods, and none gave significant performance improvements"

Supported, with the crucial caveat that this holds only after upstream quality filtering. Tables 7 and 44–46 show that across all three domains and across 8 filtering strategies, no method meaningfully outperforms either the no-filtering baseline (63,200 examples, not compute-controlled) or the random filtering baseline (31,600 examples, compute-controlled). The gaps are consistently within 1–2 points, often within standard error. However, this null result is pipeline-dependent. The authors acknowledge this implicitly by noting that their prior work (OpenThoughts-114K) used verification, and Appendix H.1.1 shows verification helped the 32B model (64.5 vs. 62.1) but hurt the 7B model (41.9 vs. 45.0). The paper does not test verification on deliberately degraded question pools (e.g., using lower-ranked sources, or skipping question filtering) to establish the boundary conditions under which verification becomes beneficial. The claim as stated — that verification doesn't help — is only demonstrated for the optimized pipeline, not as a universal principle of reasoning SFT.

A second caveat: the verification experiment generates 63,200 pairs and samples 31,600 after filtering. The "No Filtering" baseline trains on all 63,200 — it is not compute-controlled. To properly test whether filtering helps at identical compute, one would need to generate enough pairs that the filtered set equals 31,600 and the unfiltered set also equals 31,600 (by generating fewer initial pairs). The current design gives the no-filtering baseline an unfair data advantage, making the finding that filtering doesn't beat this advantaged baseline even stronger evidence of filtering's ineffectiveness, but preventing a precise estimate of how much filtering costs in lost performance per removed sample.

Claim: "Selecting questions from a small number of high-quality sources leads to better downstream performance compared to optimizing for diversity"

Strongly supported. Table 4 and Tables 35–37 show consistent, monotonic degradation as more sources are mixed. For code, the peak is at 2 sources (41.3), declining to 36.4 at 16 sources — a 4.9-point drop. For math, Top 1 (37.6) outperforms all mixtures, with Top 16 at 33.8 — a 3.8-point drop. For science, Top 2 (33.7) is best, with Top 16 at 30.8 — a 2.9-point drop. The effect size is substantial, the pattern is consistent across domains, and the monotonic nature of the decline rules out random variation. However, the experiment only tests uniform mixing (equal samples per source). A weighted mixing strategy — e.g., including top sources at 80% and lower sources at 20% — might recover some diversity benefits without full dilution. The paper does not explore this, and the conclusion "fewer sources is better" may be specific to the uniform mixing regime.

Claim: "Filtering questions by LLM labeled difficulty or LLM response length yields better results than classical pre-training data curation filters"

Supported, with a large effect size for math, smaller for code and science. For math (Table 39), LLM response-length filtering achieves 41.9 vs. the best fastText filter at 35.6 — a 6.3-point gap. For code (Table 38), difficulty-based filtering achieves 43.0 vs. the best fastText at 40.5 — a 2.5-point gap. For science (Table 40), LLM response-length achieves 35.9 vs. the best fastText at 35.1 — a 0.8-point gap within noise. The claim holds most clearly for math, moderately for code, and weakly for science. The paper does not ablate why LLM-based filtering works better — is it the rubric detail, the model's world knowledge, or the behavioral measure of question complexity? A useful ablation would compare difficulty-based filtering from GPT-4o-mini against the same rubric applied by a much weaker model (e.g., GPT-4o-mini vs. a 1B model fine-tuned for difficulty estimation) to determine whether it's the rubric or the model capability that matters.

Missing Experiments That Would Strengthen the Paper

Joint optimization of pipeline stages. The greedy sequential approach (optimize Stage 1, lock it in, optimize Stage 2, etc.) is computationally efficient but cannot detect interactions between stages. Would the optimal question filtering strategy change if using QwQ-32B instead of DeepSeek-R1 as teacher? Would the optimal mixing ratio depend on whether answer filtering is applied? The paper provides no evidence either way, and the parallel upward shift of scaling curves in Figure 3 provides some reassurance but not a rigorous test for interaction effects.

Teacher capability scale sweep. The teacher model comparison includes only three points (DeepSeek-R1 at 671B, QwQ-32B at 32B, Phi-4 at 14B). A systematic sweep over model scales within a single model family (e.g., Qwen-2.5 at 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B) would reveal whether there is an optimal teacher-to-student scale ratio, whether the relationship is monotonic or inverted-U-shaped, and whether the "weaker teacher paradox" is about absolute scale, relative scale, or model-specific factors.

Question filtering sensitivity to filter strength. The paper uses specific models (GPT-4o-mini, GPT-4.1-mini, GPT-4.1-nano) for filtering but doesn't test whether using a stronger model (GPT-4.1, or even DeepSeek-R1 itself) for difficulty labeling or response-length estimation would further improve filter quality. Given that stronger filtering models produced better filtering results (GPT-4.1-mini > GPT-4.1-nano > GPT-4o-mini), the natural question is whether this trend continues — would GPT-4.1 produce even better filters?

Testing on a deliberately degraded question pool. The answer filtering null result is only demonstrated after upstream question filtering. It would be informative to test whether answer verification does help when questions are drawn from lower-ranked sources (e.g., mixing 16 sources with no filtering). This would establish the boundary conditions for the verification null result and provide practical guidance for teams with less curated question pools.

Replication on non-Qwen base models at full scale. The paper verifies that the recipe transfers to Llama-3.1-8B-Instruct at 100K scale (Appendix G.1), but the final model is only Qwen-based. The SOTA claim would be stronger with a Llama-based OpenThinker3-8B achieving competitive results. The 100K results are promising but don't guarantee that the full 1.2M recipe is optimal for Llama.

Interaction between question domain and cross-domain transfer. The paper acknowledges (Section 6, Open Direction 1) that it optimizes for overall average performance, not per-domain, and assumes cross-domain transfer (e.g., math data helps science). But it never directly quantifies this transfer: how much does math-only training improve science scores? How much does code-only training improve math? Answering these questions would validate or refute the assumption that optimizing for average is appropriate.

Wall-clock latency vs. inference cost for repeated sampling. The 16× sampling strategy increases inference cost (more teacher calls) during dataset construction, but at deployment time, the student model generates a single response at fixed cost. The paper accounts for annotation GPU hours (22,000 H100 hours) but doesn't discuss the cost tradeoff between finding more unique questions (human or LLM generation cost) vs. annotating the same questions multiple times (teacher inference cost). For practitioners deciding between these strategies, a cost analysis would be valuable.

Statistical significance of close rankings. Many ranking differences in the tables are within 1–2 points with overlapping standard errors (e.g., Table 44: fastText at 42.3(0.5) vs. No Filtering at 42.2(0.5)). The paper bolds values within two standard errors of the highest-scoring strategy, which is an appropriate convention, but the reader should understand that many of the "winning" strategies have overlapping confidence intervals with the "losing" strategies. For the pipeline decisions with small margins (code deduplication, science question filtering), the selected strategy may not be reliably better than alternatives — it may merely be tied with them, and the choice is arbitrary. The paper correctly notes this in some places but could foreground it more prominently, particularly for stages where the chosen strategy (e.g., difficulty-based filtering for code) has only a 0.8-point edge over the runner-up (response-length with GPT-4.1-nano at 42.2).

Single dataset scale for ablations. All pipeline decisions were optimized at 31,600 examples. Figure 3 provides evidence that the rankings are stable across scales (the curves shift upward in parallel), but this is shown only for the winning strategies, not for all candidate strategies at each stage. It's possible that a strategy that performs second-best at 31.6K (e.g., AskLLM filtering for math) would outperform the winner at 1M (response-length filtering) if evaluated at that scale. The paper cannot rule this out because it only scaled the winners.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Efficiency Claims

The assumption or constraint. The entire compute-optimal framework depends on estimating prompt difficulty before allocating the inference budget. The paper's method for doing so requires generating 2048 samples per question and scoring them with the PRM to compute the average final-answer score, then binning questions into difficulty quintiles. The authors explicitly acknowledge this cost in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost would equal (difficulty estimation) + (strategy execution). Generating 2048 samples per question for difficulty estimation already consumes more compute than the largest test-time budgets studied (256–512 generations). If a practitioner attempts to deploy this system, the true efficiency gain may be substantially smaller than 4× — the difficulty oracle cost could dominate the entire budget, making the approach less efficient than simply applying a uniform best-of-N strategy without a difficulty estimation step. For applications involving many unique prompts (e.g., answering user questions), the per-prompt estimation cost would be prohibitive. For applications with a fixed prompt distribution (e.g., benchmarking), the estimation cost could be amortized but still represents a significant upfront investment before any answers are produced.

What evidence exists in the paper. Section 3.2 describes the estimation procedure explicitly: "they average the PRM's predicted final-answer correctness across the same 2048 samples per question, then bin into five quintiles." The paper acknowledges this cost is not included in any budget calculation. Figures 4 and 8 show that predicted difficulty bins track oracle bins closely, confirming the estimation method works — but these curves exclude the estimation cost from the x-axis, so the reader cannot assess whether the total cost (estimation + execution) still beats best-of-N. The paper does not provide a cost-inclusive comparison anywhere.

Mitigation status. The paper flags this as a key avenue for future work: "future work on training models to directly predict difficulty from the question text" (Section 3.2), or more dynamic approaches where "the difficulty estimator could serve double duty: it determines how much test-time compute to allocate and whether to escalate to the larger model." The paper also suggests "adaptive difficulty estimation" where initial samples inform both difficulty assessment and answer generation. However, none of these approaches are implemented or evaluated. As of this paper, the limitation remains unresolved, and the 4× figure should be understood as an upper bound on achievable efficiency under an oracle difficulty signal, not a realized deployment gain.

The Method Provides No Benefit on the Hardest Problems — a Fundamental Capability Ceiling

The assumption or constraint. The compute-optimal framework operates under the implicit assumption that the base model's proposal distribution contains correct solutions at some non-trivial rate — test-time compute can amplify existing capability but cannot create capability that is absent. When the base model's pass@1 is near zero, no amount of search, revision, or adaptive allocation can recover correct answers.

The consequence. For the hardest questions (difficulty bin 5, defined as the quintile with the lowest base model pass@1), all methods — search, revisions, and their compute-optimal combinations — show near-zero improvement regardless of the compute budget allocated. In the FLOPs-matched comparison, scaling test-time compute cannot make a smaller model competitive with a ~14× larger one on hard problems. This establishes a sharp deployment constraint: organizations deploying this system must have a separate strategy for hard problems (e.g., routing them to larger models or human experts), because throwing inference compute at them is ineffective. More fundamentally, it means that for problems genuinely outside the base model's training distribution — novel math, unfamiliar reasoning patterns — pretraining remains the only viable path to capability, and inference-time scaling offers no shortcut.

What evidence exists in the paper. The evidence is consistent and stark across multiple analyses:

Figure 3 (right): On difficulty bin 5 (hardest), beam search and best-of-N both hover at 1–3% accuracy at all budgets from 4 to 256 generations. The lines are essentially flat and overlapping — neither method makes any progress.
Figure 7 (right): On difficulty bin 5, all sequential-to-parallel ratios produce roughly 2–3% accuracy. No allocation strategy helps.
Figure 9: In the FLOPs-matched comparison, the bin 5 scaling curve for revisions is nearly flat near 0–5% across all budget levels, while the 14× larger model (represented by the star) achieves substantially higher performance. At R ≫ 1, hard problems show a -52.9% relative disadvantage from using test-time compute instead of the larger model (PRM search).
Section 7 takeaway box: The paper is transparent: "test-time compute can amplify existing capability but does not create it from nothing."

Mitigation status. The paper does not attempt to solve this limitation. It explicitly identifies it as a boundary condition: test-time compute is effective when the base model's pass@1 is non-trivially above zero, and ineffective when it is near zero. There is no proposed method within this framework to handle hard problems — the recommendation is implicit: use a larger pretrained model instead. Future work that combines test-time compute with retrieval, tool use, or iterative self-improvement loops (where the model generates its own training data) might push the capability boundary outward, but the paper does not explore these directions.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and Training Is Brittle

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct answer. This is a direct consequence of the training data construction (Section 6.1): for each training question, the authors construct multi-turn sequences of 0–4 incorrect answers followed by a correct answer, where the last incorrect answer is selected to have minimum edit distance to the correct answer. The model is never trained on sequences where a correct answer appears in context and should be preserved.

The consequence. At inference time, when a revision chain happens to produce a correct answer at some step, the model has no training signal for what to do next. Approximately 38% of correct answers get "revised" into incorrect answers in the subsequent revision step (Section 6.1). This creates a fundamental instability in the revision chain: correctness is not absorbing, and the model can oscillate between correct and incorrect states. The paper mitigates this with post-hoc selection (majority voting or verifier-based selection) across the entire chain, but this is an imperfect patch — it discards potentially useful subsequent revisions that might have refined the correct answer further, and it adds computational overhead to evaluate every step in the chain. More fundamentally, the reversion problem reveals that the revision model has not learned when not to revise — it lacks a "stop" criterion or a signal that the current answer is already acceptable.

What evidence exists in the paper. Section 6.1 explicitly states: "approximately 38% of correct answers get converted back to incorrect ones using a naive approach." Figure 6 (left) shows that pass@1 per step improves gradually but never stabilizes — it fluctuates in the 23–25% range from steps 15–64, consistent with a process that occasionally reverts correct answers. The ReST^EM experiment (Appendix K, Figure 16) provides further evidence of training brittleness: attempting to further optimize the revision model with RL-style on-policy data collection caused performance to degrade substantially with sequential revisions ("fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio"). The hypothesized mechanism is that on-policy data collection amplifies spurious correlations in revision trajectories.

Mitigation status. Partially addressed via post-hoc selection but not fundamentally solved. The paper uses majority voting or verifier-based selection across the revision chain to pick the best answer from any point, which recovers performance but does not prevent the reversion from occurring. The paper does not explore more principled solutions: training the revision model with a "no-change" target when the current answer is already correct (requiring a different data construction), incorporating a confidence or uncertainty signal that gates revision, or using the PRM's step-level scores to detect when a revision is going off-track and terminate the chain early. These are identified as directions for future work but not implemented.

The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the compute-optimal pretraining paradigm (Hoffmann et al., 2022) where both data and parameters are scaled equally. The authors explicitly acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search of any kind.

The consequence. The reported advantages of test-time compute over pretraining may be overstated because the pretraining baseline is weaker than it could be. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, potentially narrowing or reversing the reported advantages. Furthermore, giving the larger model even a modest test-time compute budget — say, best-of-8 with majority voting — would create a much stronger baseline. The FLOPs-matched comparison as presented answers the question "is test-time compute with a small model better than a naïvely scaled larger model with no inference-time augmentation?" This question is relevant for demonstrating the potential of test-time compute, but it does not answer the more practically relevant question: "given a fixed total compute budget (pretraining + inference), should I spend the marginal dollar on a larger model or on smarter inference?" Answering that question would require comparing a compute-optimally trained larger model (possibly with some test-time compute) against a compute-optimal smaller model with test-time compute.

What evidence exists in the paper. Section 7 describes the FLOP accounting and the baseline: "the comparison is between PaLM 2-S* with compute-optimal test-time scaling and a model with approximately 14× more parameters (greedy decoding, no extra test-time compute)." The paper acknowledges the departure from compute-optimal pretraining explicitly in Section 7, noting that this choice is a limitation. Figure 9 and the bar charts in Figure 1 show the results: at R ≪ 1 on easy questions, test-time compute shows +27.8% relative improvement over the larger model (revisions). These numbers should be interpreted as upper bounds on the advantage test-time compute can provide — against a stronger pretraining baseline, they would likely shrink.

Mitigation status. The paper acknowledges this as a limitation and defers it to future work: "leave the analysis of compute-optimal scaling of pretraining compute... to future work." No experiments test against a Chinchilla-optimal larger model or against a larger model with any test-time compute augmentation. The paper also does not explore giving the larger model the same test-time compute budget as part of the comparison. The authors justify their choice by noting it "is representative of a canonical approach to scaling pretraining compute" (the LLaMA paradigm), which is a reasonable practical choice but weakens the strength of the claim that test-time compute can "substitute" for pretraining.

Results Are Validated on a Single Benchmark Family (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The findings about difficulty-dependent behavior, search strategy effectiveness, revision model behavior, and FLOPs-matched tradeoffs are all demonstrated within a single (benchmark, model) pair.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution, including its calibration, error patterns, and reasoning style. A model with different characteristics — e.g., a code-generation model rather than a math model, or a model from a different family (Llama, GPT, Gemini) — might exhibit different difficulty-dependent scaling curves.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning and instruction-following capabilities, which vary substantially across model families and scales.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and exact-answer matching. It is unclear whether the difficulty-dependent patterns generalize to other reasoning domains (code generation, scientific reasoning, logical deduction) or to tasks requiring factual knowledge, subjective judgment, or open-ended generation.
The test set of 500 questions is split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample for strategy selection, and the observed gains may not be statistically robust or may not generalize to a different sample of 500 MATH questions.

What evidence exists in the paper. All figures (3, 4, 6, 7, 8, 9) and tables report results exclusively on MATH. The paper does not include any experiments on other reasoning benchmarks (e.g., GSM8K, MMLU, HumanEval, MBPP, ARC, BIG-Bench) or other model families. Section 4 acknowledges the choice of MATH as deliberate: "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences — mathematical reasoning fits this profile." This is a reasonable scope for an initial study, but it means the findings' generality is unknown. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess statistical reliability at the ~50-question-per-bin sample size.

Mitigation status. Not addressed. The paper does not claim generality beyond MATH with PaLM 2-S*, but it also does not explicitly caution readers that the findings may be domain-specific or model-specific. From the authors' perspective: "We believe this model is representative" (Section 4), but no evidence is provided to support this belief. Replication on other benchmarks and model families is identified only indirectly as future work (e.g., the paper's broader framing of "future work on scaling the recipe to other models and tasks" is not explicitly stated but implied by the open-source release). The limited scope is a practical necessity given the computational cost of the experimental program, but it means practitioners deploying similar methods on different models or different task types should expect to need their own calibration experiments rather than directly adopting the paper's difficulty thresholds or strategy preferences.

Sequential Revisions Introduce Latency That Is Not Accounted For in the Cost Model

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled from the model), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial: each revision depends on the output of the previous one, and the full chain must execute in sequence. In contrast, parallel best-of-N sampling can be executed simultaneously given sufficient hardware (e.g., batching all N generations across multiple GPUs).

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel (a hybrid strategy that the compute-optimal policy might select for medium-difficulty problems) takes roughly 64× longer wall-clock time than a strategy that runs 128 parallel samples simultaneously, even though both consume the same total FLOPs. For latency-sensitive applications — interactive assistants, real-time decision-making, chatbots where users expect sub-second responses — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their FLOPs efficiency. The paper's optimization framework (Equation 1) maximizes accuracy for a given generation budget but does not include latency as a constraint or as part of the optimization objective. A practitioner deciding between a 64-sample parallel strategy (high latency but low per-sample) and a 64-step sequential chain (much higher latency) would need to weigh accuracy against user experience, and the paper provides no guidance on this tradeoff.

What evidence exists in the paper. Section 6 describes the revision model's sequential generation process: "the revision model generates a chain of revisions: it produces an initial answer, then conditions on that answer to produce a revision, then conditions on the revision to produce another revision, and so on." The cost model in Section 5.3 counts generations but does not mention latency. Section 7's FLOP accounting (Y = 2ND_inference) counts total inference FLOPs, not wall-clock time. Nowhere does the paper discuss latency, throughput, or the practical implications of serial vs. parallel computation for deployment.

Mitigation status. Not addressed. The paper treats "generations" as the universal unit of compute, implicitly assuming that the cost of a generation is constant regardless of whether it is executed in sequence or in parallel. For throughput-oriented batch inference (evaluating thousands of problems), this assumption is reasonable because many problems can be processed in parallel, and the serial chain for each problem executes concurrently with chains for other problems. For latency-oriented interactive deployment, the assumption breaks down. The paper does not discuss this distinction, nor does it suggest latency-aware allocation strategies (e.g., capping the sequential depth, switching to parallel strategies when latency exceeds a threshold, or using speculative decoding to accelerate sequential generation).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a novel training algorithm or model architecture. Instead, it changes the landscape by transforming reasoning SFT data curation from a craft guided by heuristics and precedent into an empirical science with controlled, falsifiable hypotheses. Prior to this work, practitioners building reasoning SFT datasets made design choices—which teacher model to use, how many question sources to combine, whether to verify answers—based on intuition, prior beliefs, or institutional momentum. The paper systematically dismantles several of these beliefs through 1,000+ controlled ablation experiments, and in doing so, establishes a methodology rather than merely releasing an artifact.

The magnitude of this shift is best understood by examining what the paper overturns. Three widely-held assumptions are empirically refuted:

1. Stronger teachers produce better students (Section 4.6). DeepSeek-R1 achieves a 65.3 overall benchmark average versus QwQ-32B's 64.2 (Table 29), with particularly large leads on CodeElo (+9 points), GPQA Diamond (+8 points), and JEEBench (+22 points). Yet QwQ-32B produces training data that yields better student models across all three domains—by 1.9 points for code, 2.6 points for math, and 3.2 points for science (Tables 47–49). This is not a marginal effect within statistical noise; the gap is consistent and substantial. The implication is that teacher selection for distillation must be based on downstream student performance, not teacher benchmark scores. Teams that default to the strongest available model as their annotator are leaving performance on the table, and the paper provides a concrete alternative: systematically evaluate candidate teachers by training small student models and measuring the outcome, rather than relying on proxy metrics.

2. Diverse question sources produce better reasoning datasets (Section 4.2). The standard practice in prior reasoning datasets—OpenR1, OpenThoughts2, Amazon-Nova—was to aggregate questions from many sources under the assumption that diversity improves generalization. The paper shows this assumption is wrong under controlled conditions: mixing the top 2 code question sources achieves a 41.3 average, while mixing the top 16 sources achieves 36.4 (Table 4), a monotonic 4.9-point degradation. The mechanism is quality dilution: lower-ranked sources introduce questions with ambiguous phrasing, flawed problem structures, or inappropriate difficulty that confuse the student during training, and the negative impact outweighs any benefit from seeing a wider variety of problem types. The practical consequence is that data curation effort should be redirected from finding more sources to extracting more value from the best sources—the 16× repeated sampling strategy (Section 4.4) provides exactly this alternative path to scale.

3. Answer verification is a necessary quality-control step (Section 4.5). The authors' own prior releases (OpenThoughts-114K, OpenThoughts2) employed answer verification, as did many other reasoning datasets. The intuition is straightforward: training on incorrect answers should degrade student performance. When the authors systematically test this assumption across 8 filtering strategies and all three domains, they find that no strategy outperforms training on all available data (Tables 44–46). For math, the best verifier (random filtering, which is not a verifier at all) achieves 41.6 versus 41.9 for no filtering. For code, fastText filtering achieves 42.3 versus 42.2. These differences are well within standard error. The implication is that teacher responses—even when incorrect—contain pedagogically valuable partial reasoning (demonstrating valid approaches, encountering errors, pivoting to alternatives) that verification would remove, and that the verification methods themselves introduce classification errors that offset any benefit from eliminating genuinely misleading examples.

The paper also reconciles apparent contradictions in the prior literature. The finding that verification helped the 32B model but hurt the 7B model in earlier OpenThoughts releases (Appendix H.1.1, Table 15) is no longer mysterious: larger models may be more robust to incorrect training examples, making the "clean signal" benefit of verification outweigh the "loss of productive failure" cost, while smaller models benefit more from seeing diverse reasoning paths including errors. Similarly, the finding that Claude 3.7 with extended thinking (62,976 thinking tokens) dramatically improves its own performance (AIME24: 18.0% → 51.3%, Figure 10) but produces worse training data than R1 or QwQ-32B (Appendix H.2, Tables 19–21) resolves the apparent paradox: the properties that make a model good at solving problems (accuracy, efficiency) are not the same properties that make it good at teaching.

Beyond overturning specific assumptions, the paper establishes a reusable experimental template for reasoning data research. The design—fix a dataset size (31,600 examples, the log-scale midpoint between 10K and 100K), fix the student model (Qwen-2.5-7B-Instruct), fix the evaluation suite (8 in-distribution + 4 held-out benchmarks), vary exactly one pipeline component while holding all others constant, and measure downstream accuracy—is replicable by any team with access to teacher model inference and training compute. The full release of datasets, models, code, and ablation results at openthoughts.ai lowers the barrier for the community to test new question sources, new teacher models, new filtering strategies, or new mixing ratios within this framework. This shifts the field's trajectory from competing artifacts (each team releasing a dataset and claiming it is best) toward cumulative science (each team contributing evidence about what design choices matter and why).

The paper also redirects research attention away from certain directions. The consistent null result on answer verification (Section 4.5) suggests that further investment in more sophisticated verification methods—LLM judges, unit test generation, consensus algorithms—is unlikely to yield substantial gains when starting from already-filtered high-quality question pools. The finding that deduplication provides at most marginal benefits (Section 4.4, with spreads of 1–2 points across 9 combinations) similarly suggests that deduplication effort is better allocated elsewhere in the pipeline. Conversely, the strong performance of LLM-based question filtering—particularly response-length filtering with GPT-4.1-mini achieving 41.9 versus 35.2 for random filtering on math (Table 39)—suggests that further investment in better filtering models and methods is likely to be high-return.

Follow-Up Research This Work Enables

Systematically mapping the teacher-to-student capability ratio. The paper establishes that QwQ-32B outperforms DeepSeek-R1 (671B) as a teacher for a 7B student, but provides only three points on the teacher spectrum: DeepSeek-R1 (671B), QwQ-32B (32B), and Phi-4-Reasoning-Plus (14B). The natural next step is a controlled sweep within a single model family—for example, using Qwen-2.5 variants at 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B parameters as teachers for a fixed 7B student (Qwen-2.5-7B-Instruct). This would reveal whether the "weaker teacher paradox" is driven by absolute teacher scale, by the teacher-to-student parameter ratio, or by model-specific factors (architecture, training recipe, reasoning style). The experiment would train a student on 31,600 examples from each teacher and measure downstream performance, producing a U-shaped (or inverted-U-shaped) curve if an optimum teacher scale exists. Such a curve would have immediate practical value: given a student model size, what teacher size maximizes student performance? The paper's results suggest the optimum is not at the extreme of "strongest teacher," but whether it sits at 32B, 14B, or some other ratio is unknown.

Testing the answer filtering null result on deliberately degraded question pools. The paper's headline finding that answer verification does not help (Section 4.5) is demonstrated only within the optimized pipeline—where questions have already been filtered for quality via LLM-based methods (Section 4.3). A critical follow-up experiment would systematically degrade question quality and test whether verification becomes beneficial. Specifically: (1) take the lowest-ranked 4 question sources from Section 4.1 (e.g., for code: StackExchange, PrimeIntellect/synthetic-code-understanding, cfahlgren1/react-code-instructions, bugdaryan/sql-create-context-instruction), (2) generate 31,600 question-answer pairs without any question filtering, (3) apply the same suite of answer filtering strategies from Section 4.5, (4) train student models, and (5) measure whether verification now provides gains. If verification helps only when question quality is poor, the practical guideline becomes: "If you have a high-quality question pipeline, skip verification; if your question sources are noisy or untrusted, verification may help." If verification still does not help even on degraded data, the finding becomes much stronger—suggesting that teacher reasoning traces are robustly useful regardless of question quality. Either outcome refines the paper's central null result with actionable boundary conditions.

Joint optimization of pipeline stages to detect interaction effects. The paper's greedy sequential optimization (optimize Stage 1, lock it in, optimize Stage 2, etc.) is computationally necessary but cannot detect interactions between stages. A follow-up could test specific hypothesized interactions at smaller scale. For example: does the optimal question filtering strategy change when the teacher model switches from DeepSeek-R1 to QwQ-32B? If QwQ-32B produces reasoning traces that are more sensitive to question quality (because its responses are more variable or error-prone), then response-length filtering might become more important with QwQ-32B than with DeepSeek-R1. To test this, one would run a 2×2 factorial experiment: {DeepSeek-R1, QwQ-32B} × {No Filtering, Response-Length Filtering} at the 31,600 scale, measuring whether the filtering gain depends on the teacher. The paper's parallel upward-shifted scaling curves in Figure 3 suggest interactions are limited (since gains compound additively), but a formal test would provide stronger evidence. A particularly important interaction to test is whether repeated sampling (Section 4.4) amplifies or dampens the benefit of question filtering—if filtering removes ambiguous questions, do the 16 different teacher responses to a clear question provide more varied and useful training signal than 16 responses to an ambiguous question?

Extending the recipe to RL-based reasoning training. The paper's scope is explicitly limited to SFT-only distillation (Section 1), leaving the interaction between data curation and reinforcement learning unexplored. A natural follow-up would take the OpenThoughts3-1.2M dataset as the SFT base and apply an RL stage (e.g., using GRPO or outcome-based reward modeling on math verification or unit test execution). The key question: does the SFT data recipe that optimizes SFT-only performance also produce the best initialization for RL, or does the RL stage have different data preferences (e.g., preferring more diverse questions even at lower quality, since RL can self-correct from noisy SFT initialization)? A controlled experiment would compare: (1) SFT on OpenThoughts3-1.2M → RL, versus (2) SFT on a more diverse but lower-quality dataset (e.g., the "No Pipeline" baseline from Section 5) → RL, measuring whether the SFT quality advantage persists after RL. The paper's finding that Nemotron-Nano-8B (SFT + RL) achieves 53.2 versus OpenThinker3-7B's 55.3 (SFT only, Table 1) suggests that optimized SFT alone can match or exceed SFT+RL with less optimized data—but whether optimized SFT + RL would push even further ahead is unknown.

Cost-benefit analysis of question filtering model strength. The paper shows that stronger models produce better question filtering results—GPT-4.1-mini > GPT-4.1-nano > GPT-4o-mini for response-length filtering on math (Table 39)—but does not analyze the cost-performance tradeoff. A follow-up could map the filtering quality curve against model cost: use GPT-4o-mini, GPT-4.1-nano, GPT-4.1-mini, GPT-4.1, and even reasoning models like QwQ-32B or DeepSeek-R1 as question filters, measuring (1) downstream student accuracy and (2) per-question filtering cost in dollars or FLOPs. The key practical question: where is the knee in the curve? If GPT-4.1-mini provides 95% of the benefit of a much more expensive filtering model (like DeepSeek-R1), it is the economically optimal choice. If the curve continues to rise, it may be worth spending more on filtering than on annotation—a counterintuitive allocation that the current paper cannot rule out. The experiment would also test whether self-filtering (using the teacher model itself for question quality assessment before annotation) is more effective than using a separate filtering model, potentially simplifying the pipeline by eliminating the need for a distinct filter model.

Cross-domain transfer efficiency quantification. The paper optimizes for average performance across math, code, and science, assuming cross-domain transfer (Section 6, Open Direction 1). But the magnitude and efficiency of this transfer is never directly measured. A clean follow-up experiment would train three models: (1) math-only training on 31,600 math examples, (2) code-only training on 31,600 code examples, (3) science-only training on 31,600 science examples. Each model would be evaluated on all three domain suites. The cross-domain transfer matrix would quantify: how much does math training improve code and science scores? How much does code training improve math? If math training strongly transfers to science (as the paper's qualitative observations suggest) but code training does not, then the optimal dataset composition for a fixed 1.2M budget might be heavily math-weighted with a small code/science component. If transfer is symmetric, a balanced mix is appropriate. This experiment would replace the paper's "we followed OpenThoughts2 ratios" heuristic with empirical evidence for the optimal domain allocation.

Practical Applications and Downstream Use Cases

Cost-efficient construction of custom reasoning datasets for specialized domains. The paper's pipeline methodology is domain-agnostic: the same stages (question sourcing → filtering → teacher annotation → repeated sampling) can be applied to any domain where reasoning traces can be elicited from a strong teacher model. A concrete use case: a legal tech company building a reasoning model for contract analysis or case law interpretation. Following the OpenThoughts3 recipe, they would: (1) source questions from legal forums, bar exam questions, and case summaries; (2) filter using LLM-based response-length or difficulty assessment (with GPT-4.1-mini evaluating legal questions against a bar exam difficulty rubric); (3) annotate the top 1–2 question sources with 16× sampling from a strong legal reasoning model (e.g., fine-tuning a general-purpose model on legal data first); and (4) skip answer verification entirely, saving the cost of expert legal review for answer correctness. The paper's finding that 16× sampling on 75K unique questions produces a 1.2M-example dataset means the legal team only needs to identify and filter ~5,000 high-quality legal questions to reach a 80K-example scale. The expected benefit, grounded in the paper's scaling curves (Figure 8), is log-linear improvement with dataset size: a 100K legal reasoning SFT dataset following this recipe would be expected to substantially outperform a naively-constructed 1M-example dataset that mixes many lower-quality sources.

Open-source reasoning model development at resource-constrained institutions. The paper's total compute cost—22,000 H100 GPU hours for annotation and 25,000 A100 GPU hours for training (Appendix O)—is substantial but achievable for well-resourced academic labs or mid-size companies, especially given the trend toward cheaper and faster inference. The key practical insight is the experimental methodology: the 31,600-example ablation scale provides reliable ranking information at a fraction of the cost of the full 1.2M run. A team with a limited budget can: (1) run 10–20 controlled ablation experiments at the 31,600 scale to optimize their pipeline for their specific domain, base model, and evaluation benchmarks, (2) select the winning recipe, and (3) scale it to the maximum dataset size their budget allows, confident that the ablation rankings will generalize upward (as validated by the parallel scaling curves in Figure 3). The paper's release of the full codebase, evaluation framework (Evalchemy), and decontamination tools makes this a turnkey process for any team with GPU access.

Teacher model selection as a standard step in distillation pipelines. The paper's finding that a weaker teacher can produce a stronger student—and that teacher benchmark performance does not predict distillation effectiveness—has immediate practical implications for any team doing knowledge distillation. The recommended procedure, directly derived from the paper's methodology: before committing to a teacher model for a large-scale distillation run, train small student models (at the 10K–30K example scale) on data from 2–3 candidate teachers, evaluate on the target benchmarks, and select the teacher that maximizes student performance. The cost of this selection step is modest: 2–3 annotation runs at 30K examples plus 2–3 small training runs, perhaps $500–1000 in API costs and a few hundred GPU hours. The benefit is avoiding the much larger cost of a full-scale distillation run using a suboptimal teacher. Given that the gap between QwQ-32B and DeepSeek-R1 as teachers for a 7B student was 2–3 points on average (a ~5% relative improvement), and that a full 1.2M annotation and training run costs tens of thousands of GPU hours, the ROI of teacher selection experiments is extremely high.

Rapid iteration on reasoning data quality as new teacher models emerge. The reasoning model landscape evolves quickly: between the paper's experiments and its publication, new models (Qwen 3, Gemini 2.5 Pro, Claude 4, Llama 4) have appeared with potentially different reasoning styles and distillation properties. The paper's pipeline provides a systematic way to evaluate each new model as a teacher. A concrete workflow: when a new reasoning model is released, a team takes their existing high-quality question set (already filtered through the pipeline), generates 31,600 annotations from the new model, trains a student, and compares against the baseline teacher (QwQ-32B). If the new model produces a better student, they can scale up annotation; if not, they can continue using QwQ-32B. This transforms teacher model selection from a one-time design choice into a continuous monitoring and upgrade process, analogous to how production ML systems continuously evaluate new base models for their specific tasks. The paper's finding that Claude 3.7 with extended thinking was a worse teacher than R1 for code and science (Appendix H.2) despite being a stronger model on benchmarks demonstrates that this evaluation is not predictable from public benchmark scores—it must be measured directly.