Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

ArXiv: 2308.01825

🎯 Pitch

Pre-training loss predicts math reasoning skill better than model size, and weaker models gain more from extra data. Surprisingly, having a model generate and filter its own correct solutions—without new human labels—can boost a 7B LLaMA’s GSM8K accuracy from 35.9% to 49.3%, and mixing reasoning paths from multiple models pushes this even higher.

---

1. Executive Summary

This paper empirically studies the scaling behavior of mathematical reasoning in large language models by analyzing how three factors—pre-training loss, supervised data amount, and augmented data amount—influence the supervised fine-tuning performance of LLaMA and LLaMA2 models on the GSM8K benchmark. The paper identifies a core mechanism called Rejection Sampling Fine-Tuning (RFT), a data augmentation approach where a supervised model generates correct reasoning paths via rejection sampling (sampling $k$ candidate solutions, filtering to keep only those with correct final answers, and deduplicating by distinct equation lists), which are then used to fine-tune the base model. RFT improves LLaMA-7B from 35.9% SFT accuracy to 49.3% (a gain of +13.4 points), and establishing that combining rejection samples from multiple models pushes performance further—achieving 49.3% on LLaMA-7B, 50.3% on LLaMA2-7B, 52.1% on LLaMA-13B, and 55.4% on LLaMA2-13B—while revealing crucial boundary conditions: pre-training loss is a better performance indicator than parameter count, and the benefits of both SFT and RFT diminish as the pre-trained model becomes stronger, establishing that RFT brings more improvement only on less performant models.

2. Context and Motivation

The Core Problem: We Don't Know How to Predict or Improve Mathematical Reasoning in Supervised LLMs

This paper tackles a fundamental gap in our understanding of how large language models acquire mathematical reasoning ability through supervision. While LLMs have demonstrated considerable abilities on math reasoning tasks like GSM8K (Cobbe et al., 2021), the field lacks a systematic understanding of what factors actually govern the final supervised model's performance. Specifically, there is no clear answer to questions like: If I have a choice between a model with 7B parameters and one with 13B parameters, which should I fine-tune? How much supervised data do I really need? If I can generate more training data from the model itself, how much improvement should I expect, and on which model sizes?

This gap matters because practitioners make resource allocation decisions based on implicit assumptions about these scaling relationships. Teams deciding how to invest their compute budget—whether to pre-train a larger model, collect more human-annotated data, or invest in data augmentation techniques—lack empirical guidance grounded in controlled experiments. The paper frames this explicitly in terms of understanding "the scaling relationship of factors that influence the math reasoning abilities of a supervised LLM" (Section 3), positioning the work as an empirical investigation into the functional forms connecting model quality, data quantity, and augmented data quantity to downstream reasoning performance.

Why This Problem Matters: Real-World Deployment, Not Just Benchmarks

The paper's practical motivation is grounded in online deployment scenarios. The authors explicitly contrast their approach with the dominant paradigm of using in-context learning (ICL) or ensembling multiple inferences to boost reasoning accuracy. While few-shot prompting (Wei et al., 2022b) and techniques like self-consistency (Wang et al., 2023) improve performance, they require running multiple forward passes per query at inference time, which is "computationally expensive and not suitable for online deployment scenarios" (Section 1). For production systems where latency and throughput matter, a supervised model that generates the correct answer in a single greedy decode—what the paper measures as maj1@1 or accuracy—is the relevant deployment target.

This distinction is crucial for understanding the paper's scope. The authors are not studying how to maximize raw benchmark numbers (which could involve test-time search, verifier reranking, or majority voting across 100 samples). Instead, they are studying what makes a single forward pass from a fine-tuned model correct. This is a harder problem than ensembling, but it's the one that matters most for building efficient deployed systems. The paper's interest in RFT follows directly from this framing: RFT augments the fine-tuning dataset, potentially improving the single-pass accuracy of the resulting model without requiring test-time computation overhead.

Prior Approaches and Where They Fall Short

The paper identifies several streams of prior work and positions its contributions relative to specific limitations in each.

In-Context Learning and Multi-Inference Methods

The dominant approach in the literature for improving math reasoning accuracy at the time of this paper's writing involved leveraging multiple inferences at test time. This includes few-shot chain-of-thought prompting (Wei et al., 2022b), where a handful of worked examples are prepended to the query, and self-consistency decoding (Wang et al., 2023), where the model generates multiple reasoning paths and the most common answer is selected via majority voting. Cobbe et al. (2021) trained a verifier to score and select among candidate solutions, and Uesato et al. (2022) explored reinforcement learning with both outcome-based and process-based reward models. Lightman et al. (2023) pushed this further with large-scale human annotation of step-level correctness signals.

The limitation from the paper's perspective is straightforward: these methods all incur significant test-time computation that makes them unsuitable for online deployment. A system that must sample 100 reasoning paths per query and run majority voting requires 100× more inference FLOPs and roughly 100× more latency than a single greedy decode. While these techniques improve benchmark scores, they don't fundamentally improve the model itself—the underlying single-pass accuracy remains unchanged. The paper aims to improve exactly this quantity.

Supervised Fine-Tuning and Data Augmentation

On the supervised fine-tuning side, prior work had established that fine-tuning on human-annotated chain-of-thought reasoning traces improves mathematical reasoning (Cobbe et al., 2021). However, collecting such data is expensive and doesn't scale easily. The question of how much data is needed and how much return to expect from additional data was not well-characterized in a controlled fashion across model sizes.

For data augmentation, several methods had been proposed that use the model itself to generate training data. Zelikman et al. (2022) introduced STaR (Self-Taught Reasoner), which iteratively generates rationales using the model, filters for correct answers, and fine-tunes on the filtered rationales. This is closely related to RFT but involves multiple rounds of training and generation. Zhu et al. (2023) proposed CoRE, which trains a verifier and uses Monte Carlo Tree Search to guide reasoning path generation for data augmentation—a more complex pipeline than RFT. Ni et al. (2023) used both fully-correct and partially-correct solutions (FCS and PCS) for data augmentation, requiring additional machinery to identify and score partial solutions.

The paper positions RFT as a simpler alternative to these approaches (Section 3.3, Table 3): "Our RFT method is simpler compared to CoRE, since RFT does not require training verifier models and decoding with Monte Carlo Tree Search (MCTS)." The simplicity is not merely aesthetic—it makes RFT more practical and easier to analyze, which is essential for studying scaling relationships cleanly.

Scaling Laws Literature

On the theoretical side, Kaplan et al. (2020) and Hoffmann et al. (2022) established scaling laws for pre-training loss as a function of model size and data quantity. These laws describe how pre-training loss decreases predictably with more compute. However, they focus on next-token prediction loss, not on downstream task performance. Hernandez et al. (2021) studied scaling laws for transfer learning—how pre-training improvements translate to downstream task performance—but this line of work did not specifically investigate mathematical reasoning or the role of fine-tuning data amount.

Critically, prior scaling law work provided no guidance on how augmented data (generated by the model itself rather than human-annotated) scales. The paper's analysis of scaling with respect to "distinct reasoning path amount" (Section 3.3) fills a gap that prior work had not addressed. The relationship between rejection sampling counts and downstream accuracy had not been empirically characterized, leaving practitioners to guess how much to invest in data augmentation.

Where Prior Work Falls Short: A Summary

The paper identifies four specific gaps that motivate its investigation:

1. No systematic comparison of model quality indicators: Prior work loosely refers to "larger models" as better, but the paper observes that LLaMA-7B outperforms larger GPT-3 variants, suggesting parameter count alone is insufficient. The paper empirically shows pre-training loss is a more reliable indicator, but this relationship had not been explicitly demonstrated for math reasoning fine-tuning.

2. No characterization of supervised data scaling for reasoning: While it's intuitive that more data improves fine-tuning, the functional form of this relationship (log-linear) and its interaction with model quality (better models improve less from additional data) were unknown prior to this work.

3. No analytical framework for data augmentation gains: Rejection sampling and related techniques had been shown to work empirically, but the relationship between sampling count, reasoning path diversity, and downstream performance had not been quantified. The paper identifies "distinct reasoning path amount" as the key variable, which was not obvious ex ante—one might have assumed total sample count (including duplicates) would be the relevant factor.

4. No guidance on when augmentation helps most: The finding that RFT provides larger gains for less performant models (LLaMA-7B gains +13.4 points from RFT-U13B vs. LLaMA2-13B gains +5.4 points) is a novel observation with practical implications. This means weaker models have more to gain from self-generated data, which is counterintuitive from a pure model-capability perspective (one might assume stronger models would be better at leveraging additional data).

How This Paper Positions Itself

The paper frames itself as an empirical scaling relationship investigation, not a new method proposal. While RFT is presented as a method, it's a relatively straightforward combination of rejection sampling (well-established from Uesato et al., 2022 and Zhu et al., 2023) with simple deduplication (Algorithm 1). The paper's contribution is not RFT itself but the systematic analysis of how pre-training loss, supervised data amount, and augmented data amount jointly govern reasoning performance.

The paper makes this framing explicit in several ways:

• The title and abstract emphasize scaling relationships, not RFT: "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models" and "we empirically investigate the scaling relationship of factors that influence the math reasoning abilities of a supervised LLM" (Section 1).
• Results are presented as functional relationships: log-linear for supervised data scaling (Figure 3), diminishing returns for augmented data scaling (Figure 4), linear negative correlation for pre-training loss (Figure 2).
• The discussion section (Section 4.2) frames implications in terms of resource allocation tradeoffs, not in terms of a new state-of-the-art method. The paper explicitly computes FLOPs costs (Table 4) and discusses the relative efficiency of pre-training vs. data augmentation, concluding that "the most important thing we should do is to decrease the model's pre-training loss" (Section 4.2).

The paper also positions itself relative to the GPT-4 era of proprietary models. At the time of writing, models like GPT-4 (OpenAI, 2023) and PaLM2 (Anil et al., 2023) were setting new benchmarks on GSM8K but were not open-source. The paper's experiments on LLaMA and LLaMA2 provide the open-source community with a transparent, reproducible analysis of scaling behavior. Table 3 compares RFT results favorably against other open-source methods (ChatGLM2, InternLM, GPT-Neo, GPT-J) while acknowledging the gap with proprietary models like GPT-4 (92.0% with 5-shot ICL vs. 55.4% for LLaMA2-13B with RFT-U13B).

A final positioning point: the paper explicitly distinguishes its goals from test-time compute scaling research. While prior work (Cobbe et al., 2021; Wang et al., 2023; Lightman et al., 2023) focuses on improving performance through additional inference computation, this paper focuses on improving the model through additional training data, producing a model that is better at single-pass inference. This is a different and complementary axis of improvement—one that's more relevant for deployment scenarios where inference-time latency and cost are binding constraints.

3. Technical Approach

This is primarily an empirical scaling analysis paper whose core idea is that three factors—pre-training loss, supervised data amount, and augmented reasoning path diversity—have predictable, measurable relationships with downstream mathematical reasoning accuracy, and that understanding these relationships enables principled resource allocation between pre-training, data collection, and data augmentation.

3.1 Reader Orientation

The paper constructs a framework for predicting and improving the mathematical reasoning performance of a supervised language model. The "system" is not a single trained model but rather an analytical framework and experimental methodology for understanding how much benefit to expect from (a) choosing a model with a certain pre-training quality, (b) collecting a certain amount of human-annotated reasoning data, and (c) generating additional synthetic reasoning data from the model itself via rejection sampling. The core empirical technique is controlled scaling experiments: systematically varying one factor at a time across multiple model sizes and measuring the resulting single-pass accuracy (maj1@1) on GSM8K, revealing functional relationships (linear, log-linear, diminishing returns) that practitioners can use to make resource allocation decisions.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical framework has four major components, each producing insights that feed into the next:

1. Pre-trained Base Models (LLaMA and LLaMA2 families) — a set of transformer language models of varying sizes (7B, 13B, 33B, 65B, 70B parameters) and varying pre-training quality (measured by pre-training loss). These serve as the starting point for all supervised fine-tuning experiments. The key idea is to treat pre-training loss—not parameter count—as the independent variable representing model quality.

2. Supervised Fine-Tuning (SFT) Pipeline — a procedure that takes a base model and a dataset of (question, reasoning path, answer) triples from GSM8K and produces a supervised model via standard next-token-prediction fine-tuning. By varying the amount of training data (from 1/32 to full GSM8K) and measuring accuracy, the paper extracts the functional relationship between data amount and performance for each model.

3. Rejection Sampling Fine-Tuning (RFT) Pipeline — a data augmentation procedure that uses an already-fine-tuned SFT model to generate additional training data. The SFT model samples $k$ candidate reasoning paths per training question, filters to keep only correct answers, deduplicates by distinct equation lists (Algorithm 1), and produces an augmented dataset $D'_\pi$. This augmented dataset is then used to fine-tune the base model from scratch (not the SFT model), producing the RFT model $\pi_{\text{RFT}}$.

4. Multi-Model Aggregation — an extension where rejection samples from multiple SFT models (e.g., LLaMA-7B, LLaMA2-7B, LLaMA-13B, LLaMA2-13B) are combined into a single augmented dataset $D'_{\text{U13B}}$ or $D'_{\text{U33B}}$ (where U stands for "union of models under a certain size"). This exploits the fact that different models generate distinct reasoning paths, increasing diversity without additional human effort.

The flow of information is: base model quality (pre-training loss) → SFT performance at various data amounts → relationship between data and accuracy → SFT model generates augmented data via rejection sampling → augmented data amount and diversity → RFT performance → comparison with SFT baselines → multi-model aggregation → final accuracy gains.

3.3 Roadmap for the Deep Dive

• First, the experimental setup and core metric (maj1@1, GSM8K benchmark), since all scaling relationships are measured through this lens and the choice of metric determines what is being optimized.
• Second, the pre-training loss vs. performance analysis (Figure 2), because pre-training loss serves as the foundational independent variable that conditions all subsequent scaling relationships—they differ depending on how good the base model is.
• Third, the SFT pipeline and the supervised data scaling experiments (Figure 3), since SFT is the baseline against which RFT is measured and the log-linear relationship establishes the return on investment for human-annotated data.
• Fourth, the RFT pipeline in detail—rejection sampling mechanics, deduplication (Algorithm 1), and the scaling experiments with varying sampling count $k$ (Figure 4 and Table 2)—because this is the paper's main methodological contribution and the core mechanism for data augmentation.
• Fifth, the multi-model aggregation procedure (Figure 5 and Figure 6), since it extends RFT beyond a single model's capability and demonstrates that diversity across model families compounds the benefits.
• Sixth, the computational cost analysis (Table 4) as a practical framework for comparing the efficiency of pre-training, SFT, and RFT.

3.4 Detailed, Sentence-Based Technical Breakdown

Experimental Setup and Core Metric

The paper studies mathematical reasoning on the GSM8K benchmark (Cobbe et al., 2021), a dataset of grade-school math word problems requiring multi-step arithmetic reasoning. The training set consists of approximately 7,473 questions (referred to as "7.4K" in Table 5), and the standard test set contains 1,319 questions. Each problem is annotated with a chain-of-thought reasoning path (r) containing natural language reasoning interleaved with arithmetic calculations formatted as <<expression>>result, and a final numerical answer (a). The supervised dataset is formally defined as:

$D = \{q_i, r_i, a_i\}_i$

where $q_i$ is a question string, $r_i$ is a chain-of-thought reasoning path string containing intermediate calculations, and $a_i$ is the ground-truth numerical answer.

What it represents: $D$ is the set of human-annotated training examples where each example consists of a question, a worked solution showing the step-by-step reasoning, and the final answer.

Why this form: The reasoning path r is essential because it provides intermediate supervision—the model learns not just to produce the answer but to produce a reasoning chain that leads to the answer. This is the standard chain-of-thought fine-tuning paradigm established by Cobbe et al. (2021).

The core performance metric throughout the paper is maj1@1, which the authors also call "accuracy." This is the fraction of test set questions for which the model, using greedy decoding (temperature 0, always selecting the most probable next token), produces a final answer that matches the ground truth. Formally:

$\text{accuracy} = \frac{1}{|D_{\text{test}}|} \sum_{i=1}^{|D_{\text{test}}|} \mathbb{1}_{\hat{a}_i = a_i}$

where $\hat{a}_i$ is the model's greedily decoded final answer for question $q_i$, $a_i$ is the ground-truth answer, and $\mathbb{1}$ is the indicator function returning 1 when answers match and 0 otherwise.

What it computes: the fraction of test questions answered correctly in a single forward pass without any test-time sampling, ensembling, or verification.

Why this form: The paper explicitly motivates this choice for deployment relevance (Section 1): single-pass greedy decoding is the setting "closer to online deployment" since it requires minimal inference compute and achieves the lowest latency. Methods that sample multiple reasoning paths and apply majority voting (maj1@100) or verifier reranking improve accuracy but multiply inference cost, making them unsuitable for production systems with tight latency and throughput requirements. The paper does report maj1@100 in several tables for completeness, but the primary scaling analysis is all on maj1@1.

The authors also use few-shot in-context learning (ICL) accuracy as a comparative baseline, specifically 8-shot prompting where 8 example (question, reasoning, answer) triples are prepended to the test question before the model generates. The ICL results for LLaMA and LLaMA2 are taken directly from the original model papers (Touvron et al., 2023a;b) rather than measured by the authors, acknowledging a potential non-uniformity in prompt formatting.

Supervised Fine-Tuning (SFT) protocol. The base SFT procedure takes a pre-trained LLaMA or LLaMA2 model and fine-tunes it on the GSM8K training set using standard next-token prediction loss (only on the reasoning path and answer tokens, not on the question tokens). The hyperparameters are: 3 epochs, batch size 128, peak learning rate $2 \times 10^{-5}$ with a 3% learning rate warmup, using AdamW optimizer (specific betas not reported in the main text). Training uses 8 NVIDIA A100 80GB GPUs for 7B and 13B models, 16 GPUs for 33B, and 32 GPUs for 65B and 70B models. The model is evaluated at the end of training (final epoch) with greedy decoding for maj1@1 and with temperature 0.7 for maj1@100 (sampling 100 times and taking the majority answer). This SFT procedure is the baseline against which RFT improvements are measured.

Why 3 epochs: The authors discovered that for very small data fractions (e.g., 1/32 of GSM8K), the standard 3 epochs produce "very poor results" (Appendix A.2). The issue is that with only ~230 training examples across 3 epochs, the model sees each example only a handful of times, which is insufficient for learning when parameter updates are spread thinly. To address this without introducing confounding factors, the authors search over two epoch settings: 3 epochs (the standard) and 3 / data_fraction epochs (which scales inversely with data size, ensuring the total number of gradient steps remains constant regardless of data amount). They report the better test set result between these two settings. This is an important experimental control: by keeping the optimization budget constant, any performance differences across data amounts can be attributed to data quantity and diversity rather than to differing amounts of training.

Temperature choice for maj1@100. When computing majority-voting accuracy, the model samples with temperature 0.7 (not greedy), following Cobbe et al. (2021). Temperature 0.7 introduces stochasticity into token selection, producing diverse reasoning paths without being so random that generation quality degrades. At temperature 0 (greedy), all 100 samples would be identical, making majority voting meaningless. The maj1@100 metric then aggregates these 100 sampled answers by selecting the most frequent final answer; if no answer appears more than once, the model is considered to have failed (implicitly, since there is no majority).

Pre-Training Loss as the Foundation: Model Quality Indicator

Before analyzing supervised data or augmented data, the paper first establishes what metric best represents a "better" pre-trained model for the purpose of predicting downstream math reasoning performance. The conventional wisdom—that larger models (more parameters) perform better—is contradicted by the observation that LLaMA-7B outperforms larger GPT-3 models on reasoning tasks, even though GPT-3-175B has 25× more parameters.

The paper proposes pre-training loss (the negative log-likelihood of the next token on a held-out validation set of the pre-training corpus) as a more reliable indicator. Pre-training loss captures not just model capacity (parameters) but also training data quantity and quality, training duration, architecture efficiency, and tokenizer quality—all the factors that jointly determine how well the model has learned the statistical patterns of language.

The relationship is analyzed in Figure 2 by plotting ICL (8-shot) accuracy and SFT accuracy against pre-training loss for GPT-3, LLaMA (7B, 13B, 33B, 65B), LLaMA2 (7B, 13B, 70B), and GPT-4. The pre-training losses are taken from each model's original paper and are not directly comparable across model families (different training data, different tokenizers, different loss computation details). Nevertheless, the paper identifies a striking empirical pattern:

"The pre-training losses are approximately negatively linear correlated to the SFT and ICL accuracy during the given pre-training loss interval."

This means that as pre-training loss decreases (the model gets better at predicting text), both ICL and SFT accuracy on GSM8K increase in a roughly linear fashion. The relationship is negative linear because lower loss means better performance, so the slope is negative when plotting accuracy vs. loss.

However, the paper is careful to note caveats. First, the relationship cannot hold exactly across all loss values because accuracy is bounded in [0, 1] while loss is unbounded below—if extended far enough, the linear relationship would predict accuracy > 1 or < 0. The paper suggests using $-\log(\text{acc})$ as the dependent variable would be more theoretically sound, but the empirical linear relationship in the observed interval is a practical approximation.

Second, the slopes differ: the ICL line is steeper than the SFT line. This means that as pre-training loss decreases (model quality improves), ICL improves faster than SFT. Equivalently, SFT provides a larger boost over ICL for worse models, and this boost diminishes for better models. The paper states:

"SFT outperforms ICL consistently, while the improvements diminish when the pre-training loss is lower."

This is the first empirical evidence for a motif that recurs throughout the paper: better pre-trained models gain less from additional supervision. The interpretation is that stronger models have already internalized more reasoning capabilities during pre-training, so the supervised dataset provides less incremental signal—the model is closer to its ceiling for this task.

For practical resource allocation, this finding implies that pre-training loss is the single most informative number to look at when selecting a base model for reasoning fine-tuning, more so than parameter count. It also suggests that reducing pre-training loss (through longer training, more data, or better architectures) will predictably improve downstream reasoning.

Supervised Data Scaling: The Log-Linear Relationship

Having established pre-training loss as the foundational independent variable, the paper turns to the second factor: how much supervised data to use. The key question is: if we have a fixed pre-trained model and we double the amount of human-annotated GSM8K-style training data, how much does accuracy improve? Does this improvement depend on the model quality?

Experiment design. The authors randomly downsample the GSM8K training set to fractions of {1, 1/2, 1/4, 1/8, 1/16, 1/32} of the full 7,473 examples. For each fraction and each model size, they perform SFT and measure maj1@1. The epoch count is either 3 or 3 / data_fraction (whichever yields better test accuracy), as described above. Results are shown in Figure 3 for LLaMA-7B, LLaMA2-7B, LLaMA-13B, LLaMA2-13B, LLaMA-33B, LLaMA-65B, and LLaMA2-70B.

The core finding: log-linear scaling. Figure 3 plots accuracy against data amount on a log-scale x-axis. The curves are approximately straight lines across the range from 1/32 to 1/2 (or 1 for some models), indicating:

"The model performance has a log-linear relation versus data amount. When the data amount doubles, the performance increases by a unit."

This means that each doubling of training data produces a constant absolute improvement in accuracy (roughly constant for a given model), but the absolute data required for each additional point of accuracy grows exponentially. Going from 1/32 to 1/16 of the data (adding ~230 examples) produces about the same accuracy gain as going from 1/2 to 1 (adding ~3,700 examples). This is the hallmark of log-linear (or equivalently, power-law) scaling in the relevant range.

Interaction with model quality. The paper identifies a crucial interaction: better models benefit less from additional supervised data. The log-linear curves for LLaMA-33B and LLaMA-65B are flatter (lower slope) than those for LLaMA-7B and LLaMA-13B. When the data amount doubles, a weaker model gains more accuracy points than a stronger model. This is stated explicitly:

"Better model benefits less when supervised data amount doubles."

There's a second interaction: better models need more data to surpass their own ICL performance. Looking at Figure 3, the ICL accuracy for each model can be read off the left side (at data amount = 0, conceptually). The SFT accuracy with very little data (1/32) is sometimes below the ICL accuracy, meaning fine-tuning on insufficient data can actually hurt—the model partially forgets its pre-trained reasoning ability without gaining enough task-specific signal. Better models, having higher ICL accuracy, require more fine-tuning data to reliably exceed that baseline.

The practical implication is straightforward but powerful: for weaker models, collecting more human-annotated data yields high returns; for stronger models, the marginal benefit of additional data is smaller, and effort may be better spent on pre-training improvements.

Rejection Sampling Fine-Tuning (RFT): The Core Data Augmentation Pipeline

The SFT scaling analysis shows that more data helps, but human annotation is expensive. The RFT pipeline provides a way to generate additional training data automatically using the SFT model itself, without any human effort. The key insight is that an SFT model can generate many candidate reasoning paths for each training question, and among these, the ones that happen to produce the correct answer can be harvested as additional training examples.

Step 1: Candidate generation via rejection sampling. Starting from an SFT model $\pi$ (trained on the full GSM8K training set), the authors generate $k$ candidate reasoning paths and answers for each training question $q_i$. Generation uses temperature 0.7, which introduces randomness to produce diverse candidates. For each question, the model is prompted zero-shot (just the question text; the model has internalized chain-of-thought formatting from SFT) and generates a complete response containing reasoning steps and a final answer.

Step 2: Correctness filtering. Each of the $k$ generated candidates is checked for correctness. Two criteria are applied:

1. Answer matching: the extracted final answer must equal the ground-truth answer $a_i$. The paper uses a standard answer extraction procedure that parses the text following the #### marker (the conventional GSM8K format where the final answer appears after four hash marks).

2. Calculation verification via Python evaluation: each arithmetic expression enclosed in <<expression>> tags in the reasoning path is evaluated using Python. If any calculation is incorrect (the evaluated result does not match the reported result in the reasoning path), the candidate is discarded even if the final answer matches. This catches cases where the model makes an arithmetic error but coincidentally arrives at the right answer through a subsequent mistake—these examples would teach incorrect calculation patterns.

The filtering produces, for each training question, a set $R_q$ of reasoning paths that are both mathematically correct (verified by Python) and produce the correct final answer.

Step 3: Deduplication by distinct equation lists (Algorithm 1). The authors observe that many of the correct reasoning paths are near-duplicates—they follow the same calculation steps with only minor lexical variations (e.g., "Weng earns 12/60 = $0.2 per minute" vs. "Weng earned 12/60 =$0.2 an hour"). Training on duplicates provides little additional information and wastes compute. The deduplication procedure works as follows:

For each question $q$, the algorithm extracts the equation list from each reasoning path. An equation list is the sequence of arithmetic expressions in the order they appear, stripped of natural language and white space. For example, the reasoning path:

"Weng earns 12 / 60 = $0.2 per minute. So, Weng earned 50 * 0.2 = $10."

has the equation list [12/60=0.2, 50*0.2=10]. Two reasoning paths are considered to have the "same calculation process" if their equation lists are identical in both the expressions used and the order they appear. Note that different orders of elements within an equation (e.g., 3+4=7 vs. 4+3=7) or different orders of equations (e.g., computing A then B vs. B then A) are considered distinct—the paper explicitly states this is helpful because it teaches the model that these orders can be exchanged.

For each distinct equation list, the algorithm selects exactly one reasoning path to include in the augmented dataset. When multiple candidates share the same equation list, the algorithm chooses the one that is most dissimilar to all already-selected paths, measured by total Levenshtein (edit) distance. The rationale is to maximize lexical diversity even among paths with the same calculation structure, which the authors hypothesize improves generalization:

"The idea comes from we want diverse reasoning paths for better generalization."

The result is an augmented dataset:

$D'_\pi = D \cup \{q_i, r_{ij}, a_i\}_{i,j}$

where $D$ is the original GSM8K training set, and the additional triples $\{q_i, r_{ij}, a_i\}_{i,j}$ are the selected (deduplicated) correct reasoning paths generated by the SFT model $\pi$. The subscript $j$ indexes over distinct reasoning paths for question $i$. This augmented dataset contains the original human-annotated examples plus the machine-generated verified-correct examples.

Step 4: Fine-tuning the base model on the augmented dataset. Critically, the augmented dataset $D'_\pi$ is used to fine-tune the original pre-trained base model $\rho$ (not the SFT model $\pi$), producing the RFT model $\pi_{\text{RFT}}$. This is a design choice with important implications: the model being trained has never seen the augmented data (it was generated by a different model, $\pi$, after SFT), so the augmented data provides genuinely new training signal rather than just repeating what the model already knows. Fine-tuning uses the same hyperparameters as SFT (3 epochs, batch size 128, learning rate $2 \times 10^{-5}$, 3% warmup).

The final RFT model is evaluated identically to the SFT model: greedy decoding for maj1@1, temperature 0.7 for maj1@100.

Why this procedure works (the paper's hypothesis). The authors argue that RFT works because it provides multiple distinct reasoning paths for the same question, which teaches the model that there are different valid ways to solve the same problem. The original GSM8K dataset has exactly one reasoning path per question, which may lead the model to memorize a specific solution strategy rather than learning general reasoning principles. By exposing the model to 5-6 different correct solution paths per question (the average for $k=100$), RFT encourages the model to learn the underlying mathematical relationships rather than surface-level patterns.

This hypothesis is supported by the case studies in Table 7 and the histogram analysis in Figure 7 (showing RFT models generate more diverse solutions at test time). It also explains the ablation finding from Appendix D.2, where training a "revision model" that takes incorrect paths and revises them to correct ones worked better when the incorrect paths were selected to be maximally different (by Levenshtein distance) from the correct target—diverse reasoning paths are more informative for learning.

RFT Scaling: The Relationship Between Sampling Count $k$ and Performance

With the RFT pipeline established, the paper investigates the scaling question: how does the number of candidate samples $k$ influence downstream RFT accuracy? This is analogous to the supervised data scaling analysis but for augmented (self-generated) data.

Experiment design. For each base model (LLaMA-7B, 7B-2, 13B, 13B-2), the authors run RFT with $k \in \{1, 3, 6, 12, 25, 50, 100\}$ and also a "no dedup" variant at $k=100$ where all correct reasoning paths are kept without distinct-equation-list filtering. The SFT model used for generation is the same model family and size (self-generation). For 33B, results are reported only for $k=100$ due to computational expense. The results are in Figure 4 and Tables 1 and 2.

Key finding: distinct reasoning paths, not total sample count, drives improvement. Figure 4 plots RFT accuracy against $\log_2(k)$. The curves generally increase with $k$ but with diminishing returns—doubling $k$ at higher values produces smaller gains than doubling at lower values. The "no dedup" variant (all correct paths kept) performs similarly to or slightly worse than the deduplicated version at $k=100$, even though it contains many more total training examples (approximately 8× more, since on average 53.3 correct paths per question are generated but only 5.25 are distinct for LLaMA-7B). This is a crucial result: it shows that the diversity of reasoning paths, not the raw count of training examples, is what matters for RFT performance.

The paper formalizes this by reporting "distinct paths per question" for each $k$ in Table 2. For LLaMA-7B:

The distinct path count grows sublinearly with $k$—doubling $k$ from 50 to 100 increases distinct paths from 3.94 to 5.25 (only a 33% increase, not 100%). This sublinear growth explains the diminishing returns in accuracy: as $k$ increases, it becomes harder and harder to find genuinely new reasoning paths that the model hasn't already generated. Most additional samples are duplicates or near-duplicates of existing paths.

The paper explicitly connects this to the supervised data scaling result: "Doubling reasoning paths should improve less than doubling training samples since obtaining different reasoning paths does not obtain any new questions." In the supervised case, doubling the data adds both new reasoning paths and new questions. In the RFT case, doubling $k$ adds only new reasoning paths for existing questions, which is a less powerful form of augmentation.

Model quality interaction: better models get less from RFT. Looking at Table 1, the improvement from SFT to RFT ($k=100$) is:

• LLaMA-7B: 35.9 → 41.7 (+5.8)
• LLaMA2-7B: 41.6 → 47.5 (+5.9)
• LLaMA-13B: 43.0 → 49.1 (+6.1)
• LLaMA2-13B: 50.0 → 54.8 (+4.8)
• LLaMA-33B: 54.6 → 54.5 (−0.1)

For the 33B model, RFT actually does not improve performance. The reason is visible in Table 2: LLaMA-33B-SFT generates an average of 88.7 correct paths per question but only 2.78 distinct paths per question. The model has so thoroughly memorized the training set (or converged to a very narrow solution strategy) that it generates the same reasoning path over and over with minor variations. Despite producing many correct answers, it provides almost no new reasoning diversity for RFT augmentation.

The authors attempt a temperature increase to 1.0 for the 33B model, which yields 82.4 correct paths and 4.77 distinct paths per question—more diverse but still less than the 7B and 13B models (which achieve 5.19–5.29 distinct paths at temperature 0.7). They acknowledge that a systematic temperature sweep might find a setting that produces sufficient diversity, but this is computationally expensive and not the paper's focus. The key insight is that better models can be worse at generating diverse training data because they overfit the training distribution more tightly.

Why does $k=3$ already help? At $k=3$, RFT adds only about 1.44 distinct paths per question (only about 0.3 more than the original 1 path from GSM8K), yet it provides a consistent +2 point improvement over SFT across all models (Figure 4). This suggests that even a small amount of reasoning path diversity is valuable—the model benefits from seeing that there is more than one way to solve a problem, even if the alternative paths are closely related to the original.

Multi-Model Aggregation: Combining Rejection Samples Across Models

The single-model RFT results show diminishing returns from increasing $k$ because a single SFT model has limited reasoning diversity. The natural extension: what if we combine rejection samples from multiple different SFT models? Different models—even of the same size but from different training runs (LLaMA-7B vs. LLaMA2-7B)—may generate different reasoning strategies because they have learned different patterns during pre-training.

The aggregation procedure. The authors define two aggregated datasets:

$D'_{\text{U13B}} = D'_{\text{7B}} \oplus D'_{\text{7B2}} \oplus D'_{\text{13B}} \oplus D'_{\text{13B2}}$

$D'_{\text{U33B}} = D'_{\text{U13B}} \oplus D'_{\text{33B}}$

where $D'_{\text{7B}}$ is the augmented dataset from LLaMA-7B-SFT with $k=100$, $D'_{\text{7B2}}$ is from LLaMA2-7B-SFT, etc. The symbol $\oplus$ denotes an aggregation process: all reasoning paths from all component datasets are combined into one pool, then Algorithm 1 (deduplication by distinct equation list) is applied to the combined pool. This means that if LLaMA-7B and LLaMA2-7B both generate the same calculation process for a question, only one copy is kept (the most diverse one by Levenshtein distance). The aggregation removes redundancy across models, keeping only reasoning paths that are genuinely different from paths already contributed by other models.

For $k=100$ per model, $D'_{\text{U13B}}$ contains approximately 104K training examples (Table 5) and $D'_{\text{U33B}}$ contains approximately 110K examples. Compare this with the original GSM8K training set (7.4K examples) and the single-model RFT at $k=100$ (approximately 47K examples for LLaMA-7B). The aggregation approximately doubles the training set size compared to single-model RFT, but more importantly, it approximately doubles the reasoning path diversity.

Table 2 reports the average distinct paths per question for the aggregated datasets: $D'_{\text{U13B}}$ averages 12.84 distinct paths per question, and $D'_{\text{U33B}}$ averages 13.65. This is more than double the 5.25–5.29 achieved by any single 7B or 13B model, confirming that different models contribute non-overlapping reasoning strategies.

Figure 6 provides a Venn diagram breaking down which models contribute unique reasoning paths to $D'_{\text{U33B}}$. The proportions of reasoning paths that are exclusively found in one model's rejection samples are: LLaMA-7B contributes 14.0%, LLaMA2-7B contributes 14.9%, LLaMA-13B contributes 14.2%, LLaMA2-13B contributes 15.5%, and LLaMA-33B contributes only 6.5%. The remaining 34.9% of reasoning paths are shared by at least two models. This is striking: each of the four 7B and 13B models contributes roughly equal amounts of unique reasoning diversity, while the 33B model contributes substantially less. This quantitatively demonstrates that the 33B model, despite being larger and having higher accuracy, has less diverse reasoning than smaller models.

Why aggregation works better than increasing $k$ for a single model. The Venn diagram explains this: doubling $k$ for LLaMA-7B from 100 to 200 would mostly generate more copies of the same 5.25 reasoning paths, whereas aggregating from four different models introduces genuinely new reasoning strategies that were never generated by LLaMA-7B alone. The different pre-training runs have induced different inductive biases, leading to different solution strategies for the same problems.

Results of multi-model RFT (Figure 5 and Table 5). Fine-tuning on $D'_{\text{U13B}}$ produces:

• LLaMA-7B: 49.3% (vs. 35.9% SFT, +13.4)
• LLaMA2-7B: 50.3% (vs. 41.6% SFT, +8.7)
• LLaMA-13B: 52.1% (vs. 43.0% SFT, +9.1)
• LLaMA2-13B: 55.4% (vs. 50.0% SFT, +5.4)
• LLaMA-33B: 56.5% (vs. 54.6% SFT, +1.9)
• LLaMA-65B: 59.0% (vs. 59.3% SFT, −0.3)

The pattern is clear: RFT provides larger gains for weaker models. LLaMA-7B gains +13.4 points while LLaMA2-13B gains only +5.4, and 65B actually regresses slightly (though this may be within noise). The paper explains:

"We can assume with sufficient supervised data amounts, the performance indicator should be the model size but not the pre-training losses."

In other words, when all models are given enough diverse training data (through multi-model RFT), the accuracy differences between models of the same size but different pre-training quality (e.g., LLaMA-7B vs. LLaMA2-7B) largely disappear. The remaining performance gap is due to model capacity (parameter count), not pre-training loss. Specifically, with RFT-U13B, LLaMA-7B achieves 49.3% and LLaMA2-7B achieves 50.3%—nearly identical. But LLaMA-13B achieves 52.1% and LLaMA2-13B achieves 55.4%, both higher than the 7B models. This suggests that once data is plentiful, capacity becomes the binding constraint.

Adding the 33B model to the aggregation does almost nothing. Comparing $D'_{\text{U13B}}$ and $D'_{\text{U33B}}$ in Table 5, the performance differences are minimal: LLaMA-7B goes from 49.3% to 49.1% (essentially unchanged), LLaMA2-7B goes from 50.3% to 51.2% (+0.9), LLaMA-13B goes from 52.1% to 51.4% (−0.7), LLaMA2-13B goes from 55.4% to 55.3% (−0.1), and LLaMA-33B goes from 56.5% to 57.9% (+1.4). The only meaningful gain is for the 33B model itself, which finally benefits from RFT when trained on data generated by other models rather than its own. This directly supports the earlier finding: the 33B-SFT model generates low-diversity reasoning paths, so adding its samples to the aggregation contributes little (only 6.5% unique paths in the Venn diagram).

Why the 65B and 70B models don't benefit. Table 5 reports that 65B and 70B models show minimal improvement from RFT-U13B and RFT-U33B. The paper explains:

"The reason can be better models benefit less from the supervised sample amounts while it has learnt more reasoning ability during pre-training."

This is the same pattern observed throughout: better pre-trained models are closer to their reasoning ceiling on GSM8K, so additional training data—whether human-annotated or machine-generated—provides less incremental benefit. The 65B and 70B models likely have near-saturated GSM8K performance for their capacity; further gains would require improving pre-training (e.g., training on math-heavy corpora as in Lewkowycz et al., 2022) rather than adding more fine-tuning data.

Computational Cost Analysis: Pre-Training vs. SFT vs. RFT

Section 4.2 provides a practical framework for comparing the efficiency of different improvement strategies by estimating the FLOPs (floating-point operations) required for each. This analysis is detailed in Table 4 and Appendix E.

Pre-training FLOPs are taken from the LLaMA and LLaMA2 papers (Touvron et al., 2023a;b):

$\text{FLOPs}_{\text{pretrain}} \approx 6 \times N \times D_{\text{pretrain}}$

where $N$ is the number of non-embedding model parameters and $D_{\text{pretrain}}$ is the number of pre-training tokens. For LLaMA-7B, this is approximately $4.2 \times 10^{22}$ FLOPs. For LLaMA2-7B, which was trained on more data, it's approximately $8.4 \times 10^{22}$ FLOPs—roughly double.

SFT FLOPs are estimated using the same formula but applied to the fine-tuning data:

$\text{FLOPs}_{\text{SFT}} \approx 6 \times N \times T_{\text{SFT}}$

where $T_{\text{SFT}}$ is the total number of tokens in the SFT training set (GSM8K questions + reasoning paths) multiplied by the number of epochs. For LLaMA-7B, SFT costs approximately $1.7 \times 10^{17}$ FLOPs, which is roughly $4 \times 10^{-6}$ (or 0.0004%) of the pre-training FLOPs. In GPU hours (on NVIDIA A100 80GB): pre-training takes 82,000 GPU hours for LLaMA-7B, while SFT takes 0.6 GPU hours—SFT is essentially free compared to pre-training.

RFT inference FLOPs (generating the augmented data) are estimated using a forward-pass FLOP formula that accounts for the KV-cache during autoregressive decoding (Appendix E, Equations 3-9). For LLaMA-7B with $k=100$ on 7,473 training questions, inference costs approximately $1.4 \times 10^{18}$ FLOPs, about 8× the SFT cost but still only $3.3 \times 10^{-5}$ of pre-training. In GPU hours: 10 hours for inference on 7B, scaling up to 4,500 hours for 33B (due to the larger model's higher per-token cost and the need for distributed inference with DeepSpeed ZeRO-3).

RFT training FLOPs (fine-tuning on the augmented dataset) are comparable to SFT—just a few GPU hours for 7B models.

The key takeaways from this analysis:

1. RFT is extremely cheap compared to pre-training. The combined cost of SFT + RFT inference + RFT training is about $3 \times 10^{18}$ FLOPs for 7B, which is less than 0.01% of the pre-training cost. This means RFT should almost always be applied—the cost is negligible and the gains are reliable (especially for weaker models).

2. Pre-training improvements are expensive but fundamental. Going from LLaMA-7B to LLaMA2-7B adds $4.2 \times 10^{22}$ FLOPs (doubling the pre-training cost) and yields a roughly 2-point improvement in RFT-U33B accuracy (49.1% vs. 51.2%). Going from LLaMA-7B to LLaMA-13B adds $3.6 \times 10^{22}$ FLOPs and yields a 2.3-point improvement (49.1% vs. 51.4%). These cost-to-benefit ratios are orders of magnitude worse than RFT (which can add 13 points for 1/10,000th the cost).

3. The diminishing returns of RFT with model scale. RFT inference cost scales with model size (larger models cost more per generated token). At 33B, RFT inference costs 4,500 GPU hours vs. 530,000 GPU hours for pre-training—still a small fraction, but no longer negligible. And for 33B and 65B, the benefits are small or zero, making the investment harder to justify.

This cost analysis leads to the paper's concluding recommendation:

"Since you can obtain an RFT model without too much effort (compared to pre-training), then the most important thing we should do is to decrease the model's pre-training loss."

In other words: RFT is a cheap, effective bolt-on for weaker models, but pre-training quality remains the ultimate determinant of reasoning ability for strong models. The practical strategy is to always apply RFT (it's nearly free for models up to 13B), but invest the bulk of resources in pre-training improvements because they benefit all downstream tasks, not just math reasoning.

4. Key Insights and Innovations

Innovation 1: Pre-Training Loss as the True Performance Indicator — Not Parameter Count

The paper's most fundamental conceptual move is rejecting parameter count as the primary metric for model quality in favor of pre-training loss. Prior to this work, the dominant discourse in the scaling community treated model size (parameter count) as the de facto measure of capability—"larger models are better" was an implicit operating assumption, reinforced by the scaling laws literature (Kaplan et al., 2020) and by practical experience with model families like GPT-3 (Brown et al., 2020). The paper destabilizes this assumption with a simple but powerful observation: LLaMA-7B outperforms GPT-3-175B on GSM8K despite having 25× fewer parameters. Parameter count alone fails to predict reasoning performance.

The innovation is not the observation itself (the field already knew LLaMA was strong), but rather elevating pre-training loss to a first-class analytical variable and systematically demonstrating its predictive power across both SFT and ICL settings (Figure 2). This is a conceptual reframing: pre-training loss subsumes parameter count, training data quantity, architecture quality, and tokenizer efficiency into a single scalar that directly measures how well the model has compressed its training distribution. The paper shows that this scalar has an approximately negative linear relationship with downstream math reasoning accuracy (in the observed interval), making it a more reliable decision tool for practitioners choosing between base models.

The significance extends beyond math reasoning. If pre-training loss is the better performance indicator for reasoning, it likely is for many other capabilities. This shifts the practical question from "how many parameters should I use?" to "what's the pre-training loss?"—a more actionable framing because it accounts for training data quality and duration, not just architectural scale. It also explains why LLaMA2 models outperform LLaMA1 models of the same parameter count: they were trained longer, achieving lower pre-training loss (1.75 vs. 1.80 for 7B, per Table 1). The innovation is an analytical reframing, not a new method, but it changes how practitioners should evaluate and compare foundation models.

Innovation 2: The Log-Linear Relationship Between Supervised Data and Reasoning Performance, With Model-Quality-Dependent Diminishing Returns

The paper provides the first controlled empirical characterization of how supervised fine-tuning performance scales with data quantity for mathematical reasoning, revealing a log-linear functional form with model-quality-dependent slopes. Before this work, the field knew that more supervised data helps, but the quantitative relationship—how much improvement to expect from doubling the training set, and whether that depends on the base model's quality—was unknown. Practitioners made data collection decisions based on intuition, not empirical scaling curves.

The finding itself is clean: accuracy increases linearly with the logarithm of data amount across the range from 1/32 to 1 (the full GSM8K training set), meaning each doubling of data produces roughly constant absolute accuracy gains for a given model (Figure 3). This is a power-law (or log-linear) relationship in the relevant regime—predictable and extrapolatable. More importantly, the slope of this relationship depends on model quality: better pre-trained models (lower loss) have flatter curves, meaning they benefit less from additional supervised data. LLaMA-7B's curve is steeper than LLaMA-33B's, which is steeper than LLaMA-65B's.

This is a diagnostic insight with direct practical implications. It tells practitioners that collecting more human-annotated data yields the highest return for weaker models—if you're starting from LLaMA-7B, investing in a larger annotation budget makes sense. If you're starting from LLaMA-65B, additional annotation provides marginal gains, and resources are better spent on pre-training improvements. The insight also explains a phenomenon that would otherwise be puzzling: better models sometimes need more fine-tuning data to outperform their own few-shot ICL performance (Section 3.2). Because better models have higher ICL accuracy, the SFT curve must climb higher to surpass it, and the flatter slope means it takes more data to get there.

Conceptually, this finding connects to a broader principle that recurs throughout the paper: supervision becomes less valuable as pre-training quality improves, because the model has already internalized more of the target capability during pre-training. This is not a methodological innovation but rather an empirical scaling law for reasoning fine-tuning—analogous in spirit to the pre-training scaling laws of Kaplan et al. (2020) and Hoffmann et al. (2022), but operating at the fine-tuning stage and on a downstream task metric rather than next-token loss.

Innovation 3: Distinct Reasoning Path Diversity as the Mechanism Behind Data Augmentation Gains

The paper's most intellectually distinctive contribution is identifying distinct reasoning path count—not total sample count—as the critical variable governing how much rejection sampling fine-tuning (RFT) improves performance. This is not an obvious finding ex ante. When the paper shows that RFT with $k=100$ (deduplicated, ~47K examples) matches or exceeds RFT with $k=100$ without deduplication (~400K examples, Figure 4), it demonstrates that raw data quantity is not the operative factor—reasoning path diversity is. The paper quantifies this diversity via Table 2, showing that distinct paths per question grow sublinearly with sampling count $k$ (from 1.17 at $k=1$ to 5.25 at $k=100$ for LLaMA-7B), and that this sublinear growth directly explains the diminishing returns of increased sampling.

This is a mechanistic insight, not just a performance result. Prior work on rejection sampling for reasoning (Zelikman et al., 2022; Zhu et al., 2023; Ni et al., 2023) demonstrated that self-generated data could improve models, but none isolated what property of the generated data drives the improvement. Is it the quantity of additional examples? The correctness filtering? The fact that the model generates data at its own capability level? The paper's deduplication ablation (Figure 4) cleanly separates these hypotheses: keeping all correct paths (maximizing quantity) performs no better than keeping only distinct paths (maximizing diversity), while requiring much more training compute. The variable that matters is how many different ways of solving each problem the model sees during training.

The Venn diagram analysis (Figure 6) extends this insight to multi-model aggregation. Different models—even of the same parameter count but from different pre-training runs (LLaMA-7B vs. LLaMA2-7B)—contribute non-overlapping reasoning strategies, with each model providing roughly 14-15% of the total unique paths in $D'_{\text{U33B}}$. This explains why aggregating across models is more effective than scaling $k$ for a single model: you access genuinely different solution strategies that a single model would never generate, no matter how many times you sample from it. Conversely, the LLaMA-33B model contributes only 6.5% unique paths, directly explaining why adding its data to the aggregation provides negligible benefit—the 33B model has so thoroughly overfit the training distribution that it generates very little reasoning diversity.

This insight has architectural implications for self-improvement pipelines. It suggests that future data augmentation methods should prioritize reasoning path diversity over raw throughput—for instance, by using diverse generation temperatures, multiple model checkpoints, or explicit diversity-promoting objectives. It also provides a diagnostic for when data augmentation will work: measure the distinct-path-per-question ratio of your generation model. If it's low (like LLaMA-33B's 2.78), augmentation will likely fail regardless of scale.

Innovation 4: Better Pre-Trained Models Produce Less Diverse Augmentation Data — The Paradox of Capability

The paper uncovers a paradoxical empirical finding with significant implications for self-improvement research: better pre-trained models are worse at generating diverse training data for self-augmentation. This is not a design flaw of RFT but an emergent property of how pre-training quality interacts with fine-tuning.

The evidence is stark in Table 2. LLaMA-7B (pre-training loss 1.80, SFT accuracy 35.9%) generates an average of 5.25 distinct reasoning paths per question at $k=100$. LLaMA-33B (pre-training loss 1.62, SFT accuracy 54.6%) generates only 2.78 distinct paths per question—nearly half the diversity—despite generating far more correct answers overall (88.7 correct paths per question vs. 53.3 for 7B). The explanation is that stronger models converge more tightly to the training distribution, internalizing the specific solution strategies in the human-annotated data and reproducing them with high fidelity but low variance. They "overfit the training set and have difficulty generating more diverse paths on the training set questions" (Section 3.3). The authors attempted to mitigate this by increasing temperature (from 0.7 to 1.0 for 33B), which improved diversity modestly (to 4.77 distinct paths) but still didn't match the 7B and 13B models.

This is a negative result with diagnostic value, not a methodological breakthrough. It identifies a fundamental tension: the very property that makes a model good at solving math problems (tight fit to the training distribution) makes it bad at generating diverse training data for self-improvement. This explains why RFT provides the largest gains for the weakest models (+13.4 for LLaMA-7B with RFT-U13B) and essentially no gain for the strongest models (−0.3 for LLaMA-65B). Strong models are already near their reasoning ceiling given their training data distribution; adding more data from the same distribution—especially data that lacks diversity—provides no new signal.

The implication for self-improvement research is cautionary. The dominant paradigm of "use the model to generate data, then train on it" (STaR, ReST, self-play) implicitly assumes that the model can generate usefully diverse data. This paper shows the assumption breaks for strong models that have already saturated the diversity of their learned strategies. Self-improvement loops may hit a ceiling not because the optimization fails, but because the data generation process loses diversity as the model improves. This is a fundamentally different failure mode from reward hacking or distribution collapse, and it suggests that future self-improvement methods need explicit mechanisms for maintaining reasoning diversity (e.g., using ensembles of weaker models for generation, as the multi-model aggregation does).

Innovation 5: The FLOPs-Normalized Comparison Framework for Pre-Training vs. Data Augmentation Tradeoffs

The paper contributes a resource-theoretic framework for comparing the efficiency of different strategies for improving reasoning performance: pre-training, supervised data collection, and rejection sampling data augmentation. While Section 4.2 and Table 4 present this as a practical cost analysis, the innovation is conceptual: it provides a common currency (FLOPs and GPU hours) for comparing qualitatively different improvement strategies that are usually discussed in isolation.

The framework reveals a clear hierarchy of cost-effectiveness. SFT costs approximately $10^{-5}$ of pre-training FLOPs; RFT costs approximately $10^{-4}$ (for inference + training). Both are essentially free compared to pre-training, yet RFT can improve accuracy by up to 13 points on weaker models. This leads to the paper's central prescriptive insight: "Since you can obtain an RFT model without too much effort (compared to pre-training), then the most important thing we should do is to decrease the model's pre-training loss" (Section 4.2). The reasoning is subtle: RFT is so cheap that it should always be applied, meaning the marginal decision is about pre-training investment, which dominates the total cost.

But the framework also reveals diminishing returns at scale. RFT inference for LLaMA-33B costs 4,500 GPU hours and provides negligible improvement; for 65B models, the inference cost is substantial and the benefit is zero. This is not captured by a simple "RFT is cheap" narrative—the cost-benefit ratio degrades with model scale because (a) inference cost scales with model size, and (b) benefit shrinks due to reduced reasoning diversity (Innovation 4). The FLOPs analysis makes this degradation quantitative and actionable.

This framework is not a deep theoretical contribution, but it is methodologically important for a field where different research communities (pre-training, fine-tuning, data augmentation) rarely compare their approaches on a common cost basis. By presenting their scaling findings alongside explicit FLOPs calculations, the authors enable practitioners to make principled resource allocation decisions rather than relying on intuition or isolated benchmark comparisons. It also sets a standard for future work: any new data augmentation method should report not just accuracy gains but also the computational cost of generating the augmented data, normalized against the cost of alternative strategies (more pre-training, more supervised data).

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the GSM8K benchmark (Cobbe et al., 2021), consisting of grade-school math word problems requiring multi-step arithmetic reasoning. The training set contains approximately 7,473 questions (referred to as "7.4K" in tables), and the standard test set contains 1,319 questions. Each problem is annotated with a chain-of-thought reasoning path containing intermediate calculations formatted as <<expression>>result and a final numerical answer. The paper uses the standard GSM8K split without modification.

• Base model(s). Experiments use the LLaMA (Touvron et al., 2023a) and LLaMA2 (Touvron et al., 2023b) model families across multiple scales: 7B, 13B, 33B, 65B (LLaMA only), and 70B (LLaMA2 only) parameters. The GPT-3 family (Brown et al., 2020) and GPT-4 (OpenAI, 2023) appear only in Figure 2 as pre-training loss reference points for the ICL/SFT comparison. The LLaMA families are chosen because they represent the strongest open-source models available at the time of writing, enabling reproducible analysis across a range of pre-training qualities (as measured by pre-training loss from 1.80 for LLaMA-7B to 1.62 for LLaMA-33B, per Table 1). The paper argues these models are "representative of the capabilities of many contemporary LLMs" in the open-source ecosystem.

• Metrics. The primary metric throughout is maj1@1 (also called "accuracy"), defined as the fraction of test set questions for which the model's greedily decoded final answer (temperature 0) matches the ground-truth answer. Formally: $\text{accuracy} = \frac{1}{|D_{\text{test}}|} \sum_{i=1}^{|D_{\text{test}}|} \mathbb{1}_{\hat{a}_i = a_i}$ where $\hat{a}_i$ is the model's single forward-pass prediction and $a_i$ is the ground truth. The secondary metric maj1@100 (majority voting accuracy) is reported for completeness in several tables: the model samples 100 reasoning paths at temperature 0.7, extracts the final answer from each, and selects the most frequent answer as the prediction. This paper's primary focus is maj1@1 because it corresponds to the single-pass deployment setting the authors care about—improving the model itself rather than ensembling at test time. In-context learning (ICL) accuracy uses 8-shot prompting and is taken from the LLaMA and LLaMA2 original papers rather than re-measured.

• Baselines. The paper compares against several established approaches:

  • SFT (Supervised Fine-Tuning): The model fine-tuned on the full GSM8K training set with standard next-token prediction loss. This is the primary baseline for all RFT comparisons. SFT accuracy varies by model size: 35.9% (7B), 41.6% (7B-2), 43.0% (13B), 50.0% (13B-2), 54.6% (33B), 59.3% (65B), 63.2% (70B-2), per Table 5.
  • ICL (In-Context Learning, 8-shot): Few-shot prompting without any fine-tuning, with results taken from the original LLaMA and LLaMA2 papers (Touvron et al., 2023a;b). ICL accuracy: 11.0% (7B), 14.6% (7B-2), 17.8% (13B), 28.7% (13B-2), 35.6% (33B), 50.9% (65B), 56.8% (70B-2), per Table 5.
  • GPT-3 SFT: Results from Cobbe et al. (2021), achieving 34.0% on GSM8K with GPT-3-175B SFT. Used as a reference point to show LLaMA-7B SFT (35.9%) already outperforms a much larger model family.
  • Proprietary LLMs: GPT-4 achieves 92.0% with 5-shot ICL; PaLM2 achieves 80.7% with 8-shot ICL; PaLM-540B achieves 56.5% with 8-shot ICL; Chinchilla-70B achieves 43.7% with 5-shot ICL and 58.9% with SFT (all from their respective papers, compiled in Table 3).
  • Open-source baseline methods: GPT-Neo-2.7B with FCS+PCS (Ni et al., 2023) at 19.5%; GPT-J-6B with CoRE (Zhu et al., 2023) at 34.9% (maj1@1); ChatGLM2-6B at 32.4% (8-shot ICL) and 28.1% (human alignment); ChatGLM2-12B at 40.9% (8-shot ICL) and 38.1% (human alignment); InternLM-7B at 31.2% (4-shot ICL) and 34.5% (human alignment). All per Table 3.

• Generation budget / compute accounting. The paper uses two distinct notions of compute:

  1. For training: FLOPs are estimated following Kaplan et al. (2020). Pre-training FLOPs use $\text{FLOPs} \approx 6 \times N \times D_{\text{pretrain}}$ where $N$ is non-embedding parameters. SFT FLOPs use the same formula with SFT token counts. RFT inference FLOPs use a forward-pass formula accounting for KV-cache during autoregressive decoding (Appendix E, Equations 3-9), with total inference cost computed as $C_{\text{total}} = N_s \cdot [n_q C_{\text{forward}}(n_q) + \sum_{i=n_q}^{n_q+n_r} i \cdot C'_{\text{forward}}(i)]$ where $n_q \approx 66$ and $n_r \approx 130$ are average question and response token lengths. All GPU hour estimates are based on NVIDIA A100 80GB GPUs, with DeepSpeed ZeRO-3 (Rasley et al., 2020) used for distributed training of 33B+ models.
  2. For data augmentation: The "generation budget" for RFT is the sampling count $k$—how many candidate reasoning paths are generated per training question. This is the primary independent variable in the RFT scaling experiments (Figure 4), with $k \in \{1, 3, 6, 12, 25, 50, 100\}$. The deduplication step (Algorithm 1) means the actual number of training examples added to the augmented dataset is smaller than $k$ times the number of questions; Table 2 reports the average distinct paths per question at each $k$, and Table 5 reports the total training dataset size for each RFT configuration.

• Cross-validation / statistical protocol. The paper does not use cross-validation for model selection or hyperparameter tuning. All SFT and RFT models are trained once on the full training set (or the specified fraction thereof for the data scaling experiments) and evaluated on the standard GSM8K test set. For the downsampled SFT experiments (Section 3.2), the only hyperparameter sweep is over the number of training epochs: the authors search over {3, 3 / data_fraction} epochs because very small data fractions produce poor results at 3 epochs (insufficient gradient steps). They report the better test set result between these two epoch settings, ensuring the optimization budget (total gradient steps) is roughly constant across data fractions. No statistical significance testing, confidence intervals, or multiple-run variance estimates are reported for any experiment—all results are point estimates from single training runs, which is a limitation the paper does not discuss.

Main Quantitative Results

Pre-Training Loss vs. Model Performance (Figure 2)

The paper establishes pre-training loss as a performance indicator by plotting ICL (8-shot) and SFT accuracy against pre-training loss for GPT-3, LLaMA (7B, 13B, 33B, 65B), LLaMA2 (7B, 13B, 70B), and GPT-4. The key finding is an approximately negative linear relationship between pre-training loss and both ICL and SFT accuracy in the observed loss interval. The paper states:

"The pre-training losses are approximately negatively linear correlated to the SFT and ICL accuracy during the given pre-training loss interval."

Specifically, as pre-training loss decreases from approximately 2.0 (GPT-3-175B) to approximately 1.6 (LLaMA-33B), SFT accuracy increases from roughly 34% to roughly 55%. The ICL line is steeper: ICL accuracy increases from roughly 5% (GPT-3-175B, extrapolating) to roughly 35% (LLaMA-33B). The SFT line lies above the ICL line at all points, meaning SFT consistently outperforms ICL, but the gap between them narrows as pre-training loss decreases. The paper notes:

"SFT outperforms ICL consistently, while the improvements diminish when the pre-training loss is lower."

For LLaMA-33B (loss ~1.62), the SFT-ICL gap is approximately 19 points (54.6% - 35.6%), whereas for LLaMA-7B (loss ~1.80), the gap is approximately 25 points (35.9% - 11.0%). Better models gain less from fine-tuning because they have already internalized more reasoning ability during pre-training.

The paper acknowledges that these pre-training losses are not directly comparable across model families because they use different training data distributions, tokenizers, and loss computation methodologies. The linear relationship is claimed only within the observed interval and would break down at extremes (since accuracy is bounded in [0, 1]).

Supervised Data Scaling (Figure 3, Table 5)

The paper downsamples GSM8K to fractions {1, 1/2, 1/4, 1/8, 1/16, 1/32} of the full 7,473 training examples and performs SFT at each fraction for each model size. The results in Figure 3 reveal:

Log-linear relationship: Accuracy increases linearly with the logarithm of data amount. At 1/32 of the data (~230 examples), LLaMA-7B achieves roughly 7.8% (with 3 epochs) or 9.5% (with 96 epochs). At full data (7,473 examples), it achieves 35.9%. Each doubling of data produces roughly constant absolute improvement: going from 1/32 to 1/16 adds approximately 5 points, from 1/16 to 1/8 adds approximately 5 points, and so on. The log-linear relationship is "stable during {1, 1/2, 1/4, 1/8} amount of the training data" (Section 3.2).

Model-quality interaction: Better models benefit less from additional supervised data. The log-linear curves are flatter for stronger models. The paper states specifically:

"Better model benefits less when supervised data amount doubles."

At 1/32 of the data, LLaMA-33B achieves 18.6% (with 96 epochs) vs. LLaMA-7B's 9.5%. At full data, LLaMA-33B achieves 54.6% vs. 35.9%. The gain from full data over 1/32 data is approximately 36 points for 33B vs. 26 points for 7B—but this is total gain, not marginal gain. The slope of the log-linear curve (marginal gain per doubling) is steeper for 7B because 7B starts from a lower baseline and converges more slowly.

ICL crossover: Better models need more data to outperform their own ICL performance. At 1/32 of the data with 3 epochs, LLaMA-7B achieves 7.8% (below its ICL of 11.0%), while LLaMA-33B achieves 25.3% (below its ICL of 35.6%). At 1/16, LLaMA-7B achieves 12.7% (surpassing ICL's 11.0%), while LLaMA-33B needs 1/8 of the data (39.3%) to surpass its ICL (35.6%). The paper notes:

"Better model needs more amount of data to outperform its ICL performance."

Because better models have higher ICL accuracy, the SFT curve must climb further to exceed it, and since better models have flatter SFT curves, they require more data to make the crossing.

Numerical results are reported in Table 5. For the full data SFT (maj1@1): LLaMA-7B = 35.9%, LLaMA2-7B = 41.6%, LLaMA-13B = 43.0%, LLaMA2-13B = 50.0%, LLaMA-33B = 54.6%, LLaMA-65B = 59.3%, LLaMA2-70B = 63.2%.

Single-Model RFT Results (Table 1, Figure 4, Table 2)

RFT with a single SFT model generating $k=100$ candidate paths per training question yields the following maj1@1 results compared to SFT (Table 1):

For maj1@100: LLaMA-7B goes from 48.7% (SFT) to 52.7% (RFT), LLaMA2-7B from 55.4% to 58.7%, LLaMA-13B from 55.2% to 59.9%, LLaMA2-13B from 61.7% to 65.4%, and LLaMA-33B from 72.6% to no reported RFT value (Table 6). The gains are smaller for maj1@100 (approximately +4 points) than for maj1@1 (approximately +5-6 points for 7B/13B), suggesting RFT improves single-pass reliability more than it improves the upper bound of majority-voted accuracy.

RFT scaling with $k$ (Figure 4, Table 2): As $k$ increases from 1 to 100, RFT accuracy generally increases with diminishing returns. At $k=1$, RFT adds only 1.17 distinct paths per question (essentially the original SFT path plus a nearly-identical variant), yet it improves LLaMA-7B SFT from 35.9% to 37.6% (+1.7). At $k=3$, accuracy reaches 39.0% (+3.1). At $k=12$, 41.6% (+5.7). At $k=50$, 40.7% (a slight dip from $k=12$ for LLaMA-7B, suggesting non-monotonicity). At $k=100$, 41.7%. The improvement from $k=50$ to $k=100$ is only +1.0 point for LLaMA-7B even though distinct paths increase from 3.94 to 5.25—a diminishing return.

Deduplication ablation: The "no dedup" variant at $k=100$ (keeping all correct paths, approximately 400K training examples for LLaMA-7B vs. 47K with deduplication) achieves 43.6% for LLaMA-7B (Table 5), only marginally better than 41.7% with deduplication. This confirms that distinct reasoning path diversity, not raw training sample count, is the key factor. The paper states:

"Comparing using RFT with k = 100 and no dedup, the performance is similar and shows that it is better to estimate RFT performance based on distinct reasoning path amount instead of RFT augmented sample counts."

Why 33B fails: Table 2 shows LLaMA-33B generates 88.7 correct paths per question at $k=100$ but only 2.78 distinct paths—it produces the same reasoning path repeatedly with minor variations. The paper attributes this to overfitting: the 33B model "can well memorize the human-annotated reasoning paths" (Section 3.3). Increasing temperature to 1.0 improves diversity to 4.77 distinct paths but this configuration was not used for RFT training (the authors acknowledge a temperature grid search might help but was not conducted due to computational expense).

Multi-Model Aggregation RFT Results (Figure 5, Tables 3 and 5)

Fine-tuning on the aggregated dataset $D'_{\text{U13B}}$ (combining rejection samples from LLaMA-7B, LLaMA2-7B, LLaMA-13B, and LLaMA2-13B, each with $k=100$, then deduplicating) yields the paper's strongest results:

These are the headline numbers of the paper. LLaMA-7B RFT-U13B at 49.3% surpasses LLaMA-13B SFT (43.0%), LLaMA2-13B SFT (50.0%—essentially ties), and the GPT-3-175B SFT result of 34.0% from Cobbe et al. (2021). For maj1@100 (Table 6): LLaMA-7B RFT-U13B achieves 61.8% vs. 48.7% SFT; LLaMA2-13B RFT-U13B achieves 69.1% vs. 61.7% SFT.

Comparison with D'U33B: Adding LLaMA-33B's rejection samples to the aggregation ($D'_{\text{U33B}}$) provides negligible additional benefit (Table 5). For LLaMA-7B, 49.3% (U13B) → 49.1% (U33B); for LLaMA2-7B, 50.3% → 51.2%; for LLaMA-13B, 52.1% → 51.4%; for LLaMA2-13B, 55.4% → 55.3%. The only meaningful gain is for LLaMA-33B itself: 56.5% → 57.9% (+1.4). The paper explains this via the Venn diagram (Figure 6): LLaMA-33B contributes only 6.5% unique reasoning paths to $D'_{\text{U33B}}$, while each of the four 7B and 13B models contributes 14-15% unique paths. The 33B model generates low-diversity data that largely overlaps with what smaller models already produced.

Diminishing gains with model quality: The gains from RFT-U13B are inversely related to base model quality. LLaMA-7B gains +13.4, LLaMA2-7B gains +8.7, LLaMA-13B gains +9.1, LLaMA2-13B gains +5.4, LLaMA-33B gains +1.9, and LLaMA-65B actually regresses by 0.3 points. The paper states:

"We can assume with sufficient supervised data amounts, the performance indicator should be the model size but not the pre-training losses."

In other words, when training data is abundant (via multi-model RFT), the residual performance differences between same-size models (LLaMA-7B vs. LLaMA2-7B) nearly disappear, and the remaining gaps are due to capacity (13B > 7B). This is evidenced by LLaMA-7B RFT-U13B (49.3%) and LLaMA2-7B RFT-U13B (50.3%) being nearly identical, while LLaMA2-13B RFT-U13B (55.4%) remains clearly higher.

Comparison with open-source baselines (Table 3): RFT-U13B on LLaMA-7B (49.3%) substantially outperforms GPT-J-6B with CoRE (34.9%, from Zhu et al., 2023), GPT-Neo-2.7B with FCS+PCS (19.5%, from Ni et al., 2023), and the aligned versions of ChatGLM2-6B (28.1%), ChatGLM2-12B (38.1%), and InternLM-7B (34.5%). The paper notes that these aligned models "struggle at a level of 35 scores which are very similar to SFT performances of LLaMA-7B" and hypothesizes they "use GSM8K during their pre-training phase...or human alignment fine-tuning phase," meaning their reported numbers may not reflect the same pure fine-tuning setting.

Reasoning Path Diversity Analysis (Figure 7, Table 7)

To investigate the mechanism behind RFT's effectiveness, the paper analyzes the diversity of reasoning paths generated by RFT models at test time. For each model (LLaMA and LLaMA2, 7B and 13B, trained with SFT, RFT $k=100$, and RFT-U13B), the authors sample 100 reasoning paths per test question at temperature 0.7 and count the number of distinct calculation processes (equation lists) that lead to the correct answer.

Figure 7 shows histograms of question counts binned by the number of unique correct calculation processes produced. The key comparison is between SFT and RFT-U13B models:

• SFT models have many questions where all 100 sampled paths yield only 1 distinct calculation process. The paper explicitly notes: "There are more question counts for SFT models where all the sampled reasoning paths only correspond to one single calculation process and SFT models can barely generate more than 8 different calculation processes for a question."

• RFT-U13B models shift the distribution rightward: fewer questions with only 1 distinct process, and more questions with 2-10+ distinct processes. The paper highlights the difference "in two cases where the numbers of unique reasoning calculation paths are 1 or more than 10."

This demonstrates that training on diverse reasoning paths (from RFT-U13B) causes the model to learn multiple solution strategies that it can deploy at test time, rather than memorizing a single approach per problem. The paper concludes:

"This analysis demonstrates that diverse reasoning calculation paths in training data can equip the LLMs with finding diverse reasoning logic for solving math problems."

Table 7 provides qualitative case studies. For a simple rate problem ("Weng earns $12 an hour for babysitting..."), the RFT augmented data contains 5 distinct solution paths: computing per-minute rate then multiplying (Paths 1-2), computing fraction of an hour then multiplying (Path 3), and one-step combined expressions (Paths 4-5). All arrive at the same answer through different equation sequences. For a complex multi-step enrollment problem, 3 distinct paths are shown with different ways of grouping arithmetic operations. These cases illustrate the type of diversity RFT captures.

Computational Cost Analysis (Table 4)

The paper estimates and compares the FLOPs and GPU hours for pre-training, SFT, RFT inference, and RFT training across model sizes. Key numbers for LLaMA-7B:

• Pre-training: $4.2 \times 10^{22}$ FLOPs, 82,000 GPU hours
• SFT: $1.7 \times 10^{17}$ FLOPs, 0.6 GPU hours (~$4 \times 10^{-6}$ of pre-training)
• RFT inference ($k=100$ on 7,473 questions): $1.4 \times 10^{18}$ FLOPs, 10 GPU hours (~$3.3 \times 10^{-5}$ of pre-training)
• RFT-U33B total (inference + training): $3.0 \times 10^{18}$ FLOPs, 9 GPU hours

The cost hierarchy is unambiguous: SFT and RFT are negligible compared to pre-training. For 7B and 13B models, RFT-U33B costs 9-62 GPU hours and yields up to 13.4 accuracy points. The paper uses this to argue:

"Since you can obtain an RFT model without too much effort (compared to pre-training), then the most important thing we should do is to decrease the model's pre-training loss."

However, RFT costs grow with model size: RFT inference for 33B costs 4,500 GPU hours vs. 530,000 for pre-training—still only ~0.8% but no longer trivially small. And for 33B and 65B models, the benefit is marginal or zero, making the cost harder to justify on a return-on-investment basis.

Ablation Studies and Robustness Checks

• Deduplication vs. no deduplication (Figure 4, Table 5): At $k=100$, RFT without deduplication uses approximately 400K training examples (all correct reasoning paths kept) vs. approximately 47K with deduplication (distinct equation lists only). For LLaMA-7B, no-dedup achieves 43.6% vs. 41.7% with dedup—a marginal improvement of +1.9 points for 8.5× the training data. For LLaMA2-7B, no-dedup achieves 46.7% vs. 47.5% with dedup—actually slightly worse. For LLaMA-13B, no-dedup achieves 46.9% vs. 49.1% with dedup—worse by 2.2 points. The paper notes: "Furthermore, using deduplication has better performances for 3 of 4 models and needs much less training time." This ablation establishes that distinct reasoning path diversity, not raw quantity, is the operative variable.

• Varying $k$ (Figure 4, Table 2): RFT accuracy is measured at $k \in \{1, 3, 6, 12, 25, 50, 100\}$. For all models, increasing $k$ yields diminishing returns because distinct paths per question grow sublinearly (e.g., for LLaMA-7B: 1.17 at $k=1$, 2.20 at $k=12$, 3.94 at $k=50$, 5.25 at $k=100$). The paper explicitly connects this to the supervised data scaling result: "Doubling reasoning paths should improve less than doubling training samples since obtaining different reasoning paths does not obtain any new questions." This ablation also reveals that $k=3$ already provides a consistent +2 point improvement across all models despite adding only ~0.3 distinct paths per question, suggesting even minimal reasoning diversity is valuable.

• Single-model RFT vs. multi-model aggregation (Figure 5, Table 5): For LLaMA-7B, RFT $k=100$ (self-generated data) yields 41.7% vs. RFT-U13B (multi-model data) at 49.3%—a +7.6 point gap using the same $k=100$ budget per contributing model. The Venn diagram (Figure 6) explains this: each contributing model adds unique reasoning paths not generated by the others, so the aggregated dataset has 12.84 distinct paths per question vs. 5.25 for single-model RFT. This ablation demonstrates that cross-model reasoning diversity is the key driver of RFT's gains, not just within-model diversity.

• Adding LLaMA-33B data to the aggregation (Figure 5, Tables 2 and 5): Comparing RFT-U13B and RFT-U33B, the performance difference is minimal (often within ±1 point). Table 2 shows $D'_{\text{U33B}}$ has only 13.65 distinct paths per question vs. 12.84 for $D'_{\text{U13B}}$—an increase of only 0.81 paths despite adding data from a much larger model. The Venn diagram (Figure 6) shows LLaMA-33B contributes only 6.5% unique paths. This ablation demonstrates that model scale does not guarantee reasoning diversity; the 33B model's tendency to overfit the training set makes its generated data redundant.

• Epoch count for small data fractions (Appendix A.2): For the supervised data scaling experiments, the authors compare two epoch settings: 3 epochs (the standard) and 3 / data_fraction epochs (keeping total gradient steps constant). For LLaMA-7B at 1/32 data: 3 epochs yields 7.8% while 96 epochs yields 9.5%. For LLaMA-13B at 1/16 data: 3 epochs yields 7.4% while 48 epochs yields 27.7%. The paper reports the better result for each data fraction, ensuring that poor performance at small data sizes is not an artifact of insufficient optimization. The large gaps at small fractions (e.g., 7.4% vs. 27.7%) indicate that with very little data, the model needs many more passes through the same examples to learn effectively.

• Self-query augmentation and self-revising augmentation (Appendix D, Figure 8): The paper tested two alternative data augmentation methods that failed to provide meaningful improvements:

  • Self-query augmentation (Appendix D.1, Figure 8 left): Generating new queries for incorrect reasoning chains using a reversed GSM8K model (predict query from reasoning). Fine-tuning on this augmented data (alone or mixed with original data) produced worse results than SFT. The paper identifies two defects: incorrect reasoning chains contain errors that make them poor training examples, and the query generation model produces mismatched or low-quality queries.
  • Self-revising augmentation (Appendix D.2, Figure 8 middle/right): Training a model to revise incorrect reasoning paths into correct ones. This provided only marginal improvement at $k=1$ (36.09% vs. 35.90% SFT for LLaMA-7B) and degraded as $k$ increased. Two mitigations were tried: selecting the most lexically diverse incorrect path (by Levenshtein distance from the correct path) improved performance uniformly across $k$, while N-fold cross-validation (training the SFT model on N-1 folds and generating on the held-out fold) did not help. These negative results motivate RFT as the simpler and more effective alternative.

• Temperature effect on reasoning diversity (Section 3.3, discussion of 33B): For LLaMA-33B, increasing sampling temperature from 0.7 to 1.0 increases distinct paths per question from 2.78 to 4.77, while decreasing correct paths from 88.7 to 82.4. The authors "admit there should be a temperate (or generation config) that can produce more distinct paths and generate good results for RFT in 33B and even larger models while it does need more computation resources for inference." This is a sensitivity analysis in principle but not a systematic ablation—only one alternative temperature was tested, and no RFT model was trained on the temperature-1.0 data, so the downstream effect on RFT accuracy is unknown.

Critical Assessment

Claim: "Pre-training loss is a better indicator of model performance than parameter count"

Supported with qualifications. Figure 2 demonstrates an approximately negative linear relationship between pre-training loss and both ICL and SFT accuracy for the LLaMA and LLaMA2 families. The observation that LLaMA-7B (loss ~1.80, 35.9% SFT) outperforms GPT-3-175B (loss ~2.0, 34.0% SFT) despite 25× fewer parameters is a compelling illustration that parameter count alone is insufficient. However, the claim's generality is constrained by several factors:

First, the pre-training losses are not comparable across model families in a strict sense—they're computed over different training datasets with different tokenizers, making any cross-family loss comparison approximate. The paper acknowledges this: "we should notice that pre-training losses correspond to different pre-training datasets and different tokenizers which means they could not be compared strictly." The linear relationship is therefore a within-family regularity (LLaMA, LLaMA2) combined with suggestive cross-family correlation. The claim that loss is better than parameter count is directionally true in this dataset, but the magnitude of the advantage cannot be precisely quantified from these non-comparable loss values.

Second, the relationship is characterized only for a specific task (GSM8K math reasoning) and a specific model architecture family (decoder-only transformers). Whether pre-training loss would predict code generation, commonsense reasoning, or factual QA performance as reliably is untested.

Third, the functional form is an empirical approximation. The paper notes that $\log(\text{acc})$ would be "theoretically" more appropriate than raw accuracy as the dependent variable, and the linear relationship would necessarily break down as accuracy approaches 1.0 or 0.0. The observed relationship holds in the specific interval of losses and accuracies studied, but extrapolation beyond this range is not supported.

An experiment that would strengthen this claim: systematic comparison of models with identical architecture and tokenizer but different pre-training losses (e.g., intermediate checkpoints from a single long training run), which would eliminate the cross-family comparability confound.

Claim: "SFT improves in a log-linear manner with the increase of supervised data amount, and better models benefit less"

Supported, with a nuance about epoch tuning. Figure 3 shows log-linear scaling across 1/32 to 1 data fractions, and the slopes are visibly flatter for better models (e.g., LLaMA-65B's curve is nearly flat, LLaMA-7B's is steep). The numerical results in Table 5 support this: LLaMA-7B gains ~26 points from 1/32 to full data, while LLaMA-33B gains ~36 total points but with a clearly lower per-doubling increment at higher data fractions.

The nuance concerns the epoch tuning. For small data fractions, the standard 3-epoch setting produces very poor results (e.g., LLaMA-13B at 1/32 with 3 epochs: 0.0% accuracy). The authors report the better of 3 epochs and 3/data_fraction epochs, which means the optimization budget is not constant across data fractions—models trained on less data get more gradient steps per example. This is a reasonable experimental design choice (it avoids confounding data quantity with optimization quality), but it means the "log-linear relationship" is partially a function of this tuning. Would the same log-linear relationship hold if all models received exactly 3 epochs? The very poor performance at 3 epochs for small fractions suggests the relationship might be even steeper (worse performance at low data) without epoch tuning.

Additionally, the paper does not extrapolate the log-linear curves to predict what accuracy would be achieved with 2×, 4×, or 10× the GSM8K training data. Given the acknowledged difficulty of collecting new math word problems, this extrapolation would be practically valuable. The flattening of curves for stronger models (especially 65B, where 1/4 data already achieves 50.4% vs. 59.3% at full data—most of the gain coming from the first quarter of the data) suggests that additional supervised data beyond the full GSM8K set would provide minimal benefit for strong models, but this is stated qualitatively rather than quantitatively extrapolated.

Claim: "RFT performance improves as distinct reasoning path amount increases, and RFT brings more improvement for less performant LLMs"

Well-supported, with a critical caveat about the 33B failure mode. The deduplication ablation (Figure 4, comparing dedup vs. no-dedup at $k=100$) cleanly isolates distinct path count as the key variable. Table 2 and Figure 4 together show the relationship: more distinct paths → higher accuracy, with the sublinear growth of distinct paths explaining the diminishing returns. The Venn diagram (Figure 6) further supports this by showing that multi-model aggregation works because it increases distinct path count beyond what any single model can achieve.

The "more improvement for less performant models" claim is empirically clear: LLaMA-7B gains +13.4 from RFT-U13B, LLaMA2-7B gains +8.7, LLaMA-13B gains +9.1, LLaMA2-13B gains +5.4, LLaMA-33B gains +1.9, LLaMA-65B gains −0.3. The pattern is monotonic.

The caveat concerns the 33B failure mode. The paper attributes LLaMA-33B's low reasoning diversity (2.78 distinct paths per question) to overfitting, but the evidence is circumstantial. The 33B model generates 88.7 correct paths per question (the highest of all models), suggesting it has strong single-pass accuracy on the training distribution, but nearly all these paths are near-duplicates. This could be due to overfitting, or it could be due to the specific temperature setting (0.7), or it could be an artifact of how the SFT model was trained (the authors don't explore whether different SFT hyperparameters—learning rate, number of epochs, data ordering—affect generation diversity). The paper tested only one alternative temperature (1.0) for 33B and found 4.77 distinct paths—better but still below 7B/13B levels. A systematic sweep of generation parameters (temperature, top-p, top-k) might find a regime where 33B generates diverse paths, which would change the conclusion about RFT's applicability to larger models.

Additionally, the 65B and 70B models were not evaluated for RFT beyond the multi-model aggregation setting (RFT-U13B and RFT-U33B). We don't know their single-model RFT diversity (distinct paths per question), so we can't determine whether their failure to benefit from RFT is due to low generation diversity (like 33B) or due to being near their reasoning ceiling (saturation). The paper states "better models benefit less from the supervised sample amounts" but this is a general principle; the specific mechanism for 65B/70B is not diagnosed.

Claim: "The combination of rejection sampling from multiple models further enhances RFT performance"

Strongly supported. RFT-U13B substantially outperforms single-model RFT at $k=100$ for all 7B and 13B models (Figure 5). For LLaMA-7B: 49.3% (U13B) vs. 41.7% (self RFT $k=100$), a gap of +7.6 points. For LLaMA2-7B: 50.3% vs. 47.5%, +2.8. For LLaMA-13B: 52.1% vs. 49.1%, +3.0. For LLaMA2-13B: 55.4% vs. 54.8%, +0.6. The diminishing gap for stronger models is consistent with the broader pattern of better models benefiting less from additional data.

The Venn diagram (Figure 6) provides mechanistic evidence: each contributing model adds 14-15% unique reasoning paths not generated by other models, confirming that cross-model diversity is real and quantifiable. The ablation showing $D'_{\text{U33B}}$ provides negligible additional benefit over $D'_{\text{U13B}}$ (Table 5) further reinforces that what matters is the effective diversity added, not the size or perceived quality of the contributing model.

A missing experiment that would strengthen this claim: what is the contribution of each individual model to the aggregated performance? Removing one model at a time from the aggregation (leave-one-model-out) would quantify the marginal value of each source. The Venn diagram addresses this at the level of reasoning path counts, but not at the level of downstream accuracy.

Claim: RFT is "simpler" and more effective than prior augmentation methods

Supported by the numbers but with an incomplete baseline comparison. Table 3 shows RFT-U13B on LLaMA-7B (49.3%) substantially outperforms GPT-J-6B with CoRE (34.9%, from Zhu et al., 2023) and GPT-Neo-2.7B with FCS+PCS (19.5%, from Ni et al., 2023). However, these comparisons are across different base models, making it impossible to isolate the method's contribution from the base model's quality. CoRE was evaluated on GPT-J-6B, not LLaMA-7B. FCS+PCS was evaluated on GPT-Neo-2.7B. A proper method comparison would require implementing CoRE and FCS+PCS on the same LLaMA base models, which the paper does not do.

The paper also does not compare RFT against STaR (Zelikman et al., 2022), which is the most closely related prior method. STaR iteratively generates rationales, filters for correctness, and fine-tunes—essentially multi-round RFT. Would multiple rounds of RFT (using the RFT model to generate data for the next round) provide additional gains beyond single-round RFT? The paper doesn't investigate this, and the ReST$^{EM}$ failure reported in a different context (Appendix D.1 mentions self-query augmentation failed; Appendix D.2 mentions self-revising augmentation failed) doesn't directly test multi-round RFT. This is a significant gap, since STaR's iterative approach is a natural extension of single-round RFT.

General weaknesses

Single benchmark, single task type. All experiments are on GSM8K, a dataset of grade-school math word problems. The findings may not generalize to other reasoning tasks (algebraic reasoning, geometry, code generation, logical deduction), other math benchmarks (MATH, which is harder), or non-reasoning tasks. The paper doesn't acknowledge this limitation or discuss domain specificity of the scaling relationships.

Point estimates without variance. All results are from single training runs. No error bars, confidence intervals, or multiple-seed experiments are reported. For the downsampled SFT experiments with very small data fractions (e.g., 1/32 of GSM8K = ~230 examples), random variation in which examples are selected could substantially affect results. The 33B single-model RFT result (54.5%, 0.1 points below SFT) could easily be within noise—we cannot determine if it's a real degradation or statistical fluctuation.

No test of extrapolation. The supervised data scaling experiments go from 1/32 to 1 of GSM8K, but the paper doesn't attempt to predict what would happen with 2× or 4× the GSM8K data. The RFT scaling experiments go up to $k=100$ but don't test $k=200$ or $k=400$ to see if the distinct path count eventually saturates completely. The extrapolability of the log-linear and diminishing-return patterns is assumed but not validated.

Table 3 comparisons are confounded by base model differences. The paper's strongest comparative claim—that RFT outperforms prior augmentation methods—relies on cross-model comparisons. CoRE's 34.9% on GPT-J-6B vs. RFT-U13B's 49.3% on LLaMA-7B may reflect LLaMA-7B being a stronger base model, not RFT being a better method. The paper would need to implement competing methods on the same base models to make this comparison valid.

Missing systematic generation parameter study for diversity. The paper diagnoses LLaMA-33B's low reasoning diversity as the cause of RFT failure but tests only two temperature settings (0.7 and briefly 1.0). A full study varying temperature, top-p, top-k, and repetition penalty across model sizes would reveal whether diversity is fundamentally limited in larger models or merely requires different generation settings. This is practically important because it determines whether RFT can be extended to larger models with appropriate decoding.

No investigation of data quality vs. data diversity. RFT's correctness filtering ensures all augmented paths produce the correct answer and have correct calculations verified by Python. But some of these correct paths may be poorly written, contain redundant steps, or use inefficient solution strategies. Does the quality of reasoning paths (beyond correctness) affect downstream RFT performance, or is diversity the only factor? Could selecting only the most elegant or efficient correct paths (rather than one per equation list) improve results further? This dimension is unexplored.

Computational cost of difficulty estimation or data generation not amortized into main results. The paper's cost analysis (Table 4) separates RFT inference cost from training cost, but the headline accuracy numbers (e.g., "RFT-U13B achieves 49.3%") don't include the cost of generating the augmented data. For a fair comparison with methods that don't require data generation (e.g., just collecting more human data), the total cost to reach a given accuracy should include both generation and training. The paper argues RFT is "cheap" but the RFT inference for 33B costs 4,500 GPU hours—not trivial, and the resulting benefit is near zero. A cost-normalized accuracy curve (accuracy per total FLOP invested, including generation) would be more informative than the raw accuracy numbers.

6. Limitations and Trade-offs

Limitation 1: The Method Fails Completely on Large Models (33B+) — Self-Generated Data Lacks Diversity

The assumption or constraint. RFT assumes that the SFT model can generate diverse correct reasoning paths when sampled at moderate temperature. The paper effectively demonstrates this for 7B and 13B models (LLaMA-7B generates 5.25 distinct paths per question at k=100), but the assumption breaks for larger models. The authors acknowledge this explicitly for the 33B case in Section 3.3:

"For 33B models, RFT does not improve performance compared to SFT. The main reason comes from the augmented samples from rejection sampling."

They further diagnose the root cause:

"it overfits the training set and has difficulty generating more diverse paths on the training set questions."

The consequence. RFT with self-generated data provides zero or negative benefit for models at 33B scale and above. LLaMA-33B RFT (k=100) achieves 54.5%, actually 0.1 points below SFT at 54.6% (Table 1). LLaMA-65B RFT-U13B achieves 59.0%, 0.3 points below SFT at 59.3% (Table 5). This means RFT is not a universal method that scales with model size—it has a capability ceiling where the model becomes too specialized to generate useful training data for itself. The implication is stark: a practitioner deploying RFT on a 33B or 65B model (which are the ones most likely to be used in production) would see no benefit while still incurring the full computational cost of generating rejection samples (4,500 GPU hours for 33B, per Table 4).

What evidence exists in the paper. The evidence is comprehensive. Table 2 shows LLaMA-33B generates only 2.78 distinct paths per question at k=100 vs. 5.25 for LLaMA-7B, despite generating many more correct paths (88.7 vs. 53.3). The Venn diagram (Figure 6) quantifies that 33B contributes only 6.5% unique paths to the aggregated dataset—the four 7B and 13B models each contribute 14-15%. Table 5 shows near-zero or negative gains for 33B, 65B, and 70B models across RFT configurations. The paper also tests a higher temperature (1.0) for 33B, which improves diversity to 4.77 distinct paths but this was not used for RFT training, and it's still below the 5.19-5.29 achieved by 7B/13B models at temperature 0.7.

Mitigation status. The paper partially mitigates this through multi-model aggregation (RFT-U13B, RFT-U33B), which allows large models to benefit from data generated by smaller models that have higher reasoning diversity. For LLaMA-33B, RFT-U33B achieves 57.9% vs. SFT's 54.6% (+3.3), which is a meaningful gain. However, this mitigation implicitly concedes that large models cannot self-improve via RFT—they depend on smaller models' generation, which has its own cost (training and running the smaller models). The paper does not explore whether systematic hyperparameter tuning (temperature sweep, top-p, repetition penalty) could recover diversity for large models, stating only that "it does need more computation resources for inference compared to sampling using 7B and 13B models" (Section 3.3). This leaves open the question of whether the failure is fundamental or merely requires better generation settings, making it a partially mitigated but unresolved limitation.

---

Limitation 2: Difficulty Estimation Cost Is Not Included in the Efficiency Analysis — and the Method Has No Difficulty-Aware Allocation

The assumption or constraint. Unlike the "Scaling Relationship on Learning Mathematical Reasoning" paper you initially read, this paper's RFT method does not estimate question difficulty or adapt its data generation strategy per question. It applies the same k=100 sampling uniformly across all 7,473 training questions. The paper's framing of RFT as "cheap" (Section 4.2) compares RFT inference cost against pre-training cost, showing it's ~0.01% for 7B models (Table 4). However, this comparison hides an important practical question: does the method need to generate and filter k samples for every training question, or could the budget be allocated more efficiently? The paper offers no mechanism for determining which questions would benefit most from additional reasoning paths.

The consequence. In deployment, a practitioner wanting to apply RFT to a new dataset would either (a) pay the full k-sample cost for every question, regardless of whether a question already has sufficient reasoning diversity from the human annotations alone, or (b) guess at a smaller k and potentially leave performance on the table for questions that needed more diversity. The paper's data scaling experiments (Figure 3) show that doubling supervised data produces log-linear accuracy gains—but RFT does not exploit this by targeting data augmentation toward questions where the base SFT model already shows uncertainty or error. The method is uniform and uncalibrated: questions that the SFT model already solves correctly 99% of the time get the same augmentation budget as questions it solves 10% of the time.

This is not merely a theoretical concern. The paper reports (Table 2) that for LLaMA-7B at k=100, the average distinct paths per question is 5.25, but this is an average—some questions likely generate 10+ distinct paths while others generate only 1-2 (the original annotation plus a near-duplicate). The uniform allocation wastes compute on questions that already have sufficient diversity while under-investing in questions where more paths would be most valuable. A difficulty-aware allocation could achieve the same downstream accuracy with a fraction of the generation budget, or higher accuracy with the same budget.

What evidence exists in the paper. The paper does not measure or discuss this issue directly. There is no experiment comparing uniform vs. targeted augmentation, no analysis of per-question diversity variation, and no cost-benefit analysis that accounts for the fact that some questions need more augmentation than others. The data scaling experiments (Section 3.2, Figure 3) randomly downsample the training set—they don't test whether selectively augmenting specific questions would outperform uniform augmentation. The computational cost analysis (Table 4, Section 4.2) compares total RFT cost to pre-training cost but does not discuss whether the RFT budget itself is optimally spent.

Mitigation status. The paper does not address this limitation. It presents RFT as a uniform procedure applied identically to all questions, with k as the only hyperparameter controlling the augmentation budget. There is no suggestion of future work on difficulty-adaptive augmentation, no analysis of per-question diversity distributions, and no comparison of RFT against an "oracle" that would allocate more samples to questions where additional diversity is most needed. This is a significant gap because difficulty-aware allocation could substantially improve RFT's cost-effectiveness, especially for practitioners with limited inference budgets.

---

Limitation 3: Single Benchmark (GSM8K), Single Task Type (Grade-School Math Word Problems)

The assumption or constraint. All experiments in the paper are conducted exclusively on the GSM8K benchmark, which consists of grade-school-level math word problems requiring multi-step arithmetic reasoning. The paper makes no claims about generalization to other math reasoning benchmarks (e.g., MATH, which is substantially harder), other types of reasoning (code generation, logical deduction, commonsense QA), or non-reasoning tasks. The findings about log-linear supervised data scaling, the effectiveness of rejection sampling augmentation, and the relationship between pre-training loss and downstream performance are all demonstrated on a single dataset. The paper does not explicitly state this as a limitation in the main text or in the limitations section (Section 7), though it is implicitly acknowledged by the consistent focus on GSM8K throughout.

The consequence. A practitioner cannot assume that the scaling relationships and RFT gains demonstrated on GSM8K will transfer to other tasks or domains. Several aspects of GSM8K make it particularly amenable to the paper's approach: (a) answers are cleanly extractable and verifiable (numerical values that can be exact-matched), (b) reasoning paths follow a relatively constrained format with explicit calculation steps (<<expression>>result), (c) the problems, while multi-step, are within the capability range of 7B models (SFT accuracy of 35.9% for LLaMA-7B means the model produces correct answers at a non-trivial rate, enabling rejection sampling to find correct paths). For harder benchmarks like MATH, where base model pass@1 might be near zero, rejection sampling would generate almost no correct paths to augment with—RFT would be impossible. For tasks where "correctness" is ambiguous or multi-dimensional (summarization, dialogue, creative writing), the filtering mechanism breaks down entirely.

Additionally, the specific relationship between pre-training loss and downstream performance (Figure 2) may be task-dependent. GSM8K requires arithmetic reasoning that benefits from the procedural knowledge LLMs acquire during pre-training. A task requiring factual recall (e.g., closed-book QA) might show a different relationship with pre-training loss, and one requiring multilingual capabilities might depend more on pre-training data composition than on aggregate loss.

What evidence exists in the paper. None. The paper does not conduct any experiments on datasets other than GSM8K, does not discuss domain-specific characteristics that might affect generalizability, and does not speculate about which findings would or would not transfer. The limitations section (Section 7) does not mention the single-benchmark scope as a limitation. The only indirect evidence is the paper's use of GSM8K's specific format (equation extraction for deduplication, answer verification by matching, Python-based calculation checking), which would require non-trivial adaptation for other reasoning benchmarks.

Mitigation status. Not addressed. The paper does not test on any secondary benchmark, does not discuss domain transfer, and does not suggest that future work should validate the scaling relationships on other tasks. This is a straightforward and consequential limitation: the entire empirical edifice rests on one dataset, and the paper provides no reason to believe the findings are general.

---

Limitation 4: The FLOPs-Matched Pretraining Comparison Makes the Wrong Comparison — Parameters Are Scaled, Not Total Compute

The assumption or constraint. Section 4.2 presents a computational cost comparison (Table 4) arguing that SFT and RFT are negligible compared to pre-training: SFT is ~0.0004% of pre-training FLOPs for LLaMA-7B, and RFT-U33B is ~0.007%. The paper uses this to conclude that "the most important thing we should do is to decrease the model's pre-training loss" because pre-training dominates the total cost and benefits all tasks. However, this comparison makes an implicit assumption: that pre-training cost scales only with model size, not with training data quantity or duration. The paper compares the cost of RFT for a 7B model against the pre-training cost of a 7B model as reported in the LLaMA paper—it does not ask whether a smaller model trained on more data (achieving lower pre-training loss at lower total cost) would be a more efficient baseline.

The consequence. The cost comparison in Table 4 is not a true FLOPs-matched comparison of different strategies for improving reasoning performance. The paper never asks: given a fixed total FLOPs budget, should I spend it on (a) pre-training a 13B model and doing SFT, or (b) pre-training a 7B model, doing SFT, and doing RFT-U13B? The LLaMA-7B RFT-U13B achieves 49.3%, and LLaMA-13B SFT achieves 43.0%—but the LLaMA-13B pre-training uses 1.86× the FLOPs of LLaMA-7B pre-training ($7.8 \times 10^{22}$ vs. $4.2 \times 10^{22}$). If we account for the RFT-U33B cost for the 7B model ($3.0 \times 10^{18}$ FLOPs, per Table 4), the total 7B investment is still only ~$4.2 \times 10^{22}$ FLOPs—about 54% of the 13B pre-training cost—and achieves better accuracy. This is actually a stronger argument for RFT than the paper makes, but it's never explicitly formulated.

Conversely, the paper does not explore whether pre-training the 7B model for longer (spending the extra FLOPs on more pre-training data rather than on RFT) would achieve the same accuracy gain more efficiently. LLaMA2-7B was trained on approximately 2× the data of LLaMA-7B (doubling pre-training FLOPs from $4.2 \times 10^{22}$ to $8.4 \times 10^{22}$) and achieves SFT accuracy of 41.6% vs. 35.9% for LLaMA-7B—a gain of +5.7 points. RFT-U13B on LLaMA-7B adds +13.4 points for $3.0 \times 10^{18}$ FLOPs (0.007% of the pre-training cost). This superficially suggests RFT is vastly more efficient than extended pre-training. But this comparison is misleading: pre-training improves all downstream tasks, while RFT improves only GSM8K-style math reasoning. A proper comparison would need to account for the breadth of capabilities improved by pre-training vs. the narrow improvement from RFT.

What evidence exists in the paper. Table 4 provides FLOPs and GPU hour estimates for pre-training, SFT, and RFT at each model scale. Figure 2 and Table 5 can be used to trace the accuracy vs. pre-training loss (and thus implicitly vs. pre-training FLOPs) relationship. However, the paper does not construct a controlled FLOPs-matched experiment—for example, training a 7B model on 2×, 3×, or 4× data and comparing its SFT and RFT accuracy against a larger model trained on the original data. The cost comparison in Section 4.2 is a descriptive cost breakdown, not a prescriptive optimization.

Mitigation status. The paper acknowledges that its pre-training cost figures come from the LLaMA papers and that "we believe other abilities may follow a similar pattern and better pre-training can benefit all other tasks" (Section 4.2). But this is stated as a belief, not demonstrated. The paper does not perform any form of FLOPs-controlled comparison between pre-training scaling and data augmentation, does not discuss the breadth-of-capability tradeoff, and does not address the fact that pre-training improvements are amortized across all downstream uses while RFT improvements are task-specific. The recommendation to prioritize pre-training loss reduction, while reasonable, is not quantitatively justified by the experiments in the paper.

---

Limitation 5: No Multi-Round or Iterative RFT — The Method Stops After One Generation

The assumption or constraint. RFT as presented is a single-round procedure: train an SFT model, use it to generate augmented data via rejection sampling, then train the RFT model on the augmented dataset. The RFT model itself is never used to generate a second round of augmented data. This is a deliberate simplification—the paper states RFT is "a simpler way of generating augmented samples without any trained process-level reward models" (Section 3.3)—but it ignores the possibility that iterative rounds of RFT could compound the gains. The closely related STaR method (Zelikman et al., 2022) uses multiple rounds of generation and fine-tuning, and the paper does not investigate whether this would provide additional benefit in their setting.

The consequence. The paper's reported RFT gains represent a lower bound on what bootstrapped data augmentation could achieve. If the RFT model generates more diverse reasoning paths than the SFT model (as suggested by Figure 7, showing RFT models produce more distinct calculation processes at test time), then using the RFT model for a second round of rejection sampling could yield even more diverse augmented data, potentially improving a third-round model further. Conversely, iterative RFT might hit a diversity ceiling faster (as the model converges to a fixed set of strategies) or might amplify errors if the RFT model occasionally generates superficially correct but subtly flawed reasoning paths that pass the correctness filter.

The paper's decision to stop at one round also prevents any analysis of whether RFT's gains are bounded or can be compounded. For LLaMA-7B, RFT-U13B achieves 49.3%—a +13.4 point gain over SFT. Could a second round using the RFT-U13B model as the generator add another 5-10 points? Or would the diversity of the 49.3%-accurate model be too low to provide benefit (as happens with the 54.6%-accurate 33B model in single-round RFT)? Without multi-round experiments, the paper cannot distinguish between "RFT gains saturate at one round" and "RFT gains compound with iteration."

What evidence exists in the paper. None directly. The paper does not conduct any multi-round RFT experiments, and STaR is mentioned only as a related work, not as a baseline. However, there is suggestive indirect evidence: Appendix D, Figure 8 shows experiments with alternative augmentation methods (self-query augmentation, self-revising augmentation) that failed to provide improvements. The self-revising augmentation experiment (Appendix D.2) attempted to train a model to revise incorrect paths into correct ones, which is a form of iterative refinement, and found only marginal gains. But this is not the same as multi-round RFT (where each round generates new solutions from scratch, not revisions of old ones). The paper's single-round focus means there is no evidence about the behavior of iterative RFT.

Mitigation status. The paper does not acknowledge this as a limitation. The authors frame RFT's simplicity as a strength ("Our RFT method is simpler compared to CoRE, since RFT does not require training verifier models and decoding with Monte Carlo Tree Search"), and they do not discuss whether iterative RFT was attempted, considered, or deferred to future work. Section 7 (Limitations) does not mention the absence of multi-round experiments. This is a significant gap because the question of whether self-improvement can be applied iteratively without degradation is a central concern in the broader self-training and self-play literature, and the paper's findings on diminishing diversity at larger model scales (Limitation 1) suggest iterative RFT might fail for a different reason: each round could reduce diversity, eventually hitting the same ceiling that prevents 33B from benefiting from single-round RFT. Investigating this would directly inform whether the paper's approach is a one-time boost or a scalable self-improvement paradigm.

---

Limitation 6: Point Estimates Only — No Variance, Significance Testing, or Stability Analysis

The assumption or constraint. All results in the paper are reported as point estimates from single training runs. There are no error bars, confidence intervals, standard deviations, or multiple-seed experiments for any table or figure. The supervised data scaling curves (Figure 3), the RFT scaling curves (Figure 4), and the multi-model aggregation results (Table 5) are each based on a single training run with a single random seed for data ordering, weight initialization (for the base model, which is fixed), and downsampling (for the data fraction experiments). The paper assumes that these point estimates are reliable estimates of the true expected accuracy of each configuration, without quantifying the variance.

The consequence. Several of the paper's claims depend on small numerical differences that could plausibly be within sampling noise. For example:

• The claim that LLaMA-33B RFT (k=100) does not improve over SFT is based on 54.5% vs. 54.6%—a difference of 0.1 percentage points, corresponding to approximately 1 test question out of 1,319. This could easily be noise.
• The claim that adding LLaMA-33B data to the aggregation provides negligible benefit (RFT-U13B vs. RFT-U33B) relies on differences of 0.2-1.4 points across models (Table 5). LLaMA-13B goes from 52.1% (U13B) to 51.4% (U33B)—a 0.7-point degradation. Is this a real signal that 33B data is slightly harmful, or just noise?
• The deduplication ablation (Figure 4) shows no-dedup performing better for LLaMA-7B (43.6% vs. 41.7%) but worse for LLaMA-13B (46.9% vs. 49.1%). The paper concludes deduplication "has better performances for 3 of 4 models"—but with point estimates only, these rankings could flip under different random seeds.
• The supervised data downsampling experiments (Section 3.2) use a single random draw of which examples to include at each fraction. For 1/32 of GSM8K (~230 examples), the random selection of training examples could substantially affect the measured accuracy. LLaMA-7B at 1/32 achieves 9.5% (with 96 epochs)—if a different random subset had been selected, could this be 7% or 12%? The paper provides no way to assess this.

Beyond the impact on specific claims, the absence of variance estimates makes it impossible to determine whether the scaling relationships the paper identifies (log-linear SFT scaling, diminishing RFT returns) are statistically reliable or merely visual fits to noisy data. The log-linear appearance of Figure 3 could partially reflect the particular random downsampling, and the non-monotonicities in Figure 4 (e.g., LLaMA-7B RFT dips from 41.6% at k=12 to 40.9% at k=25 before rising to 41.7% at k=100) could be noise rather than a meaningful pattern.

What evidence exists in the paper. The paper provides no variance-related evidence at all—no error bars, no multiple seeds, no statistical tests. The epoch count tuning for small data fractions (Appendix A.2) reports "better results among these two different epoch settings," which is a form of model selection but not a variance estimate. The GSM8K test set has 1,319 questions, which provides a reasonable sample size for point estimates (a 1-percentage-point difference corresponds to ~13 questions), but the standard error of a proportion at 50% accuracy is approximately $\sqrt{0.5 \times 0.5 / 1319} \approx 1.4$ percentage points, meaning differences of 1-3 points could easily arise from sampling variation alone.

Mitigation status. Not addressed. The paper does not discuss the absence of variance estimates, does not justify the use of single training runs, and does not suggest that future work should replicate findings with multiple seeds. This is a standard practice in many LLM papers (driven by computational cost—running 5 seeds for each of the dozens of configurations in this paper would multiply the already substantial compute budget by 5×), but it weakens the quantitative reliability of the claims. For the supervised data scaling experiments in particular, where the training sets are very small at low fractions and the random selection of examples could have large effects, multiple resampled datasets would provide a more robust characterization of the scaling relationship. The paper's practical recommendations (e.g., "doubling data increases performance by a unit," Section 3.2) are stated as precise quantitative relationships but may be less precise than the point estimates suggest.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around improving LLM reasoning from a methods-focused discourse (which augmentation technique is best?) toward a scaling-relationships discourse (how do pre-training quality, supervised data quantity, and augmented data diversity jointly govern performance?). Before this work, the field had individual demonstrations that rejection sampling (Zelikman et al., 2022), verifier-guided search (Zhu et al., 2023), and partial-solution augmentation (Ni et al., 2023) could improve math reasoning, but there was no systematic framework for predicting how much improvement to expect, for which models, and under what conditions. The paper provides that framework, even if empirically bounded to GSM8K and the LLaMA family.

The paper's most consequential reframing is the elevation of pre-training loss as the primary independent variable for downstream reasoning performance, displacing parameter count as the default metric of model quality. This is not a deep theoretical advance—the connection between pre-training loss and downstream performance has been discussed since the scaling laws literature (Hernandez et al., 2021; Henighan et al., 2020)—but the paper makes it operationally concrete for practitioners choosing base models. The finding that LLaMA-7B SFT (35.9%) outperforms GPT-3-175B SFT (34.0%) despite a 25× parameter difference (Table 3, Figure 2) is a memorable, decision-relevant data point that makes the abstract principle tangible. This reframing pushes the field toward evaluating models by their pre-training efficiency (loss per FLOP) rather than raw scale, aligning with the compute-optimal pre-training paradigm of Hoffmann et al. (2022) but applied at the downstream-task level.

The identification of distinct reasoning path diversity as the operative mechanism behind data augmentation gains is a genuine conceptual contribution. Prior work on rejection sampling for reasoning implicitly assumed that more correct examples would help, with diversity being an unarticulated benefit. The paper's deduplication ablation (Figure 4, comparing $k=100$ with and without deduplication) cleanly separates these hypotheses: keeping all correct paths (~400K examples) performs no better than keeping only distinct paths (~47K examples), demonstrating that raw data quantity is not the active variable. This is not obvious ex ante—one might reasonably expect that more training examples, even near-duplicates, would provide robustness through repeated exposure. The finding that they don't (and sometimes hurt, as with no dedup for LLaMA-13B achieving 46.9% vs. 49.1% with dedup) redirects attention toward diversity-maximizing generation strategies rather than throughput-maximizing ones.

The paper also provides a diagnostic for when self-improvement will fail: measure the distinct-path-per-question ratio of your generation model. The LLaMA-33B case (2.78 distinct paths per question at $k=100$ vs. 5.25 for 7B, Table 2) is a clear negative result that generalizes beyond this paper. Any self-training pipeline that uses a model to generate its own training data should monitor whether the generation model is producing diverse outputs or merely re-sampling the same solutions. This converts a potential failure mode ("why didn't self-training help my 33B model?") into a diagnostic procedure ("check distinct path count before committing to the full pipeline"). This is practically valuable because large-model RFT inference is expensive (4,500 GPU hours for 33B, per Table 4), and the paper's framework can save practitioners from wasting that compute on configurations unlikely to help.

The paper reconciles a latent tension in the self-improvement literature that was not previously articulated as a tension. On one hand, STaR (Zelikman et al., 2022) showed that iterative self-training on self-generated rationales improves reasoning. On the other hand, the paper's finding that better models generate less diverse data (Innovation 4 in the Key Insights section) implies that self-improvement becomes harder as the model improves—a paradox where capability increases but the fuel for further improvement (diverse training data) decreases. This isn't a contradiction of STaR's results (which were on smaller models where diversity was still high) but rather a boundary condition: STaR-style iteration works when the model still has reasoning diversity to exploit, and fails (or plateaus) when the model converges to a narrow solution distribution. The paper's multi-model aggregation (Section 3.3) is a partial resolution—use weaker, more diverse models to generate data for stronger models—but it also implies that fully autonomous self-improvement (a single model improving itself without external diversity injection) may have a fundamental ceiling determined by the model's propensity to collapse to low-diversity outputs during fine-tuning.

Finally, the paper establishes a pragmatic cost hierarchy for practitioners: SFT costs ~$10^{-5}$ of pre-training, RFT costs ~$10^{-4}$, and both are negligible compared to the cost of reducing pre-training loss (Table 4). The practical takeaway—"always apply RFT to 7B and 13B models, it's essentially free"—is a concrete recommendation grounded in cost accounting. This is not a paradigm shift but a useful calibration for resource allocation decisions. The flip side is equally important: for 33B+ models, RFT with self-generated data is not cost-effective (near-zero benefit for substantial inference cost), so resources should flow entirely to pre-training improvements or to multi-model aggregation using smaller models as generators.

Follow-Up Research This Work Enables

1. Extrapolating the supervised data scaling curves to predict the value of larger annotation budgets. The paper demonstrates a log-linear relationship between SFT data amount and accuracy (Figure 3), but stops at the full GSM8K training set (7,473 examples). A natural extension would be to train on 2×, 4×, and 8× the GSM8K data by combining GSM8K with other math word problem datasets (e.g., MAWPS, ASDiv, SVAMP) that share a similar format but provide different questions. This would test whether the log-linear relationship continues beyond the range studied, or whether accuracy saturates at some data quantity for each model size. The paper's finding that better models have flatter log-linear slopes (Figure 3, comparing LLaMA-7B vs. LLaMA-65B) predicts that the saturation point should be reached earlier for stronger models—a specific, testable hypothesis. A strong experiment would measure SFT accuracy at 1×, 2×, 4×, and 8× GSM8K-scale data, fit a curve to each model size, and report the extrapolated asymptotic accuracy and the data quantity needed to reach 90% of that asymptote. This would directly inform annotation budget decisions: "if I have $X to spend on human annotation for math reasoning, should I annotate more examples for my 7B model or switch to a 13B model with existing data?"

2. Systematic generation parameter optimization to recover reasoning diversity in large models. The paper diagnoses LLaMA-33B's RFT failure as a diversity problem (only 2.78 distinct paths per question) but tests only one alternative temperature (1.0, which improves diversity to 4.77 but wasn't used for RFT training). This leaves open a critical question: can large models be made to generate diverse reasoning paths with appropriate decoding parameters, or is low diversity an inherent consequence of stronger fine-tuning? A follow-up study would systematically sweep temperature (0.7, 0.8, 0.9, 1.0, 1.2), top-p (0.9, 0.95, 1.0), and repetition penalty (1.0, 1.1, 1.2) for LLaMA-33B-SFT, measure distinct paths per question and correctness rate at each setting, and train RFT models on the most diverse configuration that maintains >50% correct paths. If a generation configuration can be found that yields >5 distinct paths per question (matching 7B/13B diversity), then the paper's claim that RFT fails for large models would be a generation-configuration artifact rather than a fundamental limitation. If no configuration achieves this, it would strengthen the paper's overfitting diagnosis and motivate research into SFT regularization techniques (dropout, label smoothing, data augmentation during SFT) that preserve generation diversity.

3. Multi-round RFT: does self-improvement compound or collapse? The paper's RFT is single-round—the RFT model is never used to generate a second round of augmented data. The closely related STaR method (Zelikman et al., 2022) uses multiple rounds, and a direct comparison would answer: does RFT's diversity-driven improvement compound across rounds, or does each round reduce diversity until gains stop? A follow-up would take the LLaMA-7B RFT-U13B model (49.3% accuracy), use it to generate $k=100$ samples per training question, measure the resulting distinct path count, and train a second-round RFT model. The paper's diversity analysis (Figure 7) shows that RFT models generate more diverse reasoning paths at test time than SFT models, which predicts that second-round RFT should have higher diversity than first-round RFT—and thus should provide additional gains. However, the paper's finding that better models generate less diverse data (33B having 2.78 distinct paths vs. 7B's 5.25) predicts the opposite: as the model improves, diversity drops, so second-round gains should be smaller than first-round gains. Resolving this tension—by measuring the distinct-path-per-question ratio of RFT models during generation on the training set, not just at test time—would clarify whether RFT's gains are a one-time boost from injecting external diversity (via multi-model aggregation) or the first step in an iterative self-improvement process. A negative result (second-round RFT provides negligible gain) would be equally informative, establishing that RFT's mechanism is diversity injection from different models, not bootstrapping from the same model.

4. Difficulty-stratified RFT: does augmentation help most where the model is weakest? The paper applies RFT uniformly across all training questions regardless of how hard each question is for the base SFT model. This is a missed opportunity: if RFT works by providing diverse reasoning paths, the marginal value of an additional reasoning path should be highest for questions where the SFT model currently fails (because it hasn't found any correct path) or produces only one strategy (which might be fragile). A follow-up would stratify the GSM8K training set by SFT model accuracy (using the SFT model's pass@1 rate on each training question, measured by sampling multiple solutions), then apply RFT with varying $k$ per difficulty bin. The hypothesis is that easy questions (SFT already solves them 90%+ of the time) need minimal augmentation, while medium-hard questions (SFT solves 20-50% of the time) benefit most from additional reasoning paths. This would directly test whether RFT's gains can be concentrated on the hardest questions, improving cost-effectiveness. A strong experiment would compare uniform $k=100$ RFT against a budget-matched "oracle" RFT that allocates more samples to questions in the lowest SFT-accuracy quartile, measuring whether targeted allocation achieves the same overall accuracy with fewer total generated samples. This connects the paper's work to the broader test-time compute allocation literature and would provide a principled answer to "how many samples should I generate per question?"

5. Replication on MATH: does RFT work when base accuracy is lower? All experiments use GSM8K, where LLaMA-7B SFT achieves 35.9% accuracy. The MATH benchmark (Hendrycks et al., 2021) is substantially harder—a typical 7B model achieves <10% accuracy—and involves more complex reasoning (algebra, geometry, precalculus). Replicating the RFT pipeline on MATH would answer: does RFT require a minimum base accuracy to work, or does it provide proportional gains regardless of absolute performance? The paper's rejection sampling mechanism requires that the SFT model generates some correct reasoning paths during the $k=100$ sampling. If base accuracy is 5%, then on average only 5 out of 100 sampled paths per question will be correct—and the distinct-path count among those 5 correct paths is likely very low (perhaps 1-2). This predicts that RFT gains on MATH would be minimal for 7B models. However, the paper's finding that even small diversity increases help ($k=3$ already provides +2 points on GSM8K, Figure 4) suggests that modest gains might still be possible. A successful replication would measure (a) the distinct paths per question at various $k$, (b) the resulting RFT accuracy, and (c) whether multi-model aggregation (combining samples from differently-trained SFT models) can compensate for low per-model accuracy by increasing the pool of correct paths. This would establish the boundary conditions for RFT's applicability and inform whether practitioners should attempt data augmentation on harder benchmarks or invest exclusively in pre-training improvements.

6. What properties of the pre-training data and objective drive the pre-training loss vs. reasoning performance relationship? Figure 2 shows a striking negative linear correlation between pre-training loss and both ICL and SFT accuracy, but the paper acknowledges this relationship is established only for the LLaMA family on GSM8K. A deeper follow-up would investigate what aspects of pre-training cause this correlation by training multiple models from scratch with controlled variations. For example: train a 7B model on (a) a general web corpus, (b) the same corpus plus math-heavy data (following Lewkowycz et al., 2022), (c) the same corpus with different proportions of code vs. natural language, and measure both pre-training loss (on a held-out general corpus) and GSM8K SFT accuracy. If math-heavy pre-training improves GSM8K SFT accuracy without proportionally improving general pre-training loss, then the loss-accuracy correlation breaks—suggesting that pre-training data composition, not just aggregate loss, matters. This would refine the paper's claim that "pre-training loss is a better indicator than parameter count" by specifying when loss is a reliable indicator (when pre-training data mixtures are similar across models) and when it fails (when models are specialized to different domains). For practitioners choosing between a general-purpose model and a code/math-specialized model of similar size, this distinction is practically important.

Practical Applications and Downstream Use Cases

Budget-constrained math reasoning deployment (e.g., educational technology, tutoring systems). An organization building an automated math tutoring system needs high single-pass accuracy (low latency for student interaction) but has a limited budget for model training and inference. The paper's strongest practical finding is that a LLaMA-7B model with RFT-U13B achieves 49.3% accuracy—matching or exceeding LLaMA-13B SFT (43.0%) and LLaMA2-13B SFT (50.0%)—while using a 7B model that is cheaper to serve (roughly 2× fewer parameters means roughly 2× lower inference latency and cost). The total cost to produce this model is the LLaMA-7B pre-training (already done and open-sourced), plus SFT (~0.6 GPU hours), plus RFT inference from four models (~10 + 10 + 40 + 40 = ~100 GPU hours for 7B and 13B models, per Table 4 scaling), plus RFT training (~9 GPU hours). The total fine-tuning cost is under 200 GPU hours on A100-80GB—less than $500 at typical cloud GPU pricing. For an educational technology company that cannot afford to pre-train models from scratch but can afford modest fine-tuning compute, this recipe provides a near-state-of-the-art 7B math solver at minimal cost. The paper's Table 3 shows this configuration outperforms all other open-source 7B models (ChatGLM2-6B, InternLM-7B) and matches larger models. The key deployment consideration is that the RFT data generation must be done once (offline), after which the model serves with standard 7B inference cost.

Data augmentation for low-resource math problem domains (e.g., non-English math word problems, specialized curricula). A curriculum developer creating math word problems in a specific domain (e.g., financial literacy, agricultural economics) or language (e.g., Hindi, Swahili) has a small set of human-annotated reasoning examples—perhaps 500-1,000—and wants to maximize the accuracy of a fine-tuned model. The paper's supervised data scaling results (Figure 3) show that for LLaMA-7B, accuracy improves log-linearly with data amount in this range (from ~10% at 230 examples to ~35% at 7,473 examples). This predicts that doubling a small dataset from 500 to 1,000 examples should produce a roughly constant accuracy gain, and that RFT could supplement the remaining gap. The multi-model RFT results show that combining rejection samples from different SFT models (e.g., fine-tuned on different random subsets or with different hyperparameters) increases reasoning diversity beyond what a single model can generate (12.84 distinct paths per question vs. 5.25, Table 2). For a low-resource domain, the practitioner could: (1) fine-tune several models on different subsets of the available data, (2) use each to generate $k=100$ candidate solutions for the full training set via rejection sampling, (3) aggregate and deduplicate, and (4) fine-tune a final model on the combined human + machine-generated dataset. The paper's results predict this would substantially outperform a single round of SFT, especially for 7B/13B models where generation diversity is highest. The main cost is inference for data generation, which Table 4 suggests is manageable (~10 GPU hours for a 7B model generating 100 samples per question on a 1,000-question set—linear scaling from the 7,473-question cost).

Inference-cost-sensitive API pipelines (e.g., batch evaluation, data labeling). An organization using LLM APIs for large-scale math problem evaluation (grading student answers, labeling training data, evaluating model outputs) pays per-token for inference. The paper's comparison of maj1@1 (single greedy decode) vs. maj1@100 (sampling 100 times with majority voting) reveals the cost-accuracy tradeoff: for LLaMA-7B RFT-U13B, maj1@1 is 49.3% while maj1@100 is 61.8% (Table 6). The majority-voting approach costs 100× more in inference tokens and latency but provides +12.5 percentage points. The paper's SFT and RFT results on maj1@1 directly inform this decision: a practitioner who can afford only single-pass inference should invest in RFT (which improves maj1@1 from 35.9% to 49.3% for LLaMA-7B, a +13.4 point gain) rather than relying on SFT with majority voting (35.9% maj1@1) or ICL (11.0% for LLaMA-7B, 8-shot). The paper provides the specific numbers to make this cost calculation: RFT-U13B single-pass costs the same as SFT single-pass (same model size) but achieves accuracy that would require roughly 4-8× majority-voted samples from the SFT model to match. For a batch evaluation of 1 million problems, RFT-U13B would cost ~1M inference calls vs. ~4-8M inference calls for SFT with majority voting to reach comparable accuracy, directly translating to cost savings.

When to Prefer This Method

The paper does not articulate an explicit tradeoff against named alternative methods (e.g., "use RFT when X, use CoRE when Y"). Its comparison with CoRE (Zhu et al., 2023) and FCS+PCS (Ni et al., 2023) in Table 3 is across different base models, making it impossible to isolate method superiority from base model quality. The paper's positioning is that RFT is "simpler" (no verifier training, no MCTS) and achieves strong results, rather than that it dominates under specific conditions. The practical recommendations that emerge from the scaling analysis are:

• Prefer RFT (especially multi-model aggregation) when using 7B or 13B base models on GSM8K-style math reasoning. The gains are large (+8 to +13 points, Tables 1 and 5), the cost is negligible relative to pre-training (Table 4), and the single-pass deployment benefit (maj1@1) directly improves the model's practical utility.

• Do not expect RFT with self-generated data to help for 33B+ models. The generation diversity is too low (Table 2: 2.78 distinct paths for 33B vs. 5.25 for 7B), and the resulting accuracy gains are near zero or negative (Table 1: 54.5% RFT vs. 54.6% SFT for 33B). Multi-model aggregation using smaller models as generators partially mitigates this (LLaMA-33B RFT-U33B achieves 57.9%, +3.3 points), but the cost-benefit is marginal at large scale (4,500 GPU hours for 33B RFT inference, per Table 4).

• If pre-training loss is already very low (e.g., LLaMA-65B at 1.62 loss), invest in further pre-training improvements rather than data augmentation. The paper's log-linear SFT curves (Figure 3) show very flat slopes for strong models, and RFT provides no benefit for 65B (59.0% RFT-U13B vs. 59.3% SFT, Table 5). At this scale, the model is near its reasoning ceiling given the existing data and fine-tuning paradigm; higher accuracy requires better pre-training (math-specialized corpora, longer training) rather than more fine-tuning data.

These are empirical guidelines derived from the paper's scaling analysis, not a formal decision rule the paper states as a contribution. The paper's framing emphasizes the scaling relationships themselves, with RFT serving as the vehicle for studying augmented data scaling rather than as a method to be positioned against alternatives.