TTRL: Test-Time Reinforcement Learning

🎯 Pitch

You can train LLMs to solve hard reasoning problems without any answer labels—just by letting the model vote for itself. Qwen-2.5-Math-7B saw a 211% improvement on AIME 2024 using only unlabeled test data, nearly matching the performance of RL trained with ground truth.

1. Executive Summary

This paper introduces Test-Time Reinforcement Learning (TTRL), a method that trains LLMs via RL on unlabeled test data by using majority voting across multiple sampled outputs as a reward signal, converting consensus predictions into rule-based rewards without requiring ground-truth labels. Evaluated across AIME 2024, AMC, MATH-500, and GPQA using models from the Qwen, LLaMA, Mistral, and DeepSeek families, TTRL boosts Qwen-2.5-Math-7B's pass@1 on AIME 2024 by approximately 211% (from 12.9 to 40.2), with an average improvement of 76% across all four benchmarks, while also demonstrating that the trained model's majority-voting performance (avg@64) consistently surpasses the initial model's maj@64 upper bound. The paper further establishes that TTRL's performance approaches that of direct RL on test data with ground-truth labels, though this self-evolution is effective only when the base model possesses sufficient prior knowledge on the target task.

2. Context and Motivation

The Core Problem: RL for Reasoning Without Ground-Truth Labels

The paper addresses a fundamental bottleneck in the current paradigm for training Large Reasoning Models (LRMs): reinforcement learning for reasoning tasks has become remarkably effective, but it remains tethered to labeled data. Recent breakthroughs like DeepSeek-R1 (Guo et al., 2025) and OpenAI's o1 (Jaech et al., 2024) demonstrate that RL—specifically, rule-based RL with verifiable rewards like mathematical answer checking—can elicit extended chain-of-thought reasoning and dramatically improve performance on challenging benchmarks. However, these methods require datasets where the correct answer is known, so that a verifier can compare the model's output against a ground-truth label and assign a binary correct/incorrect reward.

This creates a severe scalability problem. As the authors point out in Section 1:

"complex and unlabeled questions continuously emerge, posing significant challenges. For instance, o3 solves only 4% of problems on the recently released ARC-AGI-2 benchmark."

The arc of progress in AI reasoning is pushing toward harder and harder benchmarks—ARC-AGI-2 being one recent example—where by definition, ground-truth labels are either unavailable (the benchmark is designed to test generalization to novel problems) or prohibitively expensive to produce at scale. If the most effective training method (RL) requires labels, and the hardest problems are precisely the ones we lack labels for, there is a fundamental mismatch between the capabilities we want to build and the training signals we have access to.

This isn't just a benchmark problem. In any real-world deployment where an LLM encounters genuinely novel questions—scientific discovery, mathematical research, open-ended reasoning—there are no pre-existing answer keys. The model must operate without the crutch of labeled verification. Yet the current RL-for-reasoning pipeline is entirely built on the assumption that such labels exist.

Why This Problem Matters: The "Era of Experience" and Self-Evolution

The paper explicitly frames its motivation within a broader intellectual shift in AI, citing Silver & Sutton (2025) on the "era of experience":

"Silver & Sutton (2025) has recently advocated for a transition to the 'era of experience,' emphasizing the limitations of existing AI systems that rely heavily on human supervision, as well as the importance of enabling models to self-evolve through experience."

This framing is not merely rhetorical. It identifies two distinct but connected motivations:

1. Adaptation to test-time data (the focus of this paper). When a model encounters a novel distribution at test time—a new benchmark, a new class of problems, domain-specific queries it wasn't trained on—the standard approach is to hope that pretraining and post-training generalization suffice. But as models push toward increasingly difficult reasoning tasks, this hope becomes increasingly tenuous. The ability to adapt at test time, using only the unlabeled test inputs themselves, would enable models to tackle problems that no amount of static pretraining could cover.

2. Training on external unlabeled data (the broader vision). Beyond test-time adaptation, if RL can work without labels, then any unlabeled corpus of reasoning problems becomes potential training data. The Internet contains vast quantities of mathematical problems, scientific questions, and reasoning challenges without annotated solutions. Unlocking this data for RL training would dramatically expand the scale at which reasoning models can be improved, potentially shifting the bottleneck from labeled data availability to raw compute.

The paper's title—Test-Time Reinforcement Learning—makes clear which of these two the current work addresses, but the introduction positions this as a stepping stone toward the larger goal of unsupervised self-improvement.

Prior Approaches and Their Shortcomings

The paper identifies several existing lines of work that touch on related ideas, each with specific limitations that TTRL aims to overcome:

RL for reasoning requires labels (the dominant paradigm). Methods like GRPO (Shao et al., 2024) and the training recipe behind DeepSeek-R1 (Guo et al., 2025) use rule-based verifiers that check whether the model's final answer matches a ground-truth label. This works brilliantly on labeled math datasets (AIME, MATH, etc.) but leaves the model helpless on unlabeled data. The authors note:

"these methods typically train LLMs only on supervised training data, while inference involves generating extended CoT reasoning on unseen test problems."

In other words, there's a sharp disconnect: RL training happens on data where answers are known, then the trained model is expected to generalize at inference to problems where answers are not known. But the RL process itself never touches those unseen problems. TTRL proposes to close this loop by doing the RL on the test data itself, without labels.

Test-Time Training (TTT) exists but hasn't used RL. The paper carefully distinguishes its approach from prior TTT work (Section 5.1, Appendix C). TTT—originally from Sun et al. (2019)—adapts model parameters at test time using auxiliary self-supervised losses. Recent work has applied TTT to LLMs for abstract reasoning (Akyürek et al., 2024) and video understanding (Wang et al., 2025a). But the paper states:

"the integration of test-time scaling with reinforcement learning remains largely underexplored."

This is a key gap. TTT methods typically use self-supervised objectives like masked token prediction or contrastive losses—objectives that are only loosely related to the end task. They don't directly optimize for task performance in the way RL does. By bringing RL into the TTT framework, TTRL aims to achieve the best of both worlds: the adaptive, on-the-fly nature of TTT combined with the task-directed optimization of RL.

Self-training and pseudo-labeling exist but are capped by initial model quality. Traditional self-training methods (Huang et al., 2022) work by generating pseudo-labels from the model itself (e.g., via majority voting), then doing supervised fine-tuning (SFT) on those pseudo-labels. The paper identifies a fundamental limitation:

"This upper bound is also the performance limit of traditional self-training methods (Huang et al., 2022), which select self-generated CoT through majority voting for supervised fine-tuning (SFT)."

In SFT-based self-training, the model can only learn to imitate what it already (collectively) produces. If the majority vote among $N$ samples gets the right answer 40% of the time, SFT on those pseudo-labels can at best approach 40%. The model is bounded by the quality of its own initial pseudo-labels. TTRL's key claim—and a surprising empirical finding—is that it can exceed this bound through RL's exploration dynamics.

Self-play and self-rewarding methods target different domains. The paper acknowledges prior work on self-rewarding (Yuan et al., 2025) and self-play training (Chen et al., 2024) for unlabeled data. However, it notes these focus on "open-domain instruction following ... rather than mathematical reasoning" and often use "preference-based optimization strategies such as DPO ... instead of online reinforcement learning algorithms." The distinction matters: mathematical reasoning has a verifiable structure (answers are either correct or incorrect) that makes rule-based rewards meaningful, whereas open-domain instruction following requires learned reward models or human preferences. TTRL exploits this verifiable structure via majority voting, even without ground-truth labels.

How TTRL Positions Itself

TTRL positions itself at the intersection of three previously separate research threads:

Test-Time Training (TTT) — the idea that models should adapt parameters at inference time to handle distribution shift.
Test-Time Scaling (TTS) / inference-time compute — the idea that allocating more compute at inference (e.g., via majority voting, best-of-N, beam search) improves performance.
Reinforcement Learning for reasoning — the idea that optimizing a policy against verifiable rewards elicits reasoning capabilities.

The novelty is the synthesis: use TTS techniques (specifically, repeated sampling and majority voting) to estimate rewards, then use those estimated rewards to drive RL-based TTT on the unlabeled test data. Figure 2 illustrates this explicitly, showing how the majority voting output from multiple samples becomes a pseudo-label, which then generates rule-based rewards that drive policy optimization.

The paper is careful not to claim this is an entirely new RL algorithm. Section 4.3 states:

"At the algorithmic level, TTRL is not fundamentally different from existing RL algorithms"

Rather, the contribution is the reward estimation mechanism—using majority voting as a bridge from unlabeled data to verifiable rewards—and the empirical demonstration that this mechanism is stable enough to drive effective RL training across diverse models and benchmarks.

A subtle but important positioning point: unlike prior self-training work that operates offline (generate pseudo-labels, then SFT), TTRL is explicitly online. The model generates samples, estimates labels, gets rewards, and updates its parameters—all during a single continuous process. This online nature is critical to the paper's central empirical finding: that the model can "lift itself up by its own bootstraps," because as the policy improves, the quality of the majority-voting pseudo-labels also improves, creating a virtuous cycle (which the paper visualizes in Figure 6 and calls out in the Q2 analysis in Section 4.2).

The Specific Gap: No Prior Work on Unsupervised RL for Test-Time Reasoning

The paper's core claim to novelty is that "the use of TTT for reasoning has not been previously explored" (Section 3.1, under Baselines). By combining TTT, TTS, and RL into a single framework, TTRL fills a gap that—despite the maturity of each individual component—had not been previously addressed. The concurrent works cited in Section 5.2 (Xu et al., 2025; Zhang et al., 2025b; Zhao et al., 2025) suggest this is a rapidly emerging area, but the paper positions its majority-voting-based reward mechanism as a distinctive approach that "mitigates reward hacking" compared to alternatives.

The problem is well-motivated by a concrete, measurable disconnect: state-of-the-art reasoning models are trained on data with labels and evaluated on data that also has labels (for benchmarking), but in the real world, the hardest and most important problems don't come with answer keys. TTRL is a direct attempt to bridge that gap.

3. Technical Approach

3.1 Reader Orientation

TTRL is a training procedure that takes a pre-trained language model and a set of unlabeled test questions, then improves the model's accuracy on those exact questions through reinforcement learning—without ever knowing the correct answers. The system solves the problem of "how do you compute rewards for RL when you don't have ground-truth labels?" by making the model generate many candidate answers for each question, letting those candidates vote on what the answer should be, and treating the majority-vote winner as if it were the correct label for reward calculation.

3.2 Big-Picture Architecture (Diagram in Words)

The TTRL system has three major stages that execute in a continuous loop, as shown in Figure 2:

Label Estimation (green background in Figure 2). For each unlabeled test question $q$ , the current policy model (the LLM being trained) generates $N$ candidate answers by repeated sampling. These candidates are processed through an answer extractor that pulls out the final answer from each chain-of-thought response. Majority voting is applied to the extracted answers: the most frequently occurring answer becomes the estimated label $y^*$ . This stage has no access to ground-truth—it relies entirely on the model's own collective judgment.
Reward Calculation (grey background). Each of the $N$ candidate answers $\hat{y}_i$ is compared against the estimated label $y^*$ using a rule-based verifier. If a candidate's extracted answer matches the majority-voted label, it receives a reward of 1; otherwise, it receives 0. These binary rewards $R(\hat{y}_i, y^*)$ are the training signal for RL, despite being derived entirely from the model itself rather than from ground-truth.
Policy Optimization (dashed feedback arrow). The rewards are fed into a standard RL algorithm—GRPO by default, though the paper shows PPO and PRIME also work—which updates the model parameters $\theta$ to increase the probability of generating answers that receive positive rewards. The updated model is then used in the next iteration of Label Estimation, creating an online learning loop where improving the policy also improves the quality of the pseudo-labels.

Information flows in a cycle: unlabeled questions → policy model → $N$ candidate answers → majority voting → estimated labels → rule-based reward computation → GRPO parameter updates → improved policy model → (repeat). The "Test-Time" aspect means this entire loop executes on the test data itself—the data the model will ultimately be evaluated on—rather than on a separate training set.

3.3 Roadmap for the Deep Dive

First, the formal RL objective (Equations 1 and 2), which defines what TTRL optimizes and how the unlabeled setting changes the standard RL formulation—specifically, by replacing ground-truth rewards with consensus-derived rewards.
Second, the majority voting reward function (Equation 3 and Listing 1), which is the heart of TTRL—how the model converts its own sampled outputs into a training signal, why a simple binary match against the majority label works, and what the pseudo-code actually does step by step.
Third, the GRPO training configuration and the online rollout procedure, including the critical vote-then-sample strategy that reduces computational cost, the specific hyperparameters (learning rate, temperature, generation budget, episodes), and how these interact with the unsupervised reward signal.
Fourth, the training metrics (Appendix B) that enable checkpoint selection and monitoring without ground-truth access, plus the ground-truth-dependent metrics used only for analysis.
Fifth, the design choices and their justifications—why majority voting over alternative pseudo-labeling methods, why online RL over offline SFT, why GRPO as the base algorithm, and why the specific rollout sizes were chosen.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that majority voting across multiple sampled outputs can serve as a sufficiently reliable reward estimator to drive effective reinforcement learning on unlabeled data, and that this process can exceed the performance ceiling implied by the initial model's voting accuracy due to the online, self-reinforcing nature of RL exploration.

The RL Objective with Consensus-Derived Rewards

TTRL reformulates the standard RL-for-reasoning objective to operate without ground-truth labels. In standard rule-based RL for math reasoning (as in DeepSeek-R1), the reward for generating an answer $y$ given a question $x$ is $r(y, y_{\text{gt}})$ , where $y_{\text{gt}}$ is the known correct answer and the verifier checks whether the extracted answer from $y$ matches $y_{\text{gt}}$ . TTRL replaces $y_{\text{gt}}$ with $y^*$ , a consensus label derived from the model's own sampled outputs.

The core optimization is:

$\max_{\theta} \mathbb{E}_{y \sim \pi_\theta(\cdot | x)} [r(y, y^*)]$

where $\pi_\theta(y | x)$ is the policy (the LLM parameterized by $\theta$ ) that maps an input prompt $x$ to a distribution over outputs $y$ , and $r(y, y^*)$ is the reward function that scores an individual sampled output $y$ against the consensus label $y^*$ .

What it computes: This objective says "adjust the model parameters $\theta$ to maximize the expected reward of outputs sampled from the policy, where reward is determined by agreement with the consensus label $y^*$ ." The consensus label $y^*$ is itself a function of multiple samples from $\pi_\theta$ , making this a self-referential objective: the target the model is optimizing toward is generated by the model itself.

Why this form: Standard RL for reasoning maximizes $\mathbb{E}[r(y, y_{\text{gt}})]$ , which requires $y_{\text{gt}}$ to be known. By substituting $y^*$ for $y_{\text{gt}}$ , TTRL preserves the same mathematical structure—an expectation over policy samples of a scalar reward—while removing the dependence on external labels. The key risk is that $y^*$ may be wrong, injecting noise into the reward signal. The paper's central empirical claim is that RL is robust enough to this noise, and indeed that the noise characteristics (analyzed through the "Lucky Hit" phenomenon in Section 4.2) actually provide a surprisingly clean training signal.

The parameter update follows standard policy gradient ascent:

$\theta \leftarrow \theta + \eta \nabla_{\theta} \mathbb{E}_{y \sim \pi_\theta(\cdot | x)} [r(y, y^*)]$

where $\eta$ is the learning rate. This is the generic form; in practice, TTRL uses GRPO (Group Relative Policy Optimization), which estimates this gradient from groups of sampled outputs per question, as described in the implementation section below.

What this computes: The gradient of the expected reward with respect to the policy parameters, scaled by the learning rate, is added to the current parameters. Outputs that receive higher rewards (agreement with the consensus label) have their log-probability increased; outputs that receive lower rewards have their log-probability decreased.

Why this form: This is the standard policy gradient update, chosen because it is the most well-understood and widely deployed approach for RL on language models. The paper is not proposing a new RL algorithm but rather a new way to compute the reward $r(y, y^*)$ that feeds into existing algorithms. The authors deliberately keep the RL mechanics standard to isolate the effect of the reward estimation mechanism.

The Majority Voting Reward Function (Equation 3, Listing 1)

The majority voting reward function is the mechanism that converts unlabeled test data into a usable RL training signal. It operates in three sequential steps, as shown in Listing 1:

Step 1: Generate candidate outputs. Given a question $x$ , the model samples $N$ complete outputs (chain-of-thought reasoning traces plus final answers). In the paper's implementation, $N = 64$ for label estimation during rollout (the full set used for voting), with a random subset of 32 used for the actual training update (the vote-then-sample strategy described later). The generation uses a temperature of 0.6 (or 1.0 for Qwen2.5-Math and LRM models) and top-p of 0.95.

Step 2: Extract answers and find the majority. An answer extractor processes each of the $N$ outputs to pull out the final answer string (e.g., a number, an expression, a multiple-choice letter). These extracted answers form a multiset $P = \{\hat{y}_i\}_{i=1}^N$ . Majority voting is applied: the most frequently occurring answer in $P$ becomes the estimated label $y^*$ . Formally, $y^* = \arg\max_{a} \text{count}(a \in P)$ , where $\text{count}(a \in P)$ is the number of times answer $a$ appears among the $N$ candidates.

Step 3: Compute per-output rewards. Each individual output $\hat{y}_i$ receives a binary reward by comparing its extracted answer against the consensus label $y^*$ :

$R(\hat{y}_i, y^*) = \begin{cases} 1, & \text{if } \hat{y}_i = y^* \\ 0, & \text{otherwise.} \end{cases}$

What this equation computes: A binary indicator: 1 if the output's extracted answer exactly matches the majority-voted answer, 0 if it differs. This is a rule-based reward—no learned reward model, no human annotation, just exact string matching against a consensus target.

Why this form: The binary match/non-match structure mirrors exactly the rule-based verifiers used in standard labeled RL for math (where the comparison is against ground-truth). This means that from the RL algorithm's perspective, the reward format is identical regardless of whether the label comes from ground-truth or from majority voting—only the accuracy of the label changes. Using binary rewards rather than continuous scores is deliberate: it provides a crisp signal that the model can optimize toward without needing to interpret subtle gradations of correctness. For mathematical reasoning where answers are either right or wrong, this binarization is appropriate. The key insight (developed in Section 4.2) is that even when $y^*$ is wrong, the binary reward function can still produce correct rewards for most individual outputs through the "Lucky Hit" mechanism: if an output's answer differs from both the true label AND the estimated label, it still correctly receives a reward of 0.

The pseudo-code in Listing 1: The Python implementation reveals several operational details:

extract_answer(output) is called on each raw model output to isolate the answer portion from the chain-of-thought trace. The paper doesn't specify the exact extraction logic, but it's the standard approach of pattern-matching for boxed expressions, final lines, or answer indicators.
Counter(answers) tallies the frequency of each unique answer.
counts.most_common(1)[0] extracts the most frequent answer. In case of ties, Python's most_common returns an arbitrary element among those with equal maximum frequency.
The reward assignment is a simple list comprehension comparing each extracted answer to the majority answer.

Critical property: The reward function is non-parametric and zero-cost beyond the generation itself. There is no separate reward model to train, no calibration to perform, and no distribution shift between the reward signal and the policy model (since both derive from the same parameters). The paper argues this self-consistency property helps prevent reward hacking because the reward signal evolves with the policy rather than being a fixed, gameable target.

The GRPO Implementation and Online Rollout Procedure

TTRL uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as its default RL algorithm, applied independently to each benchmark. GRPO is a variant of policy gradient methods designed specifically for reasoning tasks. Unlike PPO, which requires a separate value function model, GRPO estimates advantages by comparing rewards within a group of outputs sampled for the same prompt—hence "Group Relative."

The vote-then-sample strategy. A critical computational optimization: for each prompt during training, the model first samples 64 responses at temperature 0.6 (or 1.0 for Qwen2.5-Math and LRMs) for the purpose of majority voting and label estimation. From these 64, it then downsamples 32 responses to use for the actual training update. The paper states:

"Evidence shows that our vote-then-sample strategy effectively reduces computational costs while still achieving strong performance."

What this means operationally: The 64-sample pool provides a more accurate majority-vote label (since more samples give a better estimate of the true majority), but only 32 of those samples need their log-probabilities computed and gradients backpropagated. The remaining 32 samples serve only to improve the label quality and are discarded for gradient computation. This is a pragmatic engineering decision: label quality scales with $\sqrt{N}$ (the standard error of a proportion), so using 64 samples for voting but only 32 for training gives better labels at lower total training cost than using 64 for both or 32 for both.

Hyperparameters (from Section 3.1):

Learning rate schedule: Cosine schedule with a peak value of $5 \times 10^{-7}$ . This is notably low compared to typical fine-tuning learning rates ( $10^{-5}$ to $10^{-4}$ ), reflecting the sensitivity of RL training to step size—too large an update can collapse the policy.
Optimizer: AdamW for the policy model.
Rollout temperature: 0.6 for most models, 1.0 for Qwen2.5-Math base models and LRMs. The higher temperature for math-specialized models encourages more exploration, which is important for discovering correct reasoning paths that the model doesn't initially produce.
Top-p: 0.95 for all models.
Maximum generation length: 3,072 tokens for standard models, 32,768 tokens for LRMs (which produce longer chain-of-thought traces).
Training episodes: Varies by dataset size and difficulty—10 episodes for MATH-500, 30 for AMC, and 80 for AIME 2024. The rationale: smaller and more difficult datasets need more passes through the data to achieve sufficient exploration and policy improvement. AIME 2024 has only 30 questions, so 80 episodes means each question is seen approximately 80 times during training (though the sampling is stochastic each time).
Hardware: 8 × NVIDIA A100 80GB GPUs for all experiments.

Episodes vs. epochs. The paper uses "episodes" rather than "epochs," terminology borrowed from RL. An episode in this context corresponds to one complete pass over the dataset where for each question, the model generates rollouts, computes rewards, and updates parameters. For AIME 2024 with 30 questions and 80 episodes, the model performs 2,400 total question-rollout-update cycles.

Standard RL vs. TTRL RL. A crucial distinction: in standard GRPO training for math reasoning, the dataset is a set of questions with known answers, and the model sees these questions repeatedly during training (the usual RL fine-tuning paradigm). In TTRL, the "dataset" is the test benchmark itself—the same questions the model will be evaluated on. The model never sees answers, only its own majority-voted pseudo-labels. This is what makes it "test-time": the adaptation happens directly on the evaluation data.

Training Metrics Without Ground-Truth Access (Appendix B)

A practical challenge for TTRL is that without ground-truth labels, standard training monitoring—tracking accuracy, reward accuracy, etc.—is impossible. The paper introduces a set of ground-truth-free metrics for monitoring training progress and selecting checkpoints, and a complementary set of ground-truth-dependent metrics used only for post-hoc analysis.

Ground-truth-free metrics (available during actual deployment):

Entropy: The uncertainty of the model's output distribution. The paper tracks this as a diagnostic for training stability—persistently high or non-decreasing entropy indicates the policy is not converging (as seen in the failed training runs in Figure 11). In successful TTRL runs, entropy gradually decreases as the model becomes more confident in its answers.
Majority Voting Reward: The average reward received by sampled outputs under the majority-voting reward function. This is the direct training signal and serves as the primary indicator of whether training is progressing. If this reward increases over episodes, it means the model is increasingly agreeing with its own consensus.
Majority Ratio: The frequency of the most common answer within a rollout of $N$ samples. A high majority ratio (e.g., 0.8 when 80% of samples agree) indicates the model has converged on a consistent answer for a question; a low ratio (e.g., 0.2) indicates high uncertainty or scattered outputs. The paper uses this to track whether the model is becoming more decisive over training.

Ground-truth-dependent metrics (for analysis only, not available during deployment):

Label Accuracy (maj@n): Whether the estimated label (from majority voting) matches the true ground-truth answer. This is the accuracy of the pseudo-labeling process. The paper reports this is surprisingly low on hard benchmarks—around 37% initially on AIME 2024 for Qwen2.5-Math-7B (Figure 9)—yet training still succeeds.
Reward Accuracy: The proportion of majority-voting-derived rewards that match the rewards that would have been computed from ground-truth labels. Put differently: if we computed rewards using the true answer, what fraction of the majority-voting rewards would be identical? This metric isolates the quality of the reward signal independent of label accuracy. The paper finds this can be dramatically higher than label accuracy—92% vs. 37% on AIME 2024 (Figure 9)—due to the Lucky Hit phenomenon.
Ground-Truth Ratio: The frequency of the true correct answer within the model's $N$ sampled outputs. This measures whether the correct answer is even present in the model's generation distribution. If this is 0, no amount of voting can produce the right label, and TTRL cannot help (this connects to the prior knowledge requirement analysis in Section 4.3).

Checkpoint selection strategy. The paper states these metrics "inform the selection of the optimal checkpoint," but doesn't specify the exact protocol. Presumably, the Majority Voting Reward and Entropy trends are used to identify when training has plateaued or begun to degrade (entropy collapse, reward stagnation), and the checkpoint from that point is selected. In practice, the paper evaluates final performance after the full episode budget (10, 30, or 80 episodes) for the main results, so the checkpoint selection mechanism may be implicit in using the final state.

Design Choices and Their Justifications

Why majority voting rather than a learned reward model? A learned reward model (PRM or ORM) would require training on labeled data, which defeats the purpose of operating without labels. Even if a reward model were trained on separate labeled data and applied to unlabeled test data, there would be distribution shift between the reward model's training distribution and the test questions. Majority voting has the advantage of being self-contained: the "reward model" is the model itself, so there is zero distribution shift, zero additional training cost, and zero dependence on external labels. The paper sees this self-consistency as a form of robustness against reward hacking—a model cannot easily exploit its own consensus mechanism because the consensus evolves with the model.

Why online RL rather than offline SFT on pseudo-labels? The paper explicitly contrasts TTRL with traditional self-training (Huang et al., 2022), which uses pseudo-labels for supervised fine-tuning. The key limitation of SFT-based self-training is that it is bounded by the initial model's voting accuracy: if the initial model's majority vote is correct only 40% of the time, SFT on those pseudo-labels can at best approach 40% accuracy. TTRL's online RL approach breaks this bound because the policy improves during training, which improves the quality of pseudo-labels, which further improves the policy. This virtuous cycle is only possible with online learning. Additionally, the paper cites Chu et al. (2025) for the finding that "SFT memorizes, RL generalizes"—RL tends to produce more robust improvements than SFT on the same data.

Why GRPO rather than PPO? The paper shows GRPO, PPO, and PRIME all work (Figure 5), indicating the choice of RL algorithm is not critical. GRPO is selected as the default because it is simpler than PPO (no value function needed) and was specifically designed for the group-based reasoning setting where multiple samples per question are naturally available. In TTRL, the model already generates $N = 64$ samples per question for majority voting, which directly provides the groups that GRPO requires for advantage normalization. This makes GRPO a natural fit architecturally.

Why 64 samples for voting and 32 for training? The paper doesn't provide an ablation on these numbers, but the reasoning can be inferred: label quality (majority voting accuracy) improves with the number of samples, with diminishing returns. 64 provides a reasonable balance between label accuracy and generation cost. Using all 64 for training would double the memory and compute for gradient computation relative to 32, with relatively small improvement in the training signal (since the extra samples would have similar reward distributions to the first 32). The 64/32 split is thus a pragmatic cost-quality tradeoff.

Why temperature 0.6 vs. 1.0? The higher temperature (1.0) for math-specialized models (Qwen2.5-Math) and LRMs reflects the need for more exploration in models that already have strong priors. A math-specialized model already tends to produce structured, mathematically coherent outputs; higher temperature encourages it to explore alternative reasoning paths that might differ from its default. For general-purpose models, the default diversity at temperature 0.6 is sufficient. The paper's failure analysis (Figure 11) shows that temperature 1.0 can cause persistently high entropy and training instability in some settings, suggesting this is a sensitive hyperparameter.

Why different episode counts per benchmark? AIME 2024 has only 30 questions, so with 64 samples per question per episode, the model sees only 1,920 samples per episode. To accumulate enough training signal for effective RL, many more episodes are needed—hence 80. MATH-500 has 500 questions, providing 32,000 samples per episode, so 10 episodes provides 320,000 training samples, which is sufficient. AMC sits in between. This per-dataset episode tuning is a practical necessity for RL on small test sets.

Why the cosine learning rate schedule? Cosine schedules are standard in LLM fine-tuning because they provide a smooth decay that avoids the sharp drops in learning rate that step-based schedules can cause. The paper uses an unusually low peak learning rate ( $5 \times 10^{-7}$ ) compared to typical fine-tuning ( $10^{-5}$ to $10^{-4}$ ). This is because RL updates are higher-variance than SFT updates (due to stochastic sampling of outputs and rewards), and larger steps can easily push the policy into degenerate regions where it produces nonsensical outputs or collapses to a single answer. The conservatism of the learning rate is a direct response to the noise in the majority-voting reward signal.

Why not filter data by difficulty? The paper acknowledges in Section 4.3 that TTRL fails when the model lacks prior knowledge on the target task. A natural mitigation would be to filter questions by estimated difficulty, apply TTRL only to those within the model's capability range, and then train on harder ones as the model improves (curriculum learning). The paper does not implement this, noting it as a limitation: "TTRL does not incorporate mechanisms such as data filtering to support curriculum learning." The decision to apply TTRL to all questions indiscriminately simplifies the method and tests it in the hardest possible setting (where some questions are genuinely beyond reach), but leaves performance on the table.

The Role of Test-Time Scaling (TTS) Within TTRL

TTRL's label estimation step is essentially an application of test-time scaling (TTS)—specifically, parallel-generation majority voting—during training rather than during evaluation. This creates an interesting duality: TTS techniques that are normally used to improve inference accuracy (e.g., sampling 64 answers and taking the majority vote) are repurposed as a training signal generator.

The paper distinguishes this from standard test-time inference (TTI) in Appendix C. In TTI, majority voting is an inference strategy: you generate $N$ answers, vote, and output the winner as the final answer. The model parameters never change. In TTRL, majority voting serves a dual purpose: during training, it provides the pseudo-labels that drive RL updates; during evaluation, the improved model can again use majority voting (now with its enhanced capabilities) for even better inference performance.

This framing connects TTRL to the broader test-time compute literature (Snell et al., 2024) without conflating training and inference. The compute spent on 64-sample rollouts during training is a training cost; the compute spent on 64-sample majority voting at evaluation is an inference cost. TTRL shows that the former generates lasting improvements that make the latter more effective.

The Answer Extraction Step

The paper mentions an answer extractor (extract_answer in Listing 1) but does not specify its implementation. In standard math reasoning benchmarks, answer extraction typically involves:

For MATH and AIME: extracting content within \boxed{} LaTeX commands, which is the conventional format for final answers.
For AMC: extracting the final stated answer choice or expression.
For GPQA: extracting the final multiple-choice selection.

The answer extractor's reliability is critical: if it fails to extract the correct answer string from a chain-of-thought trace, or if it extracts a partial answer, the majority voting and reward computation become corrupted. The paper implicitly assumes the extractor works well enough, since extraction failures would show up as reduced reward accuracy. The fact that reward accuracy reaches 92% on AIME 2024 despite label accuracy of only 37% suggests the extractor is functioning—incorrect extractions would degrade reward accuracy regardless of the Lucky Hit effect.

Summary of the Training Loop

Bringing everything together, a single TTRL training episode for one benchmark proceeds as follows:

For each question in the benchmark:
- Sample 64 complete outputs (chain-of-thought + answer) from the current policy model at temperature 0.6 (or 1.0).
- Extract answers from all 64 outputs.
- Compute the majority-voted answer.
- Downsample 32 outputs for training.
- For each of the 32 training outputs, assign a binary reward: 1 if its extracted answer matches the majority-voted answer, 0 otherwise.
- Feed the 32 (output, reward) pairs into GRPO for a parameter update.
Track the ground-truth-free metrics (entropy, majority voting reward, majority ratio) to monitor training health.
Repeat for the specified number of episodes (10, 30, or 80).
After training, evaluate the final model on the same benchmark using the standard pass@1 protocol (16 samples with temperature 0.6, top-p 0.95).

The key distinction from standard RL training is that Step 1 never consults ground-truth labels—the "correct answer" for reward computation is always the model's own consensus. This is what makes the method "unsupervised" and "test-time": it operates entirely on the unlabeled test data, using only the model's own outputs as the source of supervision.

4. Key Insights and Innovations

Innovation 1: Majority Voting as a Training Signal, Not Just an Inference Strategy

The field has long recognized majority voting (or self-consistency, Wang et al., 2022) as an inference-time technique: generate multiple answers, take the most common one, and get better accuracy than any single sample. TTRL repurposes this mechanism from an evaluation trick into a training signal generator. This is a conceptual pivot, not a new algorithm: the same majority voting that practitioners use to boost benchmark scores at evaluation time becomes the reward function that drives RL parameter updates during test-time training.

What prior work did. Prior self-training methods (e.g., Huang et al., 2022) used majority voting to produce pseudo-labels, then did supervised fine-tuning (SFT) on those labels. This is fundamentally an offline process: generate labels once, train once, stop. The pseudo-label quality is frozen at whatever the initial model could produce. And SFT on self-generated labels can at best approach the voting accuracy of the initial model — the model is learning to imitate its own aggregate behavior, which creates a hard performance ceiling.

Established RL-for-reasoning methods (DeepSeek-R1, GRPO-based training) used rule-based verifiers against ground-truth labels, producing clean binary rewards that are correct by construction. This works brilliantly but requires knowing the right answer — a luxury unavailable for genuinely novel problems.

What TTRL does differently. TTRL uses majority voting as an online reward generator within an RL loop. The key shift is temporal: the model generates samples, votes to produce pseudo-labels, computes rewards, updates parameters, and then the updated model generates new samples in the next episode, producing potentially better pseudo-labels. The quality of the training signal is not fixed — it co-evolves with the policy. This converts a static bottleneck into a dynamic, self-reinforcing process.

Significance beyond the numbers. The idea that a model's own collective judgment can serve as a training signal for individual improvement is philosophically interesting. It suggests that the ensemble (the distribution of sampled outputs) contains information that no single output captures, and that RL can extract and internalize this aggregate wisdom into the policy itself. The model learns not just to produce answers, but to produce answers that its own future self would agree with — a form of self-consistency training that is more sophisticated than simply imitating past majority votes.

Evidence that this matters. Figure 6 is the smoking gun: as training progresses, both pass@1 and maj@16 accuracy rise together. In an offline SFT self-training paradigm, maj@16 would be constant (the initial model's voting accuracy is fixed) and pass@1 could at best approach it. In TTRL, avg@16 (the average of 16 samples, equivalent to the quality of the model's single-sample performance) surpasses the initial model's maj@16 by over 20 points. The training signal itself is improving because the model is improving. This virtuous cycle would be impossible under any offline approach.

This is a fundamental reframing of what test-time compute can do. Prior work on test-time scaling (Snell et al., 2024) treated compute as something you spend at inference to get better answers from a frozen model. TTRL shows that the same compute, when fed back through RL, can permanently improve the model. The inference budget becomes a training budget, with lasting effects.

Innovation 2: The "Lucky Hit" Phenomenon — When Noisy Labels Produce Clean Rewards

The most counterintuitive empirical finding in this paper is that TTRL works well even when the majority-voted pseudo-labels are mostly wrong. On AIME 2024 with Qwen2.5-Math-7B, the initial label accuracy (how often the majority vote matches ground-truth) is only 37%, yet the reward accuracy (how often individual sample rewards match what ground-truth rewards would have been) is 92% (Figure 9). This disconnect is not a bug — it is the central mechanism that makes TTRL viable, and the paper names it "Lucky Hit."

What prior work assumed. The standard intuition — reflected in the design of most self-training and pseudo-labeling methods — is that label quality is the bottleneck. If you train on noisy labels, you get noisy results. This assumption leads to extensive effort on label filtering, confidence thresholding, and curation. It also explains why prior self-training work treated the initial model's voting accuracy as a performance ceiling: if labels are only right 40% of the time, how could the model ever exceed 40%?

The Lucky Hit mechanism. The insight is that for rule-based verifiers that work by comparison (does predicted answer match label?), the reward for an individual output can be correct even when the comparison target is wrong. Specifically: if an output's extracted answer differs from both the true answer AND the estimated label, the verifier correctly assigns a reward of 0 (negative). The only way a mis-estimated label produces an incorrect reward is when the output happens to match the wrong label — and when the model's outputs are highly scattered (many different wrong answers), this is rare.

Figure 10 illustrates this with a clean toy example: estimated label is 2, true label is 3, the model's eight predictions are {1, 1, 2, 2, 2, 4, 5, 6}. Only the three outputs that predict 2 receive an incorrectly positive reward (they match the wrong label). The other five receive correctly negative rewards (they don't match either label). Reward accuracy is 62.5% even though label accuracy is 0%. On AIME 2024, with 64 samples and scattered incorrect answers, this effect is amplified: "the most frequently predicted answer accounts for only 16.6% of all predictions" (Section 4.2), meaning 83.4% of outputs predict something other than the (wrong) majority answer, and most of those receive correctly negative rewards.

A paradox with deep implications. The paper observes that "the poorer the model's performance, the more mistakes it tends to make, which paradoxically leads to more accurate reward estimation." This is genuinely surprising: it means TTRL's reward signal becomes cleaner when the model is worse, because scattered errors mean fewer outputs accidentally match the wrong label. As the model improves and converges on a single wrong answer (rather than many scattered wrong answers), reward accuracy could actually decrease before label accuracy catches up. This creates an interesting training dynamic where the initial phase of learning is powered by a surprisingly clean signal, precisely because the model is initially incompetent.

Significance. The Lucky Hit concept is not just an explanation for why TTRL works — it is a diagnostic concept that could generalize to any setting where binary comparison-based verifiers are used with self-estimated labels. It suggests that the relevant metric for self-supervised RL is not label accuracy (which is what the field has traditionally focused on) but reward accuracy, and that these two can be dramatically decoupled when outputs are diverse. This reframes the design space: rather than trying to maximize label accuracy (which may be impossible for very hard problems), practitioners should focus on ensuring output diversity so that the Lucky Hit effect can operate.

This is a new kind of insight — not a method, but a mechanism explaining why unsupervised RL for reasoning is possible at all. It is the conceptual counterpart to the observation in standard RL that reward noise is tolerable; here, the structure of the noise is not random but systematically biased toward correct negative rewards, making it far more benign than uniform label noise would be.

Innovation 3: Surpassing the Self-Training Upper Bound Through Online RL

Prior self-training methods (pseudo-labeling + SFT) operate under a clear theoretical ceiling: the model trained on self-generated labels can at best approach the accuracy of those labels. If the initial model's majority voting gets 40% of questions right, SFT on those pseudo-labels cannot exceed 40%. TTRL demonstrates that this ceiling is an artifact of the offline SFT paradigm, not a fundamental limit of self-supervised learning.

Why offline SFT is bounded. In SFT-based self-training, the training targets are fixed after the initial label generation step. The model learns to imitate its own previous aggregate behavior. Even if training causes the model's internal representations to shift, the labels it's optimizing toward remain static. There is no mechanism for the training process to discover that some pseudo-labels were wrong and self-correct — the model simply learns to reproduce its initial mistakes at higher confidence.

How online RL breaks the bound. TTRL's RL loop is fundamentally different. In each episode, the model generates new samples, produces new majority-vote labels (which may differ from previous episodes' labels), and updates toward those new labels. If the policy has improved — even slightly — the new majority votes may be more accurate than the old ones. The RL objective is not "match the initial model's majority vote" but "generate outputs that agree with the current model's majority vote." As the model improves, the target moves. This is bootstrapping in the strongest sense: the model's improvements create better training signals, which drive further improvements.

Figure 7 provides the key evidence. After TTRL training, the improved model's avg@64 (average accuracy across 64 samples) consistently and substantially exceeds the initial model's maj@64 (majority voting accuracy across 64 samples). On AIME 2024, the gap is roughly 10 percentage points. This means the final model, evaluated with a single sample, is competitive with what the initial model could achieve only by aggregating 64 samples — and when the final model itself uses majority voting, it far exceeds its own training signal's initial quality.

Why this matters beyond benchmark scores. The ability to exceed the self-training upper bound changes what is possible with unsupervised data. Previously, self-training was viewed as a way to distill ensemble knowledge into a single model, with the understanding that you're trading off some accuracy for efficiency. TTRL suggests that RL on self-generated signals can actually create capability that wasn't present in the initial ensemble — it's not just distillation, it's genuine improvement. This has implications for the scalability of AI systems: if models can bootstrap beyond their initial capabilities using only unlabeled data, the dependence on human annotation (or verifiable ground-truth) is reduced not just incrementally but qualitatively.

The paper explicitly frames this as "lifting itself up by its own bootstraps" (Section 4.1), a deliberate reference to the bootstrap metaphor in machine learning. It's a fundamental shift: from self-training as distillation (compressing ensemble knowledge into a single model) to self-training as evolution (using ensemble knowledge to guide exploration that discovers better policies).

Innovation 4: Test-Time RL as a New Paradigm — Training on Evaluation Data Without Leakage

Perhaps the most provocative conceptual move in this paper is the deliberate decision to train on test data. In standard machine learning practice, this is the cardinal sin — "training on the test set" is synonymous with cheating, leakage, and invalid evaluation. TTRL recontextualizes this practice by removing the element that makes it cheating: access to ground-truth labels.

The standard view. The train/test split exists to measure generalization. Training on test data — even without labels — has traditionally been viewed with suspicion because it allows models to adapt to the idiosyncrasies of the test distribution, potentially inflating performance in ways that don't transfer. Transductive learning and domain adaptation do allow unlabeled test data during training, but these are typically constrained to feature distribution alignment, not task-specific optimization.

The TTRL reframing. By removing ground-truth labels from the process entirely, TTRL sidesteps the information leakage concern. The model adapts to the test distribution using only the test inputs — information that would be available in any real deployment scenario. The parallel is to a student who, upon receiving an exam, spends time thinking about each question before answering, revising initial thoughts based on deeper reflection. The student isn't "cheating" — they're using test-time computation to improve their answers. TTRL simply makes this test-time adaptation parametric rather than ephemeral.

The paper directly compares TTRL to "RL (leakage)" — training on test data with ground-truth labels — and shows that TTRL's performance approaches this upper bound (Figure 8). The fact that TTRL gets close to the leakage baseline without ever seeing labels is the strongest evidence that the method is genuinely extracting signal from the test distribution rather than exploiting label information.

Implications for benchmarking. This has uncomfortable implications for how the field evaluates models. If TTRL becomes standard practice, the distinction between "training" and "evaluation" blurs. A model's performance on a benchmark would no longer reflect fixed capabilities but rather the model's ability to self-improve on that specific distribution. This is arguably more realistic — real-world deployments always involve adapting to the specific distribution of queries encountered — but it makes cross-model comparisons more complex. The paper doesn't fully grapple with this epistemic shift, but it's implicit in the very design of the method.

Connection to the "era of experience." The paper's invocation of Silver & Sutton (2025) is not just rhetorical positioning. TTRL is a concrete instantiation of the idea that AI systems should learn from streams of experience — in this case, the experience of attempting to solve problems, seeing what their own ensemble thinks, and adjusting accordingly. The Test-Time framing makes this incremental: each new benchmark is an experience stream that the model can learn from, without requiring the benchmark creators to provide labels. This aligns with a vision where AI systems continuously improve through interaction with novel problems, rather than being periodically retrained on curated labeled datasets.

5. Experimental Analysis

Evaluation Methodology

Dataset. Four benchmarks: AIME 2024 (Li et al., 2024), AMC (Li et al., 2024), MATH-500 (Hendrycks et al., 2021), and GPQA-Diamond (Rein et al., 2024). AIME 2024 and AMC are competition-level math problem sets; MATH-500 is a 500-question subset of the MATH dataset spanning five difficulty levels; GPQA-Diamond is a 198-question graduate-level science QA benchmark. For difficulty-level analysis (Table 3), MATH-500 is further split into its five annotated difficulty levels (L1 through L5). Dataset sizes vary substantially: AIME 2024 contains 30 questions, AMC contains a larger set (exact count not specified), MATH-500 contains 500, and GPQA contains 198. This size variation directly drives the episode count differences in training configuration (10, 30, or 80 episodes).
Base model(s). Evaluated across four model families, two model types (base vs. instruct), and three scales (1.5B, 7B, 32B parameters): Qwen family (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B, Qwen2.5-32B, Qwen3-8B); LLaMA family (LLaMA-3.1-8B-Instruct, LLaMA-3.2-3B-Instruct, LLaMA-3.2-3B-Oat-Zero); Mistral family (Mistral-Nemo-Instruct-2407, Mistral-8B-Instruct-2410); DeepSeek family (DeepSeek-Math-7B-Instruct, DeepSeek-R1-LLaMA-8B); and Skywork-OR1-Math-7B. The rationale for this breadth, per Section 3.1, is to validate generalizability: "Shao et al. (2025) demonstrated the importance of evaluating different models for RL-based methods to validate experimental conclusions." The inclusion of DeepSeek-R1-LLaMA-8B specifically tests whether TTRL benefits models that have already undergone expensive RL post-training.
Metrics. The primary metric is pass@1 using non-zero temperature sampling. For main experiments (Table 1, Table 2), the protocol follows DeepSeek-R1 (Guo et al., 2025): generate 16 responses per question (4 for 32k-context experiments) at temperature 0.6 and top-p 0.95, then compute:

$\text{pass@1} = \frac{1}{k} \sum_{i=1}^k p_i$

where $p_i$ indicates whether the $i$ -th response is correct. For additional analysis on Qwen2.5-Math (Table 4, Figure 4), greedy decoding is used for pass@1 to ensure fair comparison with prior work. For majority voting evaluations (Figure 7), maj@64 and avg@64 are reported—the fraction of 64-sample groups where the majority vote is correct, and the fraction of individual samples within those groups that are correct, respectively. Training-time monitoring uses ground-truth-free metrics: Entropy (output distribution uncertainty), Majority Voting Reward (average reward under the consensus-derived reward function), and Majority Ratio (frequency of the most common answer within a rollout). For analysis only, ground-truth-dependent metrics are also tracked: Label Accuracy (maj@n—whether the estimated label matches ground-truth), Reward Accuracy (fraction of majority-voting rewards matching ground-truth-derived rewards), and Ground-Truth Ratio (frequency of the correct answer within a rollout).
Baselines. Since "the use of TTT for reasoning has not been previously explored" (Section 3.1), the primary comparison is against the backbone model without any TTRL adaptation. For models where available, comparisons are made to their instruct fine-tuned versions (Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Instruct) and to leading "R1-Zero-Like" models trained on labeled data with RL: DeepSeek-R1-Distill-1.5B and 7B (Guo et al., 2025), SimpleRL-Zero-7B (Zeng et al., 2025), PRIME-Zero-7B (Cui et al., 2025), OpenReasoner-Zero-7B (Hu et al., 2025b), Oat-Zero-1.5B and 7B (Liu et al., 2025b), and LIMR-7B (Li et al., 2025). For the upper-bound analysis (Section 4.1, Figure 8), the paper compares TTRL against RL (leakage)—direct GRPO training on the test data with access to ground-truth labels, representing the strongest possible within-distribution improvement.
Generation budget / compute accounting. The fundamental unit is the number of sampled responses per question per episode. For label estimation via majority voting, the model generates 64 responses per question. For the actual RL training update, a random subset of 32 responses is downsampled from these 64 (the "vote-then-sample" strategy). The maximum generation length is set to 3,072 tokens for standard models and 32,768 tokens for LRMs. Training episodes vary by benchmark size and difficulty: 10 episodes for MATH-500, 30 episodes for AMC, and 80 episodes for AIME 2024. All experiments run on 8 × NVIDIA A100 80GB GPUs. The paper does not report total FLOPs or wall-clock time, framing cost in terms of generation budget and episode count.
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The paper applies TTRL to each benchmark individually and evaluates on the same benchmark—all questions in the benchmark serve as both training and evaluation data. This is inherent to the test-time training paradigm: the model adapts to and is evaluated on the same distribution. For generalization experiments (Figure 4), TTRL is trained on one benchmark and evaluated on others using greedy decoding. No confidence intervals, error bars, or multiple-run variance estimates are reported—all results appear to be single-run outcomes.

Main Quantitative Results

TTRL Across Models and Benchmarks (Table 1)

Table 1 presents the central results: TTRL applied independently to each benchmark using GRPO, evaluated against backbone models with no adaptation. On Qwen2.5-Math-7B, the flagship result: AIME 2024 improves from 12.9 to 40.2 (~211% relative increase), AMC from 35.6 to 68.1 (~91%), MATH-500 from 46.7 to 83.4 (~79%), while GPQA drops slightly from 29.1 to 27.7 (−4.8%). The average across all four benchmarks improves from 31.1 to 54.9 (~76.5%). The GPQA decline is the only negative result in the main table—the only case where TTRL hurts rather than helps.

On Qwen2.5-Math-1.5B, the smallest model: AIME 2024 improves from 7.7 to 15.8 (~105%), AMC from 28.6 to 48.9 (~71%), MATH-500 from 32.7 to 73.0 (~123%), and GPQA from 24.9 to 26.1 (~4.8%). The MATH-500 gain of +40.3 points is the largest absolute improvement in the table, demonstrating that even a 1.5B model can self-improve dramatically.

On vanilla base models (not math-specialized): Qwen2.5-7B improves on AIME 2024 from 7.9 to 23.3 (~195%), and Qwen2.5-32B improves from 7.9 to 24.0 (~204%). Notably, the absolute AIME scores for these general-purpose base models are nearly identical (both start at 7.9 before TTRL), yet after TTRL they reach 23.3 and 24.0 respectively—the scaling from 7B to 32B provides only a marginal benefit (+0.7 points) despite 4.6× more parameters.

On instruct models: LLaMA-3.1-8B-Instruct improves on AIME 2024 from 4.6 to 10.0 (~117%). Qwen3-8B (non-thinking mode) improves from 26.9 to 46.7 (~74%). The Qwen3-8B result is particularly notable because it starts from a much higher baseline (26.9 on AIME, already substantially above the base models) yet still gains +19.8 points—demonstrating that TTRL is not merely correcting low-hanging errors but can further improve already-competent models.

TTRL on Additional Model Families (Table 2)

Table 2 extends the evaluation to models not covered in Table 1. On the LLaMA family: LLaMA-3.2-3B-Oat-Zero (a model already trained with RL on labeled data) improves on AIME from 0.8 to 3.3, AMC from 15.1 to 25.3, and MATH-500 from 41.9 to 55.7. LLaMA-3.2-3B-Instruct improves from 6.0 to 13.3 on AIME, 19.4 to 31.3 on AMC, and 43.9 to 61.6 on MATH-500.

On the Mistral family: Mistral-Nemo-Instruct shows a mixed result—AIME drops from 0.8 to 0.0 (−0.8, the only absolute decline in Table 2), while AMC improves from 15.4 to 24.8 and MATH-500 from 40.8 to 51.0. Mistral-8B-Instruct improves on AIME from 1.3 to 3.3, AMC from 19.7 to 28.9, and MATH-500 from 52.4 to 57.8. The Nemo-Instruct AIME failure (0.8 → 0.0) is one of the few clear negative results and suggests that when the initial model's AIME performance is near floor, TTRL cannot create capability from nothing.

On the DeepSeek family: DeepSeek-Math-7B-Instruct shows modest gains—AIME from 1.9 to 2.5, AMC from 16.3 to 22.9, MATH-500 from 42.3 to 52.4. DeepSeek-R1-LLaMA-8B, the most capable model tested (starting at 51.7 on AIME), improves to 69.2 on AIME (+17.5 points), 88.9 on AMC (+7.3), and 90.9 on MATH-500 (+1.3). This is a critical result: even after DeepSeek's extensive RL post-training (which used 800K labeled examples per the Table 4 footnote), TTRL on unlabeled test data provides an additional +17.5-point boost on AIME—a 33.9% relative improvement on top of an already state-of-the-art model.

TTRL on Large Reasoning Models (Figure 3)

Figure 3 shows results on two additional LRMs evaluated with 32k context in thinking mode. Qwen3-8B improves from approximately 60 to 70 on AIME 2024 (~10-point gain). Skywork-OR1-Math-7B improves from approximately 55 to 65 on the same benchmark (~10-point gain). The paper characterizes these as "substantial performance gains" despite the models having undergone "expensive post-training." The consistency of the ~10-point improvement across both models is notable, though the specific numbers must be read from the bar chart since the paper does not tabulate them.

Generalization Beyond the Target Task (Figure 4)

Figure 4 evaluates whether TTRL improvements transfer to benchmarks not used during training. Using Qwen2.5-Math-7B with greedy decoding, the paper trains TTRL on each benchmark individually and evaluates pass@1 on all four benchmarks. The results, shown as a heatmap-style figure, indicate substantial cross-task improvement: "Despite the out-of-distribution nature of this setting, TTRL achieves substantial improvements across all benchmarks." The paper interprets this as evidence that "TTRL does not rely on overfitting, which would lead to trade-offs on other tasks, but instead acquires generalizable gains during self-improvement." Specific transfer numbers are not tabulated—only the figure is provided—so precise cross-task transfer magnitudes cannot be quoted.

Compatibility with Different RL Algorithms (Figure 5)

Figure 5 compares three RL algorithms—GRPO, PPO (Schulman et al., 2017), and PRIME (Cui et al., 2025)—used with TTRL on MATH-500 using Qwen2.5-Math-1.5B. The paper reports: "The performance trajectories of GRPO, PPO, and PRIME are closely aligned." This suggests that TTRL's effectiveness is not tied to a specific RL algorithm, but the paper does not report final accuracy numbers for PPO and PRIME individually, nor does it perform this comparison on a harder benchmark like AIME 2024, where algorithm differences might be more pronounced.

Training Dynamics: The Self-Reinforcing Loop (Figure 6)

Figure 6 tracks pass@1/avg@16 and maj@16 scores throughout TTRL training on AMC with Qwen2.5-Math-1.5B. The key observation: "both metrics demonstrate a consistent upward trend." The initial avg@16 starts around 20 and rises to approximately 50 by the end of training. The initial maj@16 starts around 30 and rises to approximately 55. The critical finding is that avg@16 exceeds the initial maj@16 by over 20 points—the model's individual-sample performance surpasses its initial ensemble performance. The paper frames this as evidence that "TTRL can generate higher-quality supervision signals as its capabilities improve" and that this allows it to exceed what offline self-training could achieve.

Surpassing the maj@n Upper Bound (Figure 7)

Figure 7 compares majority voting performance before and after TTRL on Qwen2.5-Math-7B across all four benchmarks, using 64 samples per question. The key claim: "TTRL avg@64 consistently outperforms Qwen2.5-Math-7B maj@64 across all benchmarks, with a considerable margin." This demonstrates that TTRL's improvement is not just in single-sample quality but also in ensemble quality—the model becomes better both individually and collectively. The improved model's maj@64 also substantially exceeds the initial model's maj@64, showing that TTRL's benefits compound under voting.

Approaching the RL (Leakage) Upper Bound (Figure 8)

Figure 8 compares TTRL against RL (leakage)—direct GRPO training on MATH-500 with ground-truth labels—using Qwen2.5-Math-7B. The paper reports: "the performance curve of TTRL closely approaches that of RL (leakage)." Both curves rise from the initial model's accuracy (~50% with greedy decoding) to approximately 84% by the end of training. The paper treats this as evidence that TTRL achieves "a level of self-improvement comparable to that of supervised learning (even in the information leakage scenario)." The convergence is striking because TTRL never sees ground-truth labels, yet its trajectory nearly matches the fully supervised baseline. This result is reported only for MATH-500 and only for Qwen2.5-Math-7B; the paper does not show equivalent comparisons on harder benchmarks like AIME 2024, where the gap might be larger.

Difficulty-Dependent Performance (Table 3)

Table 3 breaks down TTRL performance on MATH-500 by the dataset's five annotated difficulty levels (1 = easiest, 5 = hardest), using Qwen2.5-Math-1.5B with greedy decoding. Accuracy: TTRL provides gains at all levels, but with a clear diminishing-returns pattern. L1 improves from 25.9 to 71.2 (+45.4, ~175%), L2 from 33.0 to 76.2 (+43.2, ~131%), L3 from 36.3 to 76.3 (+40.0, ~110%), L4 from 32.5 to 58.7 (+26.2, ~80%), and L5 from 22.3 to 39.2 (+16.8, ~75%). The percentage improvement decreases monotonically as difficulty increases (175% → 131% → 110% → 80% → 75%), and absolute improvement follows a similar pattern (+45.4 → +43.2 → +40.0 → +26.2 → +16.8). Response length: TTRL consistently reduces output length, with larger reductions on easier problems. L1 length drops from 2,339 tokens to 624 (−73.3%), while L5 drops from 1,751 to 985 (−43.7%). The paper interprets this as evidence that "the available prior knowledge of the backbone is insufficient to support learning on more challenging questions"—the model can self-improve more effectively when it already has a reasonable grasp of the problem type.

Comparison with Labeled-Data RL Models (Table 4, Appendix A)

Table 4 places TTRL in context with models trained on large-scale labeled data, using greedy decoding for fair comparison. On Qwen2.5-Math-7B: TTRL reaches 43.3 on AIME 2024, compared to the backbone's 16.7 (~159% improvement). This exceeds several models trained on labeled data: DeepSeek-R1-Distill-7B (10.0 at 3k context, though 26.7 at 8k in some configurations), SimpleRL-Zero-7B (26.7), and PRIME-Zero-7B (16.7). TTRL's AIME score ties Oat-Zero-7B (43.3), which was trained on 8.9K labeled examples. On AMC, TTRL reaches 67.5, exceeding all listed labeled-data models except LIMR-7B (63.8). On MATH-500, TTRL reaches 84.2, exceeding all listed models. The paper explicitly notes: "Note that TTRL has a different setup from the previous models, which makes the comparison seem unfair"—TTRL adapts to the test distribution itself, while the labeled-data models were trained on separate datasets and evaluated zero-shot. On Qwen2.5-Math-1.5B, TTRL reaches 20.0 on AIME (matching the backbone and Oat-Zero-1.5B), 53.0 on AMC, and 80.0 on MATH-500—substantially exceeding the instruct version (74.2 on MATH-500 with 3.1M labeled examples) and DeepSeek-R1-Distill-1.5B (52.2 on MATH-500 with 800K labeled examples).

Ablation Studies and Robustness Checks

RL algorithm choice (GRPO vs. PPO vs. PRIME): Figure 5 shows that all three algorithms produce closely aligned training trajectories on MATH-500 with Qwen2.5-Math-1.5B. This suggests TTRL's reward estimation mechanism, not the specific RL algorithm, is the active ingredient. However, the comparison is limited to one (relatively easy) benchmark and one (small) model; algorithm differences might emerge on harder tasks or larger scales.
Temperature and batch size sensitivity (Figure 11): The paper identifies two hyperparameters that critically affect training success. Setting temperature to 1.0 (vs. 0.6) "increases the model's output entropy" and "promotes more extensive exploration," but Figure 11 shows this can also lead to training failure with persistently high entropy that does not diminish—the model explores but never converges. Similarly, suboptimal training batch size (details not specified) causes failed runs with "persistently high entropy that does not diminish throughout training, consistent with findings of prior work (He et al., 2025)." These negative results demonstrate that TTRL is sensitive to hyperparameters in ways that standard supervised fine-tuning is not, and that the reward noise from majority voting can amplify instability if exploration is not properly calibrated.
Episode count by benchmark difficulty: The paper sets episodes to 10, 30, and 80 for MATH-500, AMC, and AIME 2024, respectively, based on dataset size and difficulty. No ablation on episode count is reported. The choice reflects a practical observation that smaller, harder datasets require more passes, but the absence of an explicit ablation leaves open the question of whether performance would plateau earlier (suggesting overfitting to the test distribution) or continue improving with more episodes.
Vote-then-sample strategy (64 → 32 downsampling): The paper's computational optimization—sampling 64 outputs for voting, using 32 for training—is described as "effectively reduces computational costs while still achieving strong performance." No ablation compares 64/64 (all used for training), 32/32 (smaller label estimation pool), or other splits. The claim is asserted based on "evidence" but the evidence is not presented. This is a notable missing ablation since the 64/32 ratio directly affects both label quality and training efficiency.
Model scale (1.5B → 7B → 32B): The natural scaling result is observed implicitly across Table 1: Qwen2.5-7B achieves 23.3 on AIME after TTRL, while Qwen2.5-32B achieves 24.0—only +0.7 improvement for 4.6× more parameters. On the easier MATH-500, 7B reaches 80.5 and 32B reaches 83.2 (+2.7). The scaling benefit appears modest, but this is not presented as a formal ablation—the models differ in both size and base capability.
Difficulty-level breakdown (Table 3): The MATH-500 difficulty split serves as a de facto ablation on the effect of prior knowledge. As difficulty increases (L1 → L5), both accuracy improvements and response length reductions systematically decrease. At L5, TTRL still provides a non-trivial +16.8-point gain (~75% relative), but the absolute final accuracy (39.2) remains low—the model hasn't solved the hard problems, just improved from "almost never right" to "sometimes right."
Cross-task generalization (Figure 4): The paper trains on one benchmark and evaluates on others, finding substantial positive transfer. This is a robustness check against the concern that TTRL merely overfits to the test distribution in a way that doesn't transfer. The positive transfer suggests that the reasoning skills acquired are somewhat general. However, the absence of tabulated numbers means the magnitude of transfer cannot be precisely assessed—the visual in Figure 4 is the only evidence.
Majority voting reward function vs. alternatives: No ablation compares majority voting against alternative pseudo-labeling strategies (e.g., using the model's highest-confidence output as the label, using a separate reward model, using the single-sample output as its own label). The paper's choice of majority voting is motivated conceptually (Section 2.2) but not tested against alternatives.
Training on the test data vs. separate data: The entire paradigm constitutes an implicit ablation: TTRL trains on the evaluation data itself. The comparison with RL (leakage) in Figure 8 is the closest the paper comes to testing this choice. The fact that TTRL approaches RL (leakage) performance suggests that majority voting recovers most of the benefit of having ground-truth labels on that specific distribution. However, this comparison exists only for MATH-500, not for the harder benchmarks where label estimation is more difficult (37% label accuracy on AIME 2024 vs. presumably higher on MATH-500).

Critical Assessment

Claim 1: "Majority voting provides effective reward estimation for TTRL" (Takeaway 1, Section 1).

This claim is strongly supported for the benchmarks and models tested, but the definition of "effective" deserves scrutiny. The evidence: (a) TTRL produces consistent and substantial gains across 12+ models and 4 benchmarks (Tables 1, 2), with the lone notable failure on GPQA with Qwen2.5-Math-7B (−1.4 points); (b) reward accuracy on AIME 2024 reaches 92% despite label accuracy of only 37% (Figure 9), demonstrating the Lucky Hit mechanism produces a surprisingly clean training signal; (c) training dynamics (Figure 6) show stable improvement without collapse in successful runs.

However, "effective" is not "always effective." Figure 11 shows training failures due to hyperparameter sensitivity (temperature 1.0 causing non-decreasing entropy). Table 3 shows diminishing returns on harder MATH-500 problems. The Mistral-Nemo-Instruct result on AIME (0.8 → 0.0) is a clear failure case. And the paper acknowledges in Section 4.3 that TTRL "fails if the model's prior knowledge is insufficient to handle the complexity of the data." What's missing is a systematic characterization of when prior knowledge is sufficient—the difficulty-level analysis (Table 3) is the closest proxy, but it's post-hoc and specific to MATH-500.

The reliance on two specific types of benchmarks (math and science QA) is also a limitation. All four benchmarks have verifiable, discrete answers where rule-based binary comparison rewards are natural. The Lucky Hit mechanism depends on scattered incorrect outputs, which is more likely in math (many possible wrong numerical answers) than in classification or multiple-choice tasks (where wrong answers cluster on a few alternatives). No experiment tests TTRL on open-ended generation, code synthesis, or other domains without clean answer extraction.

Claim 2: "TTRL can exceed its training signal and upper limit maj@n, and closely mirrors the performance of direct training on the test data with ground-truth" (Takeaway 2, Section 1).

The "exceeding maj@n" claim is the paper's most striking result and is well-supported by Figures 6 and 7. Figure 6 shows avg@16 surpassing initial maj@16 by ~20 points on AMC. Figure 7 shows post-TTRL avg@64 exceeding initial maj@64 across all four benchmarks. This is a genuine phenomenon—the online RL loop creates a self-reinforcing cycle that offline self-training cannot—and the evidence is consistent.

The claim about approaching RL (leakage) performance is supported only by Figure 8, which shows a single experiment on MATH-500 with Qwen2.5-Math-7B. This is the paper's most important upper-bound comparison, yet it is limited to one (relatively easy) benchmark and one model. On AIME 2024, where label accuracy is only 37%, the gap between TTRL and RL (leakage) could be substantially larger—the paper does not report this comparison. The claim that TTRL "closely mirrors" the upper bound is therefore demonstrated only conditionally: on a benchmark where the model already has substantial prior knowledge and majority voting is reasonably accurate. On harder benchmarks, the gap may be larger, and the paper does not provide evidence either way.

The comparison with labeled-data models in Table 4 adds context but introduces an apples-to-oranges dimension: TTRL adapts to the test distribution itself, while the comparison models were trained on separate datasets and evaluated zero-shot. The paper is transparent about this ("TTRL has a different setup ... which makes the comparison seem unfair"), but the claim that TTRL "surpasses" or "ties" these models should be interpreted in light of this distributional mismatch. A fairer comparison would give the labeled-data models the same opportunity for test-time adaptation—but that's precisely what TTRL provides and they lack.

Claim 3: "It is possible to achieve efficient and stable RL in an unsupervised manner" (Takeaway 3, Section 1).

This claim requires unpacking "efficient" and "stable." Stability: The successful runs (Figures 6, 12) show smoothly improving entropy and accuracy curves, which is evidence of stability. However, Figure 11 shows that stability is not guaranteed—inappropriate hyperparameters cause failures. The paper's identification of temperature and batch size as critical parameters is useful, but implies that "stable" is conditional on careful tuning. The absence of multi-run statistics (standard deviations, failure rates) means we cannot assess how often TTRL succeeds vs. fails under a given configuration.

Efficiency: The paper does not explicitly define or measure efficiency. The main cost is 64 samples × episode_count × dataset_size generations per benchmark. For AIME 2024 (30 questions, 80 episodes, 64 samples each), this is 153,600 generations. For MATH-500 (500 questions, 10 episodes, 64 samples), this is 320,000 generations. These are substantial but not prohibitive—the paper ran on 8×A100 GPUs. The "efficiency" claim might refer to the fact that TTRL achieves strong performance using only test data (zero annotation cost), but this is an annotation-efficiency claim, not a compute-efficiency claim. The paper provides no comparison of TTRL's compute cost vs. standard RL on labeled data to achieve the same accuracy—a FLOPs-matched comparison would be needed to substantiate a compute-efficiency claim.

Genuine weaknesses and missing experiments:

Single-run results with no error bars. All reported numbers appear to be from single training runs. Given the demonstrated sensitivity to hyperparameters (Figure 11), variance across runs could be substantial. Without multiple seeds or confidence intervals, we cannot assess the reliability of specific accuracy numbers or the statistical significance of comparisons.
The GPQA failure is underexplored. TTRL reduces GPQA performance for Qwen2.5-Math-7B (29.1 → 27.7) and provides minimal gains for other models (e.g., Qwen2.5-Math-1.5B: 24.9 → 26.1). This is the only benchmark where TTRL shows near-zero or negative benefit, yet the paper provides no analysis of why. GPQA is a multiple-choice science QA benchmark—the Lucky Hit mechanism may break down when the output space is small (4 choices), because wrong answers cluster on the few available options rather than scattering across many wrong numerical answers, potentially reducing reward accuracy.
No analysis of reward accuracy degradation over training. The Lucky Hit mechanism relies on scattered outputs. As the model improves and converges on specific (possibly wrong) answers, reward accuracy could degrade before label accuracy improves. The paper does not track reward accuracy over training episodes or analyze this dynamic. Figure 9 shows label accuracy and reward accuracy on AIME 2024 over steps, but these are initial values, not trajectories. Understanding how reward accuracy evolves would clarify the mechanisms by which TTRL maintains effective training signals.
The answer extractor's reliability is untested. The entire reward mechanism depends on extract_answer correctly pulling answers from chain-of-thought traces. If the extractor fails (e.g., extracts a wrong substring, misses the answer entirely, or is confused by formatting), rewards become corrupted. The paper treats extraction as solved, but extraction failures are known to be a significant source of error in math reasoning benchmarks. No extraction accuracy analysis is provided.
Missing baselines. No comparison against alternative unsupervised or semi-supervised methods: (a) offline self-training with SFT on majority-voted pseudo-labels (to isolate the online RL benefit); (b) simply using majority voting with the initial model at the same generation budget (to isolate the parametric improvement from the inference-time compute benefit); (c) using a single sample as its own pseudo-label (to test whether majority voting specifically matters vs. any self-referential signal). These comparisons would sharpen the claim that majority voting + online RL is the key combination.
Limited exploration of the "prior knowledge" boundary. The paper identifies prior knowledge as critical (Table 3, Section 4.3) but doesn't systematically probe where the boundary lies. How low can initial accuracy be before TTRL fails? Is there a threshold (e.g., pass@1 < 1%, or ground-truth ratio < 5%) below which the Lucky Hit mechanism can no longer provide sufficient positive rewards? Controlled experiments gradually reducing prior knowledge (e.g., by using progressively weaker models on the same benchmark) would characterize this boundary.
Confounding of test-time adaptation with distribution matching. Because TTRL trains and evaluates on the same data, its gains could reflect domain adaptation (adjusting to the style, format, and vocabulary of specific test questions) rather than genuine reasoning improvement. The cross-task generalization results (Figure 4) partially address this by showing transfer to other benchmarks, but the absence of tabulated numbers limits confidence. A stronger test: train TTRL on AIME 2024, evaluate on AIME 2025 (same format, different problems)—this would isolate distribution matching from specific-question overfitting.

Conditional nature of the claims:

The paper's central claims hold under conditions that the experiments partially characterize but don't fully specify: (1) the base model must have non-trivial prior knowledge on the task (some correct answers present in the generation distribution); (2) the task must support rule-based binary reward comparison (discrete, extractable answers); (3) hyperparameters must be carefully tuned, particularly temperature and batch size; (4) the benchmarks are competition-level math and science QA—generalization to other domains is plausible but unproven. The paper's transparency about these limitations (particularly in Section 4.3 and 7) is commendable, and the claims are appropriately scoped—they don't assert universal applicability, and the failure modes are documented alongside the successes.

6. Limitations and Trade-offs

6.1 The Prior Knowledge Boundary — TTRL Fails When the Base Model Cannot Produce Correct Answers

The assumption or constraint. TTRL's entire mechanism depends on the base model having at least some correct answers within its generation distribution on the target task. Majority voting can only surface a correct label if the correct answer appears among the sampled outputs; the Lucky Hit mechanism can only assign correct negative rewards if the model's wrong answers are scattered. The paper states this explicitly in Section 4.3:

"for the same backbone, TTRL fails if the model's prior knowledge is insufficient to handle the complexity of the data."

Table 3 demonstrates this empirically on MATH-500: as difficulty increases from L1 to L5, TTRL's absolute improvement declines from +45.4 points to +16.8 points, and the percentage improvement drops from 175% to 75%. The final accuracy on L5 problems is only 39.2% — the model improves from "almost never correct" to "sometimes correct," but remains far from reliable performance.

The consequence. TTRL provides no path to solving problems that are fundamentally beyond the base model's reach. If the model never samples the correct answer — or samples it so rarely that it never wins a majority vote — the estimated labels remain incorrect throughout training, and the self-reinforcing loop described in Section 4.2 cannot start. This is not a gradual degradation but a hard failure boundary. The Mistral-Nemo-Instruct result on AIME 2024 (0.8 → 0.0, Table 2) illustrates the extreme case: the model's initial accuracy is so close to zero that there is essentially no correct signal to amplify, and TTRL can even make things worse (potentially by reinforcing systematically wrong answers that happen to win majority votes).

This limitation is fundamental rather than incidental. It means TTRL is an amplification method, not a capability-creation method. It can substantially improve a model's performance on tasks where the model already has some latent competence, but it cannot teach the model to solve entirely novel problem classes or reasoning patterns. For genuinely hard benchmarks — the ARC-AGI-2 problems that motivated the paper's introduction, where o3 solves only 4% — TTRL would be ineffective unless the base model already had non-trivial competence.

What evidence exists. Table 3 is the primary evidence, showing the monotonic decline in both absolute and relative improvement as MATH-500 difficulty increases. Table 2 provides additional evidence: models with very low initial AIME scores (Mistral-Nemo-Instruct at 0.8, DeepSeek-Math-7B-Instruct at 1.9) show minimal or negative gains on that benchmark. The paper does not systematically probe where the failure threshold lies — it does not, for instance, vary the base model's capability on a fixed task and measure the relationship between initial pass@1 and TTRL improvement.

Mitigation status. The paper acknowledges this limitation explicitly (Section 4.3, Section 7) but does not propose a solution. It notes that "TTRL does not incorporate mechanisms such as data filtering to support curriculum learning," suggesting this could be addressed in future work. A natural mitigation — not explored — would be to estimate question difficulty from the initial model's output statistics (majority ratio, entropy, or ground-truth ratio if treating a small labeled subset as a diagnostic) and either exclude questions below a capability threshold or route them to a stronger model. The limitation is presented as a fundamental characteristic of the approach rather than a solvable implementation detail.

6.2 Difficulty Estimation Cost Is Unaccounted For — TTRL's Headline Efficiency Excludes the Compute Needed to Make It Practical

The assumption or constraint. TTRL trains directly on the test benchmark — the same questions used for evaluation. For this to be possible in a real deployment, the system would need to identify which unlabeled questions are worth training on and how much training to allocate per question. The paper's experiments bypass this entirely: they apply a fixed training recipe (64 samples × N episodes) uniformly to all questions in a benchmark, without any per-question adaptation or difficulty estimation. In a deployment setting where unlabeled questions arrive continuously and the model must decide how to allocate its test-time training budget, this uniform allocation would be both wasteful (spending compute on questions that are trivially easy or impossibly hard) and inefficient (failing to concentrate resources where they provide the most benefit).

The paper acknowledges a related but narrower cost concern in its discussion of reward estimation:

"the vote-then-sample strategy effectively reduces computational costs"

But this refers only to the internal cost of label estimation vs. training within the fixed recipe — it does not address the meta-problem of deciding which questions to train on, for how long, or with what hyperparameters.

The consequence. The headline numbers (e.g., "211% improvement on AIME 2024") are achieved by applying TTRL indiscriminately to all 30 AIME questions for 80 episodes each. In a deployment scenario, an equivalent approach would require spending 153,600 generations (64 samples × 80 episodes × 30 questions) on a set of 30 problems — a massive compute investment that may be unjustified for problems the model already answers correctly or problems it will never answer correctly. The paper provides no mechanism for estimating in advance whether a given question will benefit from TTRL or how much training budget to allocate. Without such a mechanism, the efficiency gains over labeled-data training are difficult to assess: the annotation cost is eliminated, but the compute cost may be substantially higher than supervised RL for the same accuracy improvement, and this trade-off is never quantified.

What evidence exists. The paper provides all the necessary ingredients for this limitation to be apparent but does not analyze them as a limitation. The episode counts (10, 30, 80) are set heuristically based on dataset size and difficulty, without comparing to alternative budgets. The difficulty-level analysis (Table 3) shows that TTRL benefit varies dramatically across questions, implying that uniform allocation is suboptimal — but the paper does not calculate how much compute is wasted on questions that show minimal improvement. There is no FLOPs-matched comparison between TTRL and standard supervised RL (the RL-leakage comparison in Figure 8 uses the same compute budget, not a FLOPs-matched budget).

Mitigation status. Not addressed. The "Future Works" section mentions "Test-Time Adaptation (Liang et al., 2025)" and "learning from continuously arriving data," which gestures toward the online allocation problem, but no concrete approach is proposed. A practical mitigation would be adaptive budget allocation: after a small initial sample (e.g., 8 generations), estimate the model's competence on each question (via majority ratio or entropy), then allocate additional TTRL episodes only to questions in the "improvable" difficulty band. This would connect TTRL to the compute-optimal test-time scaling framework (Snell et al., 2024) that the paper cites, but the connection is not made.

6.3 Single-Domain Evaluation — All Results Are on Math and Science QA with Binary Correctness Criteria

The assumption or constraint. Every benchmark in this paper — AIME 2024, AMC, MATH-500, and GPQA — involves problems with discrete, extractable answers where correctness can be determined by exact string match against a ground-truth label. This enables two things that TTRL critically depends on: (1) majority voting across extracted answers produces a meaningful consensus (answers must cluster around discrete values), and (2) the Lucky Hit mechanism works because wrong answers scatter across many possible values rather than clustering on a few alternatives.

The paper does not test TTRL on any task that lacks these properties: open-ended generation (summarization, creative writing), code synthesis (where functional correctness matters but surface-form matching doesn't), multi-step reasoning without extractable final answers (debate, ethical reasoning), or tasks with continuous or fuzzy correctness criteria. The paper acknowledges this scope limitation only implicitly, by stating its focus on "reasoning tasks" and "mathematical problem-solving" in Sections 5.1 and 5.2, without discussing what would break in other domains.

The consequence. The Lucky Hit mechanism — which the paper identifies as the key to TTRL's success despite low label accuracy — is domain-specific in ways the paper does not analyze. In multiple-choice tasks, wrong answers are not "highly scattered" (as the paper says of AIME in Section 4.2) — they cluster on the 3–4 incorrect options. This means an incorrect majority-vote label will randomly coincide with individual wrong answers more often, degrading reward accuracy. The GPQA results (Table 1) may reflect this: TTRL provides minimal or negative benefit on GPQA across multiple models (Qwen2.5-Math-7B: −1.4; Qwen2.5-Math-1.5B: +1.2; Qwen2.5-7B: +1.8), making it the only benchmark where TTRL consistently underperforms. The paper does not analyze this pattern or connect it to the multiple-choice format.

In code generation, a model might produce functionally correct code that differs superficially from a reference or from other correct solutions — majority voting on raw outputs would fail to recognize this equivalence. In open-ended generation, there is no extractable "answer" to vote on at all. TTRL as presented is inapplicable to these domains without a fundamentally different reward mechanism, but the paper's framing (particularly the "era of experience" motivation in Section 1) implies broader applicability that is not tested.

What evidence exists. The GPQA results in Table 1 are the closest thing to negative evidence for domain generalization — but the paper does not analyze them as such. Table 2 does not include GPQA at all. The cross-task generalization experiment (Figure 4) shows transfer between benchmarks of the same type (math), not to qualitatively different task formats. The paper's positive results are entirely within the math-and-science-QA family, and the one science QA benchmark (GPQA) shows the weakest results.

Mitigation status. Not addressed. The paper does not discuss how TTRL would extend to non-math domains or to tasks without extractable answers. The "Future Works" section mentions "Agentic Tasks and Scientific Discovery" as an application direction, but this proposes applying TTRL to new domains without addressing how the reward mechanism would need to change for tasks that lack discrete, extractable, majority-votable answers. A natural extension — not mentioned — would be to replace majority voting with a learned verifier or an execution-based reward (e.g., unit tests for code) that could provide self-consistency signals without requiring surface-form matching.

6.4 Hyperparameter Fragility — Training Stability Depends on Specific, Untuned Choices with No Robustness Guarantees

The assumption or constraint. TTRL inherits the hyperparameter sensitivity of RL training and amplifies it with the additional noise from majority-voting reward estimation. The paper identifies two hyperparameters as critical: temperature and training batch size. Section 4.3 and Figure 11 document several failed training runs:

"Both of these failed attempts exhibit persistently high entropy that does not diminish throughout training, consistent with findings of prior work (He et al., 2025)."

The failure mode is specific and diagnosable (non-decreasing entropy), but the paper does not provide a recipe for avoiding it beyond the specific hyperparameter values that worked in the reported experiments. The temperature choice (0.6 vs. 1.0) differs between model types for reasons that are explained post-hoc (Section 3.1: higher temperature for math models "promotes more extensive exploration") but not validated through systematic sweeps.

The consequence. A practitioner attempting to apply TTRL to a new model, benchmark, or domain has no principled way to select hyperparameters. The paper's reported results come from configurations that were presumably discovered through trial and error (the unsuccessful attempts in Figure 11 imply experimentation), but the space of failed configurations is not characterized. Temperature, learning rate, batch size, number of episodes, and the 64/32 sampling ratio all interact with model size, task difficulty, and base model capability in ways that are not understood. A practitioner could easily land in a failure regime (persistently high entropy, no improvement) without knowing whether the problem is hyperparameters or insufficient prior knowledge.

This fragility is not merely a practical annoyance — it undermines the "unsupervised" claim. If deploying TTRL requires extensive hyperparameter tuning on a held-out set (which, by definition, would need ground-truth labels to evaluate), the method is not truly unsupervised in practice. The paper's own metric for detecting failure — entropy — is ground-truth-free, which is good, but the paper doesn't demonstrate that entropy monitoring alone is sufficient to guide hyperparameter selection to success.

What evidence exists. Figure 11 documents two specific failure cases (temperature 1.0 and suboptimal batch size) on AIME 2024 with Qwen2.5-Math-7B, including the entropy curves that characterize the failures. The paper also reports that the 1.0 temperature was used successfully for Qwen2.5-Math models in the main experiments (Section 3.1), suggesting the failure depends on interactions with other hyperparameters or benchmarks. The paper does not report the failure rate across different configurations, the number of configurations attempted before finding the reported ones, or any systematic sensitivity analysis.

Mitigation status. Partial. The paper identifies entropy as a monitoring signal and acknowledges that hyperparameter selection "remains to be further investigated" (Section 7). It compiles a community resource on RL hyperparameter recipes (Awesome-RL-Reasoning-Recipes), but this resource is about standard RL training, not TTRL-specific considerations like the interaction between sampling temperature and majority-voting reward quality. The paper does not propose an automated hyperparameter selection method, a robust default configuration, or guidelines for adapting the reported settings to new contexts.

6.5 The Answer Extraction Bottleneck Is Untested — Reward Accuracy Depends on a Component With Unknown Failure Rate

The assumption or constraint. Every reward computation in TTRL depends on extract_answer() (Listing 1) correctly isolating the final answer from the model's chain-of-thought output. If the extractor fails — extracting the wrong string, missing the answer entirely, or being confused by unusual formatting — the reward for that output is corrupted regardless of whether the majority-voted label is correct. The paper mentions the answer extractor in passing (Section 2.2: "An answer extractor then processes these outputs to obtain the corresponding predicted answers") but provides no details on its implementation, no analysis of its accuracy, and no discussion of how extraction failures interact with the Lucky Hit mechanism.

The extraction problem is known to be non-trivial in math reasoning benchmarks. Models can produce answers in varying formats (boxed expressions, plain numbers, multiple LaTeX encodings), include multiple candidate answers in a single trace, or fail to produce a parseable answer at all. Standard extraction approaches use regex patterns or format-specific parsers that are fragile to distribution shift — and TTRL is explicitly designed for distribution-shifted test data where format conventions may differ from those the extractor was designed for.

The consequence. Extraction failures introduce a form of reward noise that the paper does not account for and that may not be well-behaved in the Lucky Hit framework. A failed extraction can cause an output that would have matched the correct majority label to instead be scored as a mismatch (false negative reward), or can cause an output that should not match to coincidentally match (false positive). Unlike the Lucky Hit noise from mis-estimated labels — which the paper shows is benign because it systematically biases toward correct negative rewards — extraction noise is format-dependent and could be systematically biased in unknown ways. If the extractor consistently fails on certain answer formats (e.g., fractional answers, algebraic expressions, answers embedded in longer explanations), TTRL could inadvertently train the model away from those formats even when the underlying reasoning is correct.

What evidence exists. None directly. The paper's high reward accuracy on AIME 2024 (92%, Figure 9) suggests the extractor is not catastrophically failing on that benchmark, but this conflates extraction accuracy with label accuracy — a perfect extractor with 37% label accuracy and a completely broken extractor that returns random strings could both produce certain reward accuracy patterns, and the paper does not disentangle them. The paper does not report extraction failure rates, does not analyze whether extraction failures correlate with question difficulty or answer format, and does not test alternative extraction methods.

Mitigation status. Not addressed at all. The paper treats answer extraction as a solved problem, which it is not — particularly for the out-of-distribution test settings that TTRL targets. A minimal mitigation would be to report extraction coverage (what fraction of outputs produce parseable answers) and to analyze whether extraction failures systematically affect certain answer types. A more robust approach, not explored, would be to use the model's own self-consistency to detect extraction failures (e.g., if an output's final line doesn't match any common answer format, flag it as potentially unextractable) or to use LLM-based extraction that can handle formatting variability more gracefully than regex-based parsers.

6.6 No Statistical Reliability Assessment — All Results Are Single Runs Without Confidence Intervals or Replication

The assumption or constraint. The paper reports all results as point estimates from what appear to be single training runs. There are no error bars on any figure, no standard deviations in any table, no mention of multiple random seeds, and no statistical significance tests for any comparison. The paper's main numeric claims — "211% improvement on AIME 2024," "avg@64 exceeds initial maj@64 across all benchmarks" — are based on one trajectory per model per benchmark.

This matters because RL training is inherently stochastic: the random sampling of outputs for majority voting, the stochasticity in policy gradient updates, and the sensitivity to initial conditions can produce substantial run-to-run variance. The paper itself documents that some runs fail entirely (Figure 11), confirming that outcomes are not deterministic and that the aggregate numbers reported for "successful" runs represent a selected subset of attempts.

The consequence. We do not know the variance of TTRL outcomes. The reported 40.2 on AIME 2024 for Qwen2.5-Math-7B could be the mean of a tight distribution (e.g., 39–41 across seeds) or a lucky draw from a wide distribution (e.g., 30–42). The comparison between TTRL and baseline models (Table 1) could be statistically significant or could fall within overlapping confidence intervals — the paper provides no way to tell. The claim that TTRL's performance "closely approaches" RL-leakage (Figure 8) is based on two overlapping curves from one run each; with typical RL variance, two runs with different seeds might cross or diverge substantially.

This limitation is particularly consequential for the paper's headline claims about surpassing the maj@n upper bound (Section 4.1). If the initial model's maj@64 is, say, 45% with a standard deviation of 5% across different random majority-voting trials, and the post-TTRL avg@64 is 55% with a standard deviation of 8%, the "surpassing" claim might not hold at conventional significance levels. The paper provides only point estimates, so the robustness of this central finding cannot be assessed.

What evidence exists. The paper's acknowledgment of this limitation is implicit in Figure 11, which shows two different training runs with different outcomes — one successful, one failed — demonstrating that TTRL outcomes vary with hyperparameters. But this only scratches the surface of the variance question. The paper does not address within-configuration variance (same hyperparameters, different random seeds), which is the standard for assessing the reliability of RL results. No experiment reports multiple runs of the same configuration.

Mitigation status. Not addressed. The paper does not mention this as a limitation, does not commit to reporting multi-seed results in future work, and does not provide the raw training curves or data that would allow readers to assess variance indirectly. For a method that the paper positions as a "preliminary step toward RL with self-labeled rewards" (Section 6), the absence of basic statistical characterization is a significant gap. A minimal standard for future work would be to report mean ± standard deviation across 3–5 random seeds for the main results, and to include confidence intervals on the "surpassing maj@n" claim.

7. Implications and Future Directions

How This Work Changes the Landscape

TTRL does not introduce a new RL algorithm or a new inference-time strategy. What it does is more fundamental: it demonstrates that the boundary between training and inference can be erased when ground-truth labels are absent, and that this erasure enables a form of self-improvement that exceeds what offline self-training can achieve. This is not an incremental refinement of existing RL-for-reasoning pipelines — it is a conceptual reframing of what test-time compute is for.

From inference budget to training budget. Prior work on test-time scaling (Snell et al., 2024) treated inference compute as a lever to improve output quality from a frozen model: more samples, better voting, higher accuracy, but the model itself never changes. Prior work on RL for reasoning (DeepSeek-R1, GRPO-based training) treated training as a separate phase that happens on labeled data before the model ever sees a test question. TTRL collapses this distinction: the same majority voting that practitioners use to boost evaluation accuracy becomes the reward function that drives parameter updates on the test data itself. The inference budget is converted into a training budget with lasting effects. This reframing opens a third option between "train on labeled data, then evaluate" and "use inference compute on a frozen model" — you can train on unlabeled data during evaluation, and the improvements compound.

A new upper bound for self-supervised reasoning. The paper's most disruptive empirical finding is that TTRL surpasses the maj@n ceiling that previously bounded all self-training methods. Traditional self-training (Huang et al., 2022) can at best distill ensemble knowledge into a single model — if the majority vote is correct 40% of the time, SFT on those pseudo-labels can at best approach 40%. TTRL's avg@64 exceeds the initial model's maj@64 across all four benchmarks (Figure 7), and on AMC, avg@16 surpasses initial maj@16 by over 20 points (Figure 6). This demonstrates that self-supervised learning is not inherently capped by initial label quality — the cap was an artifact of offline SFT, not a fundamental limit. The online RL loop creates a virtuous cycle where improved policies generate improved labels, which generate improved policies. This is genuine bootstrapping, not just distillation.

Reconciling contradictory intuitions about label noise. The field has held two apparently contradictory beliefs: (1) RL for reasoning requires clean, verifiable rewards to work well (the DeepSeek-R1 paradigm), and (2) RL is robust to some level of reward noise (Razin et al., 2025). TTRL resolves this tension by showing that the structure of the noise matters more than its magnitude. Majority voting on AIME 2024 produces labels that are only 37% accurate, yet reward accuracy reaches 92% (Figure 9). The key is the Lucky Hit mechanism: when the model's wrong answers are scattered across many different values, an incorrect majority-vote label still generates correct negative rewards for most individual outputs. This transforms the research question from "how do we get high-quality pseudo-labels?" (the traditional self-training framing) to "how do we structure the output space so that comparison-based verifiers produce accurate rewards even when labels are wrong?" This is a more productive framing because it shifts attention from label accuracy — which is fundamentally limited for hard problems — to reward accuracy, which can remain high even when labels are poor.

Which research directions become more attractive. Verifier design becomes central. If reward accuracy can be dramatically higher than label accuracy through the Lucky Hit mechanism, then the bottleneck is not the pseudo-labeling method but the verifier's ability to exploit output diversity. Research on output space structure — how to encourage scattered incorrect answers rather than clustered ones, how to detect when the model is converging on a wrong answer and inject diversity — becomes immediately relevant in a way it wasn't before. Empirical work on the training dynamics of reward accuracy (how Lucky Hit degrades or sustains as the policy improves) is now clearly called for. And because TTRL approaches the RL-leakage upper bound on MATH-500 (Figure 8) but leaves the gap on harder benchmarks unexplored, characterizing where and why the gap opens becomes a precise, measurable research target.

Which directions become less attractive. The finding that TTRL approaches RL-leakage performance (Figure 8) reduces the urgency of developing ever-more-sophisticated pseudo-labeling strategies for SFT-based self-training — those methods are now shown to be bounded in a way that online RL is not. It also suggests that the community's heavy investment in labeled datasets for math reasoning (800K+ examples for DeepSeek-R1 distillation) may be partially circumventable for problems within a base model's capability range. This doesn't make labeled data obsolete — TTRL fails when prior knowledge is absent (Table 3, Section 4.3) — but it shifts the frontier: labeled data is needed to establish prior knowledge, not necessarily to refine it on specific distributions.

A diagnostic for model readiness. Beyond its direct training benefits, TTRL provides a diagnostic: the gap between a model's initial performance and its post-TTRL performance on a benchmark reveals how much latent competence the model possesses. A model that improves 211% on AIME through TTRL (Qwen2.5-Math-7B) had substantial untapped reasoning ability; a model that goes from 0.8 → 0.0 (Mistral-Nemo-Instruct, Table 2) has essentially none on that task. This diagnostic function could become standard practice for assessing whether a model's failures on a benchmark are due to insufficient capability or insufficient inference-time reasoning — a distinction that is practically important but currently difficult to make.

Follow-Up Research This Work Enables

Mapping the prior knowledge boundary with controlled capability degradation. The paper identifies prior knowledge as the critical enabler of TTRL (Section 4.3, Table 3) but only demonstrates this correlation post-hoc on MATH-500 difficulty levels. A controlled experiment would take a fixed benchmark (e.g., AIME 2024) and systematically degrade the base model's prior knowledge — by using progressively earlier checkpoints from pretraining, by fine-tuning on unrelated data to induce catastrophic forgetting specific to math, or by using models of different scales within the same family — and measure the relationship between initial pass@1 (or ground-truth ratio) and post-TTRL improvement. The goal would be to identify whether there is a sharp threshold (e.g., TTRL works reliably when initial ground-truth ratio > X%, fails catastrophically below Y%) or a smooth function, and whether reward accuracy collapses before label accuracy does. This would convert TTRL from a tool that sometimes works into one whose applicability can be predicted in advance without ground-truth labels, using only the model's own output statistics (majority ratio, entropy, output diversity metrics).

Characterizing reward accuracy dynamics throughout training. The paper's Lucky Hit analysis (Section 4.2, Figures 9, 10) treats reward accuracy as a static property at initialization, but the mechanism implies a non-monotonic dynamic. As the policy improves, two competing forces act on reward accuracy: (1) label accuracy increases (the majority vote becomes more often correct), which increases reward accuracy for the positive class; (2) output diversity decreases (the model converges on specific answers), which reduces the Lucky Hit effect and can increase the rate of false positive rewards (outputs that match a wrong but now-consensus answer). It is possible that reward accuracy initially worsens as the model moves from scattered incompetence to confident wrongness, before eventually improving as label accuracy catches up. Tracking label accuracy, reward accuracy, majority ratio, and output diversity (number of unique answers per 64-sample rollout) across training episodes for benchmarks of varying difficulty would reveal whether this U-shaped dynamic exists and whether it explains any of the training instability observed in Figure 11. If the U-shape is real, it would motivate adaptive temperature or diversity-injection mechanisms that maintain output scatter during the vulnerable phase of training.

TTRL on code generation with execution-based self-consistency. The paper's benchmarks all involve extractable math/science answers. Code generation presents a compelling extension because it shares the verifiable-output property (code either passes tests or doesn't) but differs in a critical way: majority voting on surface-form outputs is meaningless (different correct implementations of the same function may share no tokens, and small syntactic differences can break exact-match comparison). However, execution-based majority voting is well-defined: generate N implementations, execute each against unit tests, and use the most common output behavior as the pseudo-label. The Lucky Hit mechanism becomes even more interesting here — wrong implementations can produce a variety of incorrect outputs (scattered), making reward accuracy potentially high. A concrete experiment: apply TTRL to HumanEval or MBPP using only the problem descriptions (no ground-truth solutions or tests), generate 64 candidate solutions, execute them on a set of input-output examples generated by the model itself (self-generated tests), use the most common output behavior as the pseudo-label, and train with GRPO. The key metrics would be whether TTRL improves pass@1 on the held-out test cases, whether self-generated tests provide sufficient signal, and how reward accuracy compares to the math domain.

Combining TTRL with compute-optimal test-time budget allocation. TTRL uses a uniform budget: every question gets 64 samples, every episode, across all benchmarks (only the number of episodes varies by benchmark size). The paper's own difficulty-level analysis (Table 3) shows that TTRL benefit varies dramatically within a single benchmark — MATH-500 L1 improves by +45.4 points, L5 by only +16.8. This implies that uniform allocation is wasteful: compute is spent on questions that are already solved (easy ones where the model gets the right answer regardless) and questions that are unsolvable (hard ones where prior knowledge is absent). A natural integration with the compute-optimal test-time scaling framework (Snell et al., 2024) would work as follows: after the first episode, use per-question statistics — majority ratio (how concentrated the outputs are) and majority voting reward (how often individual outputs agree with the consensus) — to estimate each question's improvability. Allocate more episodes to questions in the "medium improvable" band (high output diversity, moderate-to-low initial correctness) and fewer to questions at either extreme. This would directly test whether TTRL's headline improvements can be achieved with substantially less total compute, and would produce a deployable TTRL system that doesn't require hand-specifying episode counts per benchmark.

Adversarial evaluation of TTRL's robustness to reward hacking. The paper argues that majority voting "mitigates reward hacking" because the reward signal co-evolves with the policy rather than being a fixed, gameable target (Section 5.2). This claim is plausible but untested. A systematic adversarial evaluation would test whether a policy can learn to exploit the majority-voting reward function: (a) can the model learn to produce outputs that format correctly (extractable answers) but contain no actual reasoning, winning majority votes through surface patterns? (b) given a benchmark with a small output space (e.g., GPQA with 4 choices), can the model converge to outputting the same wrong answer deterministically, making the majority vote always "correct" under the self-reward? (c) can the model learn to produce outputs that are syntactically diverse (different chain-of-thought) but converge to the same answer, maximizing consensus without improving actual correctness? These failure modes would manifest as high majority voting reward with low (or even decreasing) true accuracy — a signature that the paper's ground-truth-free metrics should detect. The GPQA results in Table 1 (Qwen2.5-Math-7B: 29.1 → 27.7) may already reflect this, but the paper does not analyze whether the degradation is due to reward hacking or simply insufficient prior knowledge. Controlled experiments manipulating the output space size (e.g., synthesizing math problems where the answer set is {1, 2, 3, 4} vs. problems where answers are arbitrary real numbers) would directly test whether the Lucky Hit mechanism protects against hacking or eventually fails as output diversity collapses.

Unifying TTRL with process-level or outcome reward models for harder benchmarks. The Lucky Hit mechanism is most effective when the model's incorrect outputs are scattered, providing many correct negative rewards. On very hard benchmarks like AIME 2024 (where the paper reports only 37% initial label accuracy), the model's initial correctness is low and its errors are scattered — the ideal scenario for Lucky Hit. But the paper shows TTRL works well here (211% improvement), implying the reward signal is sufficiently clean. The real test is on benchmarks of intermediate difficulty where the model's errors start to cluster — it consistently produces the same wrong answer — causing the Lucky Hit effect to break down. In these regimes, augmenting the majority-voting reward with a process-level reward model (PRM) or outcome reward model (ORM) trained on other labeled data could maintain reward accuracy. A concrete experiment: train an ORM on MATH training data (labeled), apply TTRL to AIME 2024 using a hybrid reward that weights majority-voting consensus and ORM score (e.g., reward = 1 if either consensus match OR ORM score > threshold, else 0). This tests whether external verifiers can compensate when self-consistency breaks down, and addresses the paper's acknowledged limitation that "TTRL does not incorporate mechanisms such as data filtering to support curriculum learning" (Section 4.3). More ambitiously, the ORM could be used to estimate question difficulty before training begins, enabling the adaptive budget allocation described above.

Practical Applications and Downstream Use Cases

Rapid adaptation to new evaluation benchmarks without annotation. The most direct application of TTRL is exactly what the paper demonstrates: taking a pre-trained model and a newly released benchmark (e.g., ARC-AGI-2, a new competition math dataset, a domain-specific reasoning test), and improving the model's performance on that benchmark without requiring any solutions or labels. For a research lab or competition team facing a new benchmark, the workflow would be: (1) run TTRL on the unlabeled test questions for 10–80 episodes (depending on dataset size), using the same hyperparameters the paper reports (temperature 0.6 or 1.0, 64-sample rollouts, GRPO with learning rate $5 \times 10^{-7}$ ); (2) monitor entropy and majority voting reward to detect training failure early; (3) evaluate the improved model, with the understanding that gains will be largest on questions within the model's prior knowledge range. The paper's cross-task generalization result (Figure 4) further suggests that improvements transfer to related benchmarks, so a lab could train TTRL on one unlabeled math benchmark and expect nontrivial gains on another — reducing the need to run TTRL separately for every new evaluation.

Self-improving deployment on continuously arriving unlabeled queries. For a production system handling a stream of user queries — particularly in domains like math tutoring, competitive programming, or scientific reasoning — TTRL offers a mechanism for continuous online improvement. As users submit novel problems that the model struggles with, the system can: (1) buffer arriving queries that the model shows high uncertainty on (low majority ratio, high entropy); (2) periodically run TTRL on the buffered queries during low-traffic periods, using the majority-voting reward mechanism that requires no human annotation; (3) deploy the improved model, which now handles similar future queries better. The paper's results on DeepSeek-R1-LLaMA-8B (+17.5 points on AIME, Table 2) are particularly relevant here: even a model that has already undergone expensive post-training can benefit from TTRL on a specific query distribution, making this a lightweight complement to periodic full-scale retraining. The key practical consideration is the compute cost — 64 samples × 80 episodes for a set of buffered queries — which may be justified for high-value or frequently recurring problem types but not for one-off queries.

Diagnostic tool for capability assessment in model development. When developing or fine-tuning a reasoning model, it is often unclear whether poor benchmark performance reflects a fundamental capability gap (the model genuinely doesn't understand the problem type) or an inference-time reasoning deficit (the model has the knowledge but doesn't explore enough reasoning paths to find the right answer). TTRL provides a direct diagnostic: if TTRL produces large gains (e.g., 100%+ improvement as in Table 1), the model had substantial latent competence that better inference or self-training can unlock, suggesting that additional inference-time strategies or further RL training (even unsupervised) would help. If TTRL produces minimal gains or degradation (e.g., Mistral-Nemo-Instruct on AIME: 0.8 → 0.0, Table 2), the model fundamentally lacks the necessary knowledge, and further investment in pretraining, instruction data, or domain-specific fine-tuning is needed. The paper's ground-truth-free metrics (entropy, majority ratio, majority voting reward) allow this diagnostic to be run without labels, making it suitable for use on proprietary benchmarks or internal evaluation sets where ground-truth answers aren't available. A model development team could run a standardized TTRL diagnostic (fixed hyperparameters, 10–30 episodes) on each new model checkpoint to quantify the "improvability gap" and guide resource allocation between pretraining and post-training efforts.