ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

ArXiv: 2601.06487

🎯 Pitch

Standard RL bakes in noise and stalls when its reward model compresses trajectory scores into a narrow band; ArenaRL breaks this deadlock by swapping absolute scores for a tournament-style relative ranking, and its seeded single-elimination bracket approximates full pairwise comparisons at $Σ(N)$ cost. On Qwen3-8B, this shift lifts open-ended travel planning win rates from 16% to 42% and yields nearly perfect validity in deep research, establishing that how you compare matters far more than what score you assign.

---

1. Executive Summary

This paper proposes ArenaRL, a reinforcement learning framework that replaces unstable pointwise scalar reward scoring with intra-group relative ranking to address a phenomenon the authors term discriminative collapse (where an LLM judge compresses scores of similar high-quality trajectories into a narrow, noise-dominated range) on open-ended agent tasks. Evaluating on two newly constructed benchmarks — Open-Travel (multi-constraint itinerary planning) and Open-DeepResearch (autonomous retrieval and report generation) — and on open-ended writing tasks, all using Qwen3-8B as the backbone model, ArenaRL introduces a process-aware pairwise evaluation mechanism (comparing trajectories via multi-level rubrics for reasoning coherence, tool invocation, and answer reliability) and a seeded single-elimination tournament topology (using greedy-decoded anchors for initial seeding before bracket-based elimination) that achieves near-equivalent advantage estimation accuracy to full pairwise comparisons while reducing complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$. ArenaRL achieves a 41.8% average win rate on Open-Travel (versus 16.4% for GRPO and 17.2% for GSPO) and a 64.3% win rate with 99% valid generation on Open-DeepResearch, establishing that tournament-based relative ranking enables robust policy optimization even on tasks without objective ground-truth, though the gains rely on an LLM-based arena judge whose fidelity was validated at a 73.9% agreement rate with human evaluations.

2. Context and Motivation

The Core Problem: RL for Open-Ended Agent Tasks Lacks Reliable Reward Signals

The fundamental question this paper tackles is: how do you apply reinforcement learning to improve LLM agents on tasks where there is no ground-truth answer? Over the past several years, RL has driven dramatic improvements in LLM capabilities on tasks with verifiable outcomes — mathematical reasoning, where a final numerical answer can be checked against a known solution, and code generation, where unit tests provide an unambiguous correctness signal. Algorithms like GRPO (Group Relative Policy Optimization) and its variants have been remarkably successful in these domains precisely because they can rely on deterministic, rule-based reward functions that unambiguously separate correct from incorrect outputs.

But this success has not transferred to a much broader and practically important class of tasks that the authors describe as open-ended agentic tasks — complex, multi-step problems requiring tool use, long-horizon planning, and integration of retrieved information, where the "quality" of an output is inherently subjective, multi-dimensional, and resistant to binary correct/incorrect judgments. Examples include personalized travel itinerary planning (where budget constraints, time windows, personal preferences, and spatiotemporal coherence must all be balanced), in-depth industry analysis and report generation (where factual accuracy, analytical depth, structural coherence, and practical usefulness all matter), and general open-ended writing. These tasks represent exactly the kind of real-world problems that LLM agents are increasingly expected to handle, yet the RL methods that work for math and code break down here for a specific, well-defined reason.

The Mechanism of Failure: Discriminative Collapse

The standard approach when ground-truth rewards are unavailable is the LLM-as-Judge paradigm: use a powerful LLM to assign a pointwise scalar score to each generated trajectory, then feed those scores into a standard RL algorithm like GRPO. The paper identifies a fundamental failure mode of this approach that they term discriminative collapse, illustrated in Figure 1(a).

Here is the chain of events that leads to collapse. During RL training, the policy is progressively refined, and the trajectories it generates become increasingly similar in distribution — they are all "pretty good," clustering in a narrow band of high-quality solutions. When faced with a group of such similar trajectories, the LLM judge struggles to discern subtle but meaningful advantages. Its scores become compressed into a narrow range (the paper gives an example of 0.8–0.9 on a 0–1 scale), making them effectively indistinguishable. This is not merely a precision issue — it is a signal-to-noise problem. The paper presents empirical evidence (Figure 1a) that the intra-group variance of these scores ($\sigma_{\text{group}}$) is comparable in magnitude to the noise variance ($\sigma_{\text{noise}}$) introduced by the judge's own unreliability — factors such as decoding randomness, length preferences, positional biases, and other spurious correlations that have nothing to do with actual trajectory quality. The resulting signal-to-noise ratio is extremely low.

The critical consequence arises from how algorithms like GRPO compute advantages. GRPO normalizes raw scores within a group by subtracting the group mean and dividing by the group standard deviation: $A_i = (\hat{R}_i - \mu) / \sigma$. This normalization is designed to create a relative scale within each batch. But when $\sigma_{\text{group}}$ approaches zero — precisely what happens under discriminative collapse — this normalization term amplifies the noise: small, meaningless differences in the judge's scores get blown up into large advantage signals. The policy then optimizes toward these spurious signals rather than toward genuine quality improvements. As the paper states:

"In this situation, the pointwise evaluation mechanism struggles to distinguish truly superior samples. And the RL optimization process is driven more by spurious noise than by meaningful task-specific rewards, leading performance to stagnate or even deteriorate."

This is a self-reinforcing pathology: as training proceeds and trajectories become more similar, the effective reward signal degrades further, making it impossible for the policy to identify and amplify genuinely better reasoning patterns. The optimization stagnates, and in many cases the policy can actually degrade — the paper notes that GRPO and GSPO in the Open-DeepResearch setting achieve even lower valid generation rates than the SFT baseline (17% and 21% versus 32% for SFT, Table 3), suggesting that the noisy optimization process actively damages the model's ability to complete tasks.

Why This Problem Matters

The practical stakes are substantial. Open-ended agentic tasks are not an academic curiosity — they represent the frontier of what LLM agents are being asked to do in deployment. Travel planning agents must satisfy multiple hard constraints simultaneously (budget, time, distance, user preferences) while coordinating multiple tool calls (POI search, navigation, flight/train search). Deep research agents must autonomously search, read, synthesize, and compose reports over long horizons, maintaining factual accuracy and logical coherence across potentially dozens of tool interactions. In these settings, getting RL to work is not just about incremental accuracy gains — it determines whether the agent can complete the task at all, as evidenced by the dramatic differences in valid generation rates (99% for ArenaRL versus 32% for SFT on Open-DeepResearch).

The theoretical significance lies in identifying a fundamental limitation of pointwise scoring for open-ended evaluation. The paper draws on decision theory, citing the well-established principle that pairwise preference judgments are more stable and reliable than pointwise quantitative assessments (Fürnkranz and Hüllermeier, 2010). This principle has been exploited successfully in preference-based alignment methods like RLHF and DPO (Rafailov et al., 2023), where human or LLM preferences between pairs of outputs are used to train reward models or directly optimize policies. But preference-based approaches have not been systematically integrated into online policy optimization for multi-step agent trajectories, where the computational cost of pairwise comparisons becomes a binding constraint.

Where Existing Approaches Fall Short

The paper identifies specific limitations across several categories of prior work.

1. Pointwise scoring with LLM judges (the standard GRPO approach for open-ended tasks). Mainstream RL algorithms for language models (GRPO, GSPO, DAPO) were designed for and validated on tasks with verifiable rewards. When adapted to open-ended settings by substituting an LLM judge's scalar score for a ground-truth reward, they inherit the discriminative collapse problem described above. The paper's experiments confirm that GRPO and GSPO fail to improve meaningfully over the SFT baseline on Open-Travel (16.4% and 17.2% win rates versus 16.4% for SFT, Table 3) and actually degrade task completion ability on Open-DeepResearch. Several recent works have attempted to improve this paradigm by using structured rubrics, multi-dimensional scoring, or constraint-based evaluation (Viswanathan et al., 2025; Huang et al., 2025; Ning et al., 2025), but these approaches remain fundamentally pointwise — they produce a scalar score per trajectory — and are therefore susceptible to the same discriminative collapse as the policy improves and trajectories converge.

2. Pairwise preference methods for LLM alignment (RLHF, DPO). The use of pairwise comparisons to train reward models or directly optimize policies is well-established in the alignment literature. However, these methods are designed for single-turn response generation, not for multi-step agent trajectories. They do not address the computational bottleneck of scaling pairwise comparisons to groups of trajectories during online training. Furthermore, preference-based methods typically treat the comparison as a binary signal (which response is better), whereas agent trajectory evaluation requires fine-grained, multi-dimensional assessment that captures partial advantages (e.g., trajectory A has better tool usage but trajectory B has more coherent reasoning). The binary preference signal discards this granularity.

3. Writing-Zero (Jia et al., 2025). This approach is the closest prior work to ArenaRL in philosophy: it also abandons pointwise scoring in favor of comparison-based signals. Writing-Zero assigns binary positive/negative advantages by comparing each generated response against a randomly selected reference response. While this has shown promise on open-ended writing tasks, the paper identifies two critical limitations: (a) it provides only coarse-grained binary guidance — a sample is either better than the reference or it isn't, with no ability to distinguish degrees of superiority — and (b) the random reference selection introduces variance, since the quality of the reference is uncontrolled. ArenaRL's tournament-based approach produces richer, multi-level relative rankings rather than binary comparisons, and its seeded structure ensures that comparisons are made against informative, appropriately matched opponents.

4. Pref-GRPO (Wang et al., 2025b). This method derives rewards from win rates computed via exhaustive pairwise comparisons among a group of generated samples. Conceptually, this is the round-robin approach that ArenaRL considers as its "gold standard" upper bound. The problem is computational: $\mathcal{O}(N^2)$ pairwise comparisons for a group of size $N$ becomes prohibitive for online RL training, especially when trajectories are long (deep research trajectories can span tens of thousands of tokens). Each comparison requires processing two full trajectories through a judge LLM, and $N(N-1)/2$ such comparisons per group per training step is infeasible at scale. ArenaRL's core contribution is identifying a tournament topology (seeded single-elimination) that achieves near-equivalent ranking accuracy with only $\mathcal{O}(N)$ comparisons.

5. Existing open-ended agent benchmarks are static and incomplete. The paper argues that prior benchmarks for open-ended agents (e.g., VitaBench, DeepResearchBench, DeepResearchGym) are predominantly static test suites that support only post-hoc evaluation. They provide test queries and evaluation protocols but lack the complementary training pipelines — SFT data, RL queries, and systematic evaluation rubrics — needed to study the full lifecycle of agent improvement. This makes it impossible to systematically study RL for open-ended agents because there is no standardized infrastructure for cold-start training, online exploration, and multi-dimensional evaluation. The paper constructs Open-Travel and Open-DeepResearch specifically to fill this infrastructure gap, providing a complete "SFT → RL → multi-dimensional automated evaluation" pipeline.

How This Paper Positions Itself

ArenaRL positions itself at the intersection of two converging trends: the push to apply RL to increasingly open-ended, real-world agent tasks, and the recognition from decision theory and preference learning that relative comparisons are more reliable than absolute scoring. The paper's framing makes a clean conceptual break:

"We contend that such pointwise scoring suffers from an inherent discrimination collapse... To fundamentally address discriminative collapse, we draw inspiration from decision theory, where pairwise preference judgments are known to be more stable than pointwise quantitative assessments, and advocate a paradigm shift from pointwise scalar scoring to intra-group relative ranking."

The paper does not propose a new base RL algorithm. Instead, it proposes a new reward signal generation mechanism that can in principle be plugged into any policy optimization algorithm that expects advantage signals. The specific optimization objective in Equation 8 is a standard clipped advantage update with KL regularization, similar to what GRPO and other algorithms use. The innovation is entirely in how the advantages $A_i$ are computed — through tournament-based relative ranking rather than pointwise scalar scoring.

This positioning is important because it means ArenaRL is not competing with GRPO or GSPO on algorithmic grounds; it is competing on the quality and reliability of the training signal. The paper's central empirical claim is that for open-ended agent tasks, the quality of the reward signal is the binding constraint, not the choice of policy optimization algorithm, and that shifting from pointwise to tournament-based ranking addresses this constraint directly.

The computational efficiency argument is equally central to the positioning. The paper explicitly acknowledges that full round-robin pairwise comparison would provide the best ranking signal (Table 2 confirms it achieves the highest average win rate of 32.9%), but argues that this is pragmatically impossible for online training. The research question then becomes: what is the cheapest tournament structure that preserves ranking fidelity? The investigation of five topologies (Section 4) and the identification of seeded single-elimination as the sweet spot is a core contribution that distinguishes ArenaRL from prior comparison-based methods like Pref-GRPO (which uses exhaustive comparison) and Writing-Zero (which uses single-reference comparison).

Finally, the paper positions its benchmark construction as filling a critical infrastructure gap. By providing complete training-evaluation pipelines for two distinct open-ended agent domains (travel planning and deep research), the paper enables the community to systematically study RL for open-ended agents in a reproducible way — something that was not possible with static test-only benchmarks. The inclusion of SFT training data, RL training queries, and multi-dimensional automated evaluation with dual-judge protocols establishes what the paper hopes will become a standard evaluation framework for this emerging problem class.

3. Technical Approach

3.1 Reader Orientation

ArenaRL is a reward signal generation framework that converts a group of trajectories sampled from the current policy into a reliable relative ranking, which is then used to compute advantage estimates for standard policy gradient optimization. It solves the discriminative collapse problem — the tendency of LLM judges to compress scores of similar high-quality trajectories into a narrow, noise-dominated range during online RL — by replacing pointwise scalar scoring (a single number per trajectory) with a tournament-based pairwise comparison mechanism that produces stable, fine-grained relative rankings even when absolute quality differences are subtle, all while maintaining $\mathcal{O}(N)$ computational complexity through a novel seeded single-elimination tournament topology.

3.2 Big-Picture Architecture (Diagram in Words)

The ArenaRL system has five major components that operate in sequence during each RL training step:

1. Trajectory Sampler — samples a group of $N$ trajectories from the current policy $\pi_\theta$ for a given query $x$, including one deterministic anchor trajectory (greedy decoding, temperature=0) and $N-1$ exploratory trajectories (temperature=0.8) to provide diversity.

2. Arena Judge — an LLM-based evaluation module that takes pairs of trajectories and produces separate quality scores for each, using multi-level process-aware rubrics that assess reasoning coherence, tool invocation correctness, and answer reliability. The judge employs bidirectional scoring (evaluates each pair twice with swapped positions) to eliminate positional bias.

3. Tournament Engine — organizes the $N$ trajectories into a competition bracket using a seeded single-elimination topology. The engine first computes preliminary seed rankings via anchor-based comparisons, then arranges matchups (highest seed vs. lowest seed) and conducts pairwise matches through a binary tree structure, advancing winners and recording scores at each round.

4. Ranking-to-Advantage Converter — takes the final tournament ranking (position $0$ through $N-1$) and transforms it into normalized advantage signals $A_i$ through quantile-based reward assignment followed by group-wise standardization.

5. Policy Optimizer — applies a standard clipped advantage policy gradient update with KL-divergence regularization against a reference policy, using the tournament-derived advantages $A_i$ in place of traditional scalar rewards.

Information flows as follows: a batch of queries enters → the trajectory sampler produces $N$ trajectories per query → the tournament engine seeds trajectories using anchor comparisons → pairwise tournament matches produce relative scores → rankings are converted to advantages → the policy is updated using these advantages → the updated policy generates new trajectories in the next training step.

3.3 Roadmap for the Deep Dive

• First, the formal problem setup and the RL objective (Section 3.1 of the paper), establishing the mathematical framework into which ArenaRL's reward signal fits — this defines the target distribution, the trajectory structure, and why standard reward functions fail.
• Second, the discriminative collapse mechanism (Section 3.2), because understanding exactly how pointwise scoring breaks is prerequisite to understanding why pairwise ranking fixes it.
• Third, the process-aware pairwise evaluation mechanism (Section 3.3), which is the atomic comparison unit that all tournament topologies depend on — its rubric structure, bidirectional scoring protocol, and what makes it "process-aware" rather than just outcome-based.
• Fourth, the five tournament topologies (Sections 4.1–4.5), progressing from the computationally infeasible gold standard (round-robin) through the efficient but resolution-limited (anchor-based) to the optimal balance point (seeded single-elimination), with brief treatment of the two alternatives that were tested but found inferior (double-elimination, Swiss-system).
• Fifth, the ranking-to-advantage conversion (Section 4.6), which is the bridge between the tournament output and the policy gradient — crucial because the optimization algorithm expects advantages, not tournament positions.
• Sixth, the full training pipeline, including cold-start SFT, RL hyperparameters, and the specific model and hardware configurations, because implementation details critically affect reproducibility.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that reliable reward signals for open-ended agent RL can be obtained by replacing pointwise scalar evaluation with tournament-based relative ranking, and that a seeded single-elimination topology achieves the optimal balance between ranking accuracy and computational cost.

---

Problem Formalization and RL Objective

The paper formulates the open-ended agentic task as a conditional trajectory generation problem. An agent policy $\pi_\theta$ synthesizes, for a query $x$ sampled from a task distribution $\mathcal{D}$, a multi-step interaction trajectory $\tau$ defined as an interleaved sequence of reasoning steps, tool calls, environmental feedback, and a final answer:

$\tau = [z_1, a_1, o_1, \dots, z_k, a_k, o_k, \dots, z_K, a_K, o_K, y]$

where $z_k$ is the chain-of-thought reasoning at step $k$, $a_k \in \mathcal{T}$ is a tool invocation from the set of available tools $\mathcal{T}$, $o_k$ is the environmental feedback from executing that tool, and $y$ is the final answer. The total trajectory length $K$ varies per query.

The RL objective follows the standard KL-regularized policy optimization formulation:

$\mathcal{L}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \tau \sim \pi_\theta(\cdot|x;\mathcal{T})} \left[ r_\phi(x, \tau) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta(\cdot|x) \parallel \pi_{\text{ref}}(\cdot|x)) \right]$

where $r_\phi(x, \tau)$ is the reward signal evaluating the quality of trajectory $\tau$, $\pi_{\text{ref}}$ is the reference policy (typically the SFT checkpoint used to initialize RL), and $\beta$ is a coefficient controlling KL divergence regularization strength.

What it computes: the expected reward of trajectories generated by the current policy, minus a penalty proportional to how far the current policy has diverged from the reference policy, averaged over the task distribution. The expectation over $\tau \sim \pi_\theta$ means trajectories are sampled on-policy — the current model generates its own training data.

Why this form: the KL penalty prevents the policy from collapsing to a degenerate distribution that maximizes the reward signal at the expense of general language capability (a form of reward hacking). This is standard practice in LLM RL (used in GRPO, PPO, and related algorithms). The critical component — and the one ArenaRL modifies — is the reward term $r_\phi(x, \tau)$, which in standard approaches is a pointwise scalar but in ArenaRL becomes a tournament-derived advantage signal.

The paper notes that the ground-truth reward function $R^*$ is intractable for open-ended tasks because there are no verifiable rules. Standard approaches therefore substitute an approximate reward $\hat{R}(\tau)$, modeled as the true utility corrupted by noise:

$\hat{R}(\tau) = R^*(\tau) + \epsilon$

The noise term $\epsilon$ captures all sources of judge unreliability — decoding stochasticity, length bias, positional preferences, and other spurious correlations. The discriminative collapse analysis shows that as the policy improves and $\sigma_{\text{group}}$ shrinks, the noise term $\epsilon$ becomes the dominant component of the reward signal, making $\hat{R}(\tau)$ effectively useless for optimization.

---

The Discriminative Collapse Mechanism (Why Pointwise Scoring Fails)

The paper formalizes discriminative collapse as a progressive degradation of the reward signal's informativeness during online RL training. The mechanism unfolds in four stages:

Stage 1: Trajectory convergence. As the policy $\pi_\theta$ is optimized, the trajectories $\{\tau_i\}_{i=1}^G$ it generates for a given query become increasingly similar in distribution. This is a natural consequence of successful optimization — the policy concentrates probability mass on high-quality solutions, reducing diversity. The group variance $\sigma_{\text{group}}$ (the variance of the true utilities $R^*(\tau_i)$ among trajectories in the group) shrinks toward zero.

Stage 2: Judge uncertainty. When faced with trajectories that are all similarly good, the LLM judge enters a regime of high epistemic uncertainty. It cannot reliably determine which trajectory is better because the differences are subtle and multi-dimensional. The paper describes this as the judge "hesitating," exhibiting high-variance scoring behavior driven by spurious correlations rather than genuine quality differences. The paper provides a concrete example: scores are compressed into a range of 0.8–0.9 on a 0–1 scale, making them effectively indistinguishable.

Stage 3: Noise dominance. The observed scores $\hat{R}(\tau_i)$ become primarily reflections of the noise term $\epsilon$ rather than the true utility $R^*(\tau_i)$. The paper's empirical analysis (Figure 1a) shows that the intra-group variance $\sigma_{\text{group}}$ of the true utilities is comparable in magnitude to the noise variance $\sigma_{\text{noise}}$ (the variance of $\epsilon$), resulting in an extremely low signal-to-noise ratio. The judge's output is essentially random with respect to actual trajectory quality.

Stage 4: Noise amplification through normalization. This is the catastrophic step. Algorithms like GRPO compute advantages by normalizing scores within a group:

$A_i = \frac{\hat{R}_i - \mu}{\sigma}$

where $\mu$ and $\sigma$ are the mean and standard deviation of the scores $\{\hat{R}_1, \dots, \hat{R}_G\}$. When $\sigma$ is small (because all trajectories are similar and scores are compressed), this normalization amplifies the noise-dominated score differences into large-magnitude advantage signals. The policy receives strong gradient updates in directions that are determined by judge noise rather than by genuine quality differences. The paper states:

"As $\sigma_{\text{group}}$ vanishes, the normalization term inadvertently amplifies this drift-induced noise into dominant gradient signals. Consequently, the optimization process is hijacked by the reward model's interfering noise, leading to performance stagnation or even degeneration."

This is why the paper observes that GRPO and GSPO can actually degrade task completion ability (valid generation rates dropping from 32% for SFT to 17% and 21% for GRPO and GSPO on Open-DeepResearch, Table 3). The noisy optimization process actively damages the policy's ability to produce coherent tool-use trajectories.

The discriminative collapse analysis is the paper's primary theoretical motivation. It identifies a structural limitation of pointwise evaluation that cannot be fixed by simply using a better judge model or better rubrics — the problem is inherent to the single-trajectory scoring paradigm when trajectories converge in quality.

---

Process-Aware Pairwise Evaluation (The Atomic Comparison Unit)

The foundation of ArenaRL's ranking mechanism is a pairwise comparison procedure that evaluates two trajectories simultaneously and produces separate quality scores for each. This is the basic operation that all tournament topologies call repeatedly. The paper introduces several design choices that make this evaluation more reliable than a naive "which is better?" comparison.

The Arena Judge $\mathcal{J}$. At inference time, the judge is an LLM (specifically Qwen3-Max, as stated in Appendix A) that receives three inputs:

1. The user query $x$ — the original task description that both trajectories were attempting to solve.

2. The core context of trajectories $\tau_i$ and $\tau_j$ — containing, for each trajectory, the chain-of-thought reasoning $z_k$ at each step, the tool invocations $a_k$ and their parameters, and the final answer $y$. Environmental feedback $o_k$ (the raw outputs from tool calls) is not included in the judge's input. This is a deliberate choice: the paper states in Appendix A that "tokens corresponding to environmental feedback are masked out from the loss computation" during RL, focusing optimization on the agent's own reasoning and action choices rather than on memorizing tool outputs.

3. A comprehensive process-aware rubric $u$ — a structured evaluation guide that instructs the judge to assess trajectories along three dimensions:

   • Logical consistency of the chain-of-thought — does the reasoning follow coherently from step to step, without contradictions or leaps?
   • Precision of tool calls — are the right tools invoked with the right parameters at the right times? Are tool results used appropriately in subsequent reasoning?
   • Reliability of the final answer — does the answer correctly satisfy the user's constraints, incorporate information from tool calls, and avoid hallucinations?

The rubrics are domain-specific. For Open-Travel, the rubric evaluates whether the itinerary respects time windows, budget constraints, travel distances, and user preferences. For Open-DeepResearch, the rubric uses seven sub-criteria (Framework, Tool Usage, Coverage, Relevance, Accuracy, Depth, Clarity) defined in Section 6.1. The complete judge prompts for each domain are provided in Appendix E (Figures 5, 6, 7, 8).

Bidirectional scoring protocol. A critical practical issue with LLM-based pairwise evaluation is positional bias — judges tend to prefer the answer presented first (or sometimes second), regardless of quality. To eliminate this bias, ArenaRL evaluates each pair of trajectories twice, swapping their presentation order:

$(s_i, s_j) = \mathcal{J}(x, \tau_i, \tau_j, u) + \mathcal{J}(x, \tau_j, \tau_i, u)$

where the first call $\mathcal{J}(x, \tau_i, \tau_j, u)$ presents $\tau_i$ first and $\tau_j$ second, the second call $\mathcal{J}(x, \tau_j, \tau_i, u)$ presents them in reverse order, and the two sets of scores are summed element-wise. The final score $s_i$ for trajectory $\tau_i$ is the sum of its score when presented first and when presented second, and similarly for $s_j$.

What it computes: for each trajectory in the pair, a scalar quality score that aggregates the judge's assessment across all rubric dimensions, with positional bias cancelled out by averaging over both presentation orders. The scores $s_i$ and $s_j$ are on an arbitrary scale determined by the judge's output format but are comparable within the pair.

Why this form: The summation (rather than averaging) preserves the full range of the scores. The bidirectional protocol is a standard technique in LLM evaluation literature (citing Wu et al., 2025a) and is particularly important here because the tournament relies on fine-grained score differences — a positional bias of even a few points could systematically advantage whichever trajectory happens to appear first in the bracket. The process-aware rubric, with its three-level structure (reasoning, tools, answer), ensures that the optimization signal reinforces "the agent's intrinsic reasoning capabilities rather than merely overfitting to surface-level features of the final answer." This is crucial because in agent tasks, a correct final answer produced through flawed reasoning or tool misuse represents a brittle capability that will not generalize.

Positioning against alternatives. The paper contrasts this pairwise mechanism against two alternatives. Standard pointwise scoring (assigning a single number to each trajectory independently) suffers from discriminative collapse because the judge has no reference point — it cannot calibrate its scale across trajectories seen at different times or in different contexts. Binary preference judgments (simply stating "A is better than B") discard information about the magnitude of the quality difference, which matters for ranking within a group. The process-aware pairwise mechanism with separate per-trajectory scores provides a continuous signal that captures both the direction and the magnitude of quality differences.

---

Tournament Topologies: From Round-Robin to Seeded Single-Elimination

The central technical question in ArenaRL is: given a group of $N$ trajectories, how do you organize pairwise comparisons to produce an accurate relative ranking while minimizing the number of expensive judge calls? The paper systematically investigates five tournament topologies, using the computationally infeasible round-robin as a gold standard to benchmark the fidelity of more efficient alternatives. All topologies were evaluated under a unified RL configuration of group size $N=8$ and number of groups $K=8$, with results reported in Table 2.

Round-Robin Tournament (Gold Standard)

In this topology, every trajectory competes against every other trajectory exactly once, producing $N(N-1)/2$ pairwise comparisons. For each trajectory $\tau_i$, the score is the normalized win rate:

$\text{Score}(\tau_i) = \frac{1}{N-1} \sum_{j \neq i} \mathbb{I}(s_i > s_j)$

where $\mathbb{I}(\cdot)$ is the indicator function that returns 1 if $s_i > s_j$ (the pairwise score of $\tau_i$ exceeds that of $\tau_j$) and 0 otherwise.

What it computes: the fraction of opponents against which trajectory $\tau_i$ wins, ranging from 0 (lost to everyone) to 1 (beat everyone). The group ranking is determined by sorting these scores in descending order.

Why this form: the round-robin is theoretically unbiased because every trajectory is compared against the same set of opponents. There is no dependence on initial ordering or bracket structure. The paper uses it as the "gold standard" to establish an upper bound on achievable ranking accuracy — any more efficient topology should approach this accuracy. However, the $\mathcal{O}(N^2)$ complexity (for $N=8$, this is 28 comparisons per group) makes it "intractable for online training with a large group size $N$." Each comparison requires processing two full trajectories (potentially thousands of tokens each) through Qwen3-Max, making round-robin infeasible at the scale needed for RL training (hundreds of training steps with $K$ groups per step).

Anchor-Based Ranking

This topology introduces the concept of a quality anchor — a deterministic reference trajectory $\tau_{anc}$ generated via greedy decoding (temperature $T=0$) from the current policy. The remaining $N-1$ trajectories are generated via high-entropy sampling ($T=0.8$) to ensure exploration diversity. Each exploratory trajectory $\tau_i$ is compared only against the anchor, producing $N-1$ comparisons total:

$(s_i, s_{anc}^i) = \text{bidirectional pairwise evaluation of } \tau_i \text{ vs. } \tau_{anc}$

The anchor's overall score is computed as the average of its scores across all comparisons:

$s_{anc} = \frac{1}{N-1} \sum_{i=1}^{N-1} s_{anc}^i$

The group ranking is then determined by sorting the set of scores $\{s_1, s_2, \dots, s_{N-1}, s_{anc}\}$, which includes each exploratory trajectory's score and the anchor's averaged score.

What it computes: a linear-complexity ranking where each trajectory's position is determined by how it performs against a single common reference. The anchor trajectory serves as a fixed calibration point — since it is generated deterministically ($T=0$), it represents the policy's "best guess" for the query at the current training step.

Why this form: the $N-1$ comparisons achieve $\mathcal{O}(N)$ complexity, making this topology computationally viable for online training. The use of greedy decoding for the anchor is deliberate: greedy decoding produces the single most likely trajectory under the current policy, which serves as a stable reference point. If the anchor were also stochastic, its quality would vary across comparisons, introducing noise into the ranking.

However, the paper identifies a critical limitation: this topology suffers from a "loss of resolution." It effectively quantifies the extent to which each trajectory outperforms (or underperforms) the anchor, but it "fails to capture subtle differences between two exploratory samples." Two trajectories that both beat the anchor by similar margins will receive similar scores and may be ranked arbitrarily relative to each other, even if one is meaningfully better. This is acceptable for coarse ranking but problematic when the policy needs to distinguish among several high-quality candidates — precisely the regime where discriminative collapse is most severe.

Seeded Single-Elimination (The Chosen Topology)

This is ArenaRL's primary contribution to tournament design. It operates in two distinct phases:

Phase 1: Seeding. The anchor-based ranking mechanism from Section 4.2 is used to compute a preliminary score for each of the $N$ trajectories. These scores are used to assign seed rankings $s_{seed}^i$, where the highest-scoring trajectory receives seed 1, the next receives seed 2, and so on. This phase requires $N-1$ comparisons (each exploratory trajectory versus the anchor, plus the anchor's averaged score).

Phase 2: Elimination tournament. A binary tournament tree is constructed where matchups are arranged according to seed rankings. The pairing scheme follows the standard sports tournament convention: the highest seed faces the lowest seed, the second-highest faces the second-lowest, and so on (seed 1 vs. seed $N$, seed 2 vs. seed $N-1$, etc.). In each match, the two trajectories are evaluated via the bidirectional pairwise protocol:

$\tau_{\text{win}} = \arg\max_{\tau \in \{\tau_i, \tau_j\}} (s_i, s_j)$

The winner advances to the next round, and the loser is eliminated. The paper provides a detailed bracket-filling algorithm (Algorithm 1 in Appendix D) that handles the case where $N$ is not a power of 2 by using head and tail pointers to interleave high and low seeds.

Final ranking. The ranking is primarily determined by the depth of survival in the tournament bracket. The champion (last trajectory standing) receives rank 0. Trajectories eliminated in the finals receive ranks 1–2. Trajectories eliminated in the semi-finals receive ranks 3–6, and so on. Within each elimination tier (trajectories that lost in the same round), ties are broken using the accumulated average scores from all matches played so far, including the seeding phase.

What it computes: a complete ordering of all $N$ trajectories with $\mathcal{O}(N)$ comparisons — specifically, $N-1$ comparisons for seeding plus $N-1$ comparisons for the elimination phase (each of the $N-1$ matches eliminates one trajectory, requiring one comparison per elimination). The total is $2N-2$ comparisons, linear in $N$.

Why this form — the seeding phase rationale. The seeding phase is the key innovation that distinguishes this topology from a standard single-elimination tournament. Without seeding, a random pairing in the first round could cause two high-quality trajectories to meet early, eliminating one prematurely and degrading the accuracy of the final ranking. The paper's empirical analysis (Table 2) confirms this sensitivity: the standard Double-Elimination format (which the paper tests with random seeding to maintain comparable computational budget) achieves only 30.2% average win rate versus 32.5% for Seeded Single-Elimination, despite having a theoretically more robust structure (a loser's bracket that gives trajectories a second chance).

The seeding phase uses the anchor-based scores as a "low-bias initial estimate" of trajectory quality. The paper argues that while these scores lack resolution among similar trajectories (the limitation noted in Section 4.2), they are sufficiently accurate to prevent severely mismatched first-round pairings. The critical insight is that seeding does not need to be perfectly accurate — it only needs to be good enough to prevent the highest-quality trajectories from meeting each other in early rounds. Once the tournament begins, the head-to-head comparisons between appropriately matched opponents provide the fine-grained resolution that anchor-based ranking alone lacks.

Why this form — the pairing scheme rationale. Pairing the highest seed with the lowest seed (rather than adjacent seeds) maximizes the expected quality difference in early matches. This serves two purposes: (a) it reduces the probability of upsets (a slightly worse trajectory beating a slightly better one due to judge noise), and (b) it ensures that the strongest trajectories survive to later rounds where they face progressively stronger opponents. This is the standard seeding logic from sports tournaments, adapted here for the specific purpose of producing reliable advantage estimates for RL.

Why this form — empirical validation. Table 2 shows that Seeded Single-Elimination (32.5% average win rate) approaches the Round-Robin gold standard (32.9%) while requiring only $2N-2$ comparisons versus $N(N-1)/2$. For $N=8$, this is 14 comparisons versus 28 — exactly half. On two subtasks (Search and 1-Day), Seeded Single-Elimination actually outperforms Round-Robin. The paper attributes this counterintuitive result to the seeding mechanism "effectively filtering out noise" — in a round-robin, a high-quality trajectory might occasionally lose to a lower-quality one due to judge noise, contaminating its win-rate estimate. In a seeded single-elimination, such an upset would eliminate the high-quality trajectory, but the seeding makes upsets less likely by pairing trajectories with appropriately large quality gaps.

Double-Elimination Tournament

This topology introduces a losers' bracket, so a trajectory is eliminated only after sustaining two defeats. The ranking criteria mirror those of Seeded Single-Elimination (advancement depth and accumulated scores). To maintain a comparable computational budget (approximately $2N$ comparisons), the paper initializes this format with random seeding rather than anchor-based seeding. The total comparison count is approximately $2N-2$ (similar to Seeded Single-Elimination if one assumes most trajectories lose exactly once or twice).

What it computes: a ranking that is theoretically more robust to isolated upsets because a single unlucky loss does not eliminate a trajectory. The losers' bracket gives strong trajectories a path back to the finals even if they lose once.

Why it was not chosen: despite the theoretical robustness, empirical results (Table 2) show Double-Elimination achieves only 30.2% average win rate, well below Seeded Single-Elimination's 32.5%. The paper attributes this to the lack of high-quality initial seeds — without anchor-based seeding, the random initial pairings cause premature elimination of strong trajectories in the winners' bracket, and the losers' bracket only partially recovers from this. This result demonstrates that seeding quality is more important than bracket structure for ranking accuracy in this noise regime. The paper concludes that "without high-quality initial seeds, its ranking fidelity falls short of that achieved by Seeded Single-Elimination."

Swiss-System Tournament

This is a non-elimination format where all trajectories participate in a fixed number of rounds $K \approx \log_2 N$. In each round, trajectories with identical win-loss records are paired against each other (each round involves $N/2$ matches). The final ranking is determined by total wins plus the Buchholz score (the sum of wins achieved by a trajectory's past opponents, providing a strength-of-schedule tiebreaker).

What it computes: a ranking derived from approximately $K \times N/2 = \mathcal{O}(N \log N)$ comparisons. For $N=8$, this is about 12 comparisons (3 rounds × 4 matches per round). Every trajectory plays the same number of games, so there is no risk of a single unlucky match eliminating a strong candidate.

Why it was not chosen: Table 2 shows Swiss-System achieves 28.3% average win rate, below both Seeded Single-Elimination (32.5%) and Double-Elimination (30.2%). The paper suggests this is due to "insufficient comparison depth" — with only $\log_2 N$ rounds, the number of pairwise comparisons per trajectory is small, and the pairing algorithm may not create enough high-quality matchups to produce a reliable ordering. Additionally, at $N=8$ with 3 rounds, a trajectory only plays 3 matches regardless of its quality, providing limited signal for fine-grained ranking among closely matched candidates.

Summary of Topology Trade-offs

The paper's systematic comparison (Table 2) establishes that Seeded Single-Elimination achieves the best trade-off:

• Round-Robin: 32.9% average win rate, $\mathcal{O}(N^2)$ complexity — too expensive.
• Seeded Single-Elimination: 32.5% average win rate, $\mathcal{O}(N)$ complexity — the chosen topology.
• Double-Elimination: 30.2% average win rate, $\mathcal{O}(N)$ complexity — lacks effective seeding, less accurate.
• Swiss-System: 28.3% average win rate, $\mathcal{O}(N \log N)$ complexity — insufficient comparison depth.
• Anchor-Based Ranking: 27.8% average win rate, $\mathcal{O}(N)$ complexity — lacks resolution between similar trajectories.

The paper selects Seeded Single-Elimination as the primary topology for all subsequent experiments based on its near-optimal accuracy at linear computational cost. The detailed algorithm pseudocode is provided in Algorithm 1 (Appendix D), which includes the bracket-filling logic (alternating head and tail pointers to handle odd/even pairing), the hierarchical sorting within elimination tiers, and the accumulated score tracking.

---

Ranking-to-Advantage Conversion

The tournament engine produces, for each trajectory $\tau_i$ in the group of size $N$, a discrete rank $\text{Rank}(\tau_i) \in \{0, 1, \dots, N-1\}$, where 0 denotes the highest quality (the tournament champion) and $N-1$ denotes the lowest quality. Standard policy gradient algorithms expect continuous advantage signals, not discrete ranks, so ArenaRL applies a two-step conversion:

Step 1: Quantile-based reward mapping. Ranks are mapped to rewards in $[0, 1]$ using linear interpolation:

$r_i = 1 - \frac{\text{Rank}(\tau_i)}{N-1}$

What it computes: the best trajectory (rank 0) receives $r_i = 1.0$, the worst (rank $N-1$) receives $r_i = 0.0$, and intermediate ranks receive linearly spaced values. For $N=8$, ranks 0 through 7 map to rewards $1.0, 0.857, 0.714, 0.571, 0.429, 0.286, 0.143, 0.0$.

Why this form: linear spacing is the simplest mapping that preserves the ordinal information from the tournament while producing continuous values suitable for advantage computation. The paper does not explore nonlinear mappings (e.g., top-weighted or bottom-compressed), which could be an avenue for future work. The choice of linear spacing implicitly assumes that the quality difference between adjacent ranks is roughly constant, which is reasonable for a well-functioning tournament but may not hold in all cases.

Step 2: Group-wise advantage standardization. The rank-based rewards are standardized within the group to produce advantages:

$A_i = \frac{r_i - \mu_r}{\sigma_r + \epsilon}$

where $\mu_r$ and $\sigma_r$ are the mean and standard deviation of the rank-based rewards $\{r_1, \dots, r_N\}$, and $\epsilon$ is a small constant (not explicitly specified in the paper, but following standard practice, likely $10^{-8}$ or similar) to prevent division by zero.

What it computes: zero-mean, unit-variance advantages within the group. The best trajectories receive positive advantages (they should be imitated more), the worst receive negative advantages (they should be imitated less), and the magnitude of the advantage reflects how far a trajectory is from the group average.

Why this form: this standardization mirrors the advantage computation in GRPO ($A_i = (\hat{R}_i - \mu)/\sigma$), but with a crucial difference: the inputs $r_i$ are derived from relative rankings rather than absolute pointwise scores. In GRPO, when discriminative collapse occurs, $\sigma$ becomes very small (because all scores are compressed) and the normalization amplifies noise. In ArenaRL, the linear spacing of rank-based rewards ensures that $\sigma_r$ remains stable regardless of how similar the trajectories are in absolute quality — the ranks are always 0 through $N-1$, so the variance of $\{r_i\}$ is determined entirely by $N$ and the spacing scheme, not by the judge's score compression. This is the fundamental mechanism by which ArenaRL avoids the noise amplification problem.

Numerical example. For $N=8$ with the linear spacing above, $\mu_r = 0.5$ (the midpoint of a uniform distribution from 0 to 1 with 8 evenly spaced points) and $\sigma_r \approx 0.286$ (the standard deviation of $\{0, 1/7, 2/7, \dots, 1\}$). The best trajectory (rank 0, $r=1.0$) receives $A_i \approx (1.0 - 0.5) / 0.286 \approx 1.75$. The worst trajectory (rank 7, $r=0.0$) receives $A_i \approx (0.0 - 0.5) / 0.286 \approx -1.75$. These values are stable regardless of how the judge's actual pairwise scores distribute — the ranking structure guarantees a fixed advantage distribution.

---

Policy Optimization with Tournament-Derived Advantages

The final step plugs the tournament-derived advantages into a standard clipped advantage policy gradient objective:

$\mathcal{L}_{\text{ArenaRL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \mathcal{G} \sim \pi_\theta} \left[ \frac{1}{N} \sum_{i=1}^{N} \left( \min\left( \frac{\pi_\theta(\tau_i \mid x)}{\pi_{\text{old}}(\tau_i \mid x)} A_i, \text{clip}\left( \frac{\pi_\theta(\tau_i \mid x)}{\pi_{\text{old}}(\tau_i \mid x)}, 1 - \epsilon, 1 + \epsilon \right) A_i \right) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x)) \right) \right]$

where $\pi_\theta$ is the current policy, $\pi_{\text{old}}$ is the policy at the start of the current update (used for importance sampling correction), $\epsilon$ is the clipping parameter (not explicitly stated in the paper, but typically 0.2 in standard PPO/GRPO implementations), and $\beta$ is the KL penalty coefficient.

What it computes: the expected clipped advantage over the group of trajectories, minus a KL penalty, where the expectation is over queries sampled from the task distribution and trajectory groups sampled on-policy from the current model. The clipping prevents the policy ratio $\pi_\theta / \pi_{\text{old}}$ from deviating beyond $[1-\epsilon, 1+\epsilon]$, which bounds the effective step size per update. The minimum operator ensures that when the advantage is positive, the policy does not increase probability too aggressively (the clip provides a ceiling), and when the advantage is negative, the policy does not decrease probability too aggressively (the clip provides a floor).

Why this form — the clipping mechanism. The clipping is standard PPO-style optimization, chosen because it is the same mechanism used in GRPO and GSPO. This ensures that any performance differences between ArenaRL and baselines are attributable to the reward signal (tournament-based ranking vs. pointwise scoring) rather than to differences in the optimization algorithm. The paper is careful to isolate the contribution of the reward mechanism: "these baseline algorithms utilize the exact same judge models and evaluation rubrics as ArenaRL, and only evaluate the answer portion" (Section 6.1, describing the GRPO and GSPO baselines).

Why this form — the KL penalty. The KL divergence $\mathbb{D}_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ is computed between the current policy and the reference policy (the SFT checkpoint from which RL was initialized). The coefficient $\beta$ controls the trade-off between optimizing for tournament rank and staying close to the reference. The paper does not report the specific value of $\beta$, but the inclusion of this term follows standard practice to prevent reward hacking and maintain general language capabilities.

Implementation detail — token-level masking. Appendix A specifies that "tokens corresponding to environmental feedback are masked out from the loss computation." This means that the policy gradient update only affects the model's probability of generating its own reasoning tokens and tool calls, not the probability of generating the environment's responses (which are determined by external tools, not the model). This is standard in tool-use RL and ensures that the policy is optimized for decision-making, not for memorizing tool outputs.

---

Training Pipeline and Hyperparameters

The paper follows a "Cold-start → RL" paradigm with specific hyperparameter choices for each phase, as detailed in Appendix A.

Cold-start phase (SFT). The base model Qwen3-8B-Base is fine-tuned on domain-specific SFT datasets:

• For Open-Travel: 2,600 SFT samples covering four subtasks (Direction, Search, Compare, 1-Day). The M-Day subtask is excluded from SFT and used only for generalization evaluation.
• For Open-DeepResearch: 2,662 SFT samples covering multi-turn search, reading, synthesis, and report generation.
• For open-ended writing: 10,000 examples randomly sampled from DeepWriting-20K.

Training configuration: TRL framework with DeepSpeed ZeRO-3, 32 NVIDIA H20 GPUs, 3 epochs, learning rate $2 \times 10^{-5}$, batch size 1 per GPU (effective batch size of 32 with gradient accumulation across 32 GPUs — the paper does not explicitly state gradient accumulation, but this is standard for large-model SFT on these frameworks).

The SFT data is constructed using high-performing closed-source models as "base models to generate large-scale tool-use trajectories" (Stage II of benchmark construction, described in Section 5). The paper does not specify which closed-source models were used for SFT data generation.

RL phase. ArenaRL and all baselines are implemented on the Slime framework (Zhu et al., 2025). Key hyperparameters:

• Group size $N$: 16 for Open-Travel and open-ended writing; 8 for Open-DeepResearch (reduced to "enhance training efficiency" due to longer trajectories).
• Number of groups $K$: 8 for Open-Travel and open-ended writing; 4 for Open-DeepResearch (per training step).
• Optimizer: Adam with learning rate $1 \times 10^{-6}$.
• Arena Judge model: Qwen3-Max, used both for pairwise evaluations during RL training and for final benchmark evaluation (along with Claude-4-Sonnet as a dual judge for evaluation).
• Hardware: 8 NVIDIA H20 GPUs for RL training.
• Training steps: The paper does not specify the total number of RL training steps for main results, but the direct RL experiment (Figure 4c) shows training up to 160 steps on the Search subtask.

Anchor trajectory generation. The anchor $\tau_{anc}$ is generated using greedy decoding (temperature $T=0$) from the current policy. The $N-1$ exploratory trajectories are generated with temperature $T=0.8$ to ensure diversity. The paper does not specify the sampling algorithm (top-p, top-k, or pure temperature sampling), but the temperature difference between anchor (0) and exploratory (0.8) trajectories is the key mechanism for balancing stability and exploration in the tournament.

Cross-validation for tournament topology selection. The comparison of five tournament topologies (Table 2) was conducted under a "unified RL configuration" with $N=8, K=8$. The paper does not describe a separate validation procedure for topology selection — the final choice of Seeded Single-Elimination was based on the table of results, which appears to be evaluated on the Open-Travel test set directly. This is a minor methodological concern since the topology is effectively tuned on the test set, though the paper argues the choice is principled (balancing accuracy and efficiency) rather than cherry-picked.

---

Summary of Design Choices and Their Justifications

• Pairwise comparison over pointwise scoring: fundamentally avoids discriminative collapse by providing relative signals that remain informative even when absolute quality differences narrow. Pairwise judgments are known from decision theory to be more stable than absolute ratings.

• Process-aware rubrics over outcome-only evaluation: ensures the optimization signal reinforces the agent's reasoning and tool-use capabilities, not just surface-level answer features. Three-level rubric structure (reasoning coherence, tool precision, answer reliability) provides multi-dimensional feedback that discourages shortcuts.

• Bidirectional scoring over single-pass evaluation: eliminates positional bias in the LLM judge, which would otherwise systematically advantage whichever trajectory appears first in the comparison. Two evaluations with swapped positions are summed to cancel the bias.

• Seeded single-elimination over round-robin: reduces computational complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$ while preserving near-optimal ranking accuracy (32.5% vs. 32.9% average win rate). The $\mathcal{O}(N)$ complexity makes online RL training feasible at scale.

• Seeded single-elimination over anchor-based ranking: adds the elimination phase to capture fine-grained differences between similar trajectories, which the anchor-based approach alone cannot distinguish. The seeding phase provides sufficient initial ordering to prevent premature elimination of strong trajectories.

• Seeded single-elimination over double-elimination: anchor-based seeding is more important for ranking accuracy than the robustness of a losers' bracket. Without seeding, even a double-elimination format cannot recover from poor initial pairings.

• Linear rank-to-reward mapping over nonlinear alternatives: the simplest approach that preserves ordinal tournament information. More complex mappings (e.g., top-heavy weighting) could be explored but were not in this paper.

• Separate per-trajectory scores over binary win/loss signals: provides continuous advantage magnitudes rather than just direction, enabling the policy to learn not only which trajectories are better, but also how much better they are.

• Greedy anchor ($T=0$) over sampled anchor: provides a stable, deterministic reference point for seeding that does not vary across groups. A stochastic anchor would introduce noise into the seeding phase, potentially degrading the initial ordering and causing the elimination phase to inherit poor matchups.

• On-policy sampling over off-policy: trajectories are generated from the current policy $\pi_\theta$ at each training step, ensuring that the tournament evaluates the policy's current capabilities rather than outdated behavior. This is standard in online RL and is critical for the tournament to provide relevant gradient signals as the policy improves.

4. Key Insights and Innovations

Innovation 1: Discriminative Collapse as a Named, Formalized Failure Mode of Pointwise Reward Scoring

The paper's most conceptually significant contribution is not the ArenaRL algorithm itself, but the identification and formalization of discriminative collapse as the fundamental mechanism by which pointwise scalar reward scoring fails during online RL for open-ended tasks. This is a novel diagnostic concept that provides a unified explanation for why standard RL approaches (GRPO, GSPO, and their variants) fail on open-ended agent tasks despite working well on math and code.

Prior to this work, the field's understanding of why RL struggled on open-ended tasks was fragmented. Some attributed it to the absence of ground-truth rewards, others to the unreliability of LLM judges, and still others to the difficulty of credit assignment in long-horizon trajectories. The paper's contribution is to identify a specific, falsifiable mechanism that connects these symptoms into a coherent causal chain: (1) policy improvement causes trajectory convergence → (2) convergence causes the LLM judge's scores to compress into a narrow range → (3) score compression reduces the intra-group variance σ_group, making it comparable to judge noise σ_noise → (4) the normalization step in algorithms like GRPO ($A_i = (\hat{R}_i - \mu) / \sigma$) amplifies this noise into dominant gradient signals → (5) the optimization process is hijacked by spurious correlations, leading to stagnation or degeneration.

The diagnostic value of this framing is substantial. It explains the counterintuitive empirical finding that GRPO and GSPO can degrade task completion ability — the valid generation rate on Open-DeepResearch drops from 32% (SFT) to 17% (GRPO) and 21% (GSPO), per Table 3. Under a naïve view where LLM judges are merely "noisy but unbiased," one would expect RL to plateau at SFT-level performance rather than deteriorate. The discriminative collapse model explains the deterioration: the noise amplification step actively damages the policy by creating large-magnitude gradient updates in directions that are effectively random with respect to true quality. This is analogous to what happens in standard supervised learning when training on noisy labels — the model overfits to noise rather than signal — but the mechanism is specific to the normalization procedure used in RL advantage computation.

This framing also explains why prior work reached contradictory conclusions about the effectiveness of RL for open-ended tasks. Methods tested on tasks where the policy never achieved sufficient quality to trigger score compression (i.e., where σ_group remained large because trajectories were genuinely of varying quality) would observe positive RL gains. Methods tested on tasks or models where trajectories quickly converged to a narrow quality band would observe stagnation or deterioration. The discriminative collapse model predicts that the failure of pointwise RL is not a property of the task per se, but of the interaction between the policy's capability and the judge's discriminative resolution. This is a more precise and actionable diagnosis than simply saying "LLM judges are unreliable."

The formalization in Section 3.2 (modeling observed scores as $\hat{R}(\tau) = R^*(\tau) + \epsilon$, with σ_group and σ_noise as the key quantities governing signal-to-noise ratio) is deliberately simple, but its power lies in making the mechanism testable. Figure 1(a) provides empirical evidence that σ_noise is indeed comparable to σ_group in practice, validating the model's central premise. This transforms "LLM judges are noisy" (a vague, widely acknowledged limitation) into "the signal-to-noise ratio of pointwise evaluation degrades as the policy improves, creating a fundamental scaling ceiling" (a precise, falsifiable claim with direct implications for algorithm design).

This is a fundamental contribution rather than an incremental refinement. Prior work on LLM-based reward models (Viswanathan et al., 2025; Huang et al., 2025; Liu et al., 2025) focused on improving judge quality — better rubrics, better calibration, better prompt design. The discriminative collapse analysis argues that these improvements, while valuable, cannot fundamentally solve the problem because the failure is structural: when trajectories converge, any pointwise evaluator (regardless of quality) will face a vanishing signal-to-noise ratio because the true quality differences among trajectories become genuinely smaller than the evaluator's irreducible noise floor. The solution is not a better judge but a different evaluation paradigm.

Innovation 2: The Paradigm Shift from Pointwise Scoring to Intra-Group Relative Ranking as the Solution Strategy

The paper's second key conceptual contribution is the architectural insight that the solution to discriminative collapse is to replace pointwise scoring with intra-group relative ranking, and the demonstration that the mathematics of how advantages are computed in algorithms like GRPO already contains the seed of this solution — the normalization step $A_i = (\hat{R}_i - \mu)/\sigma$ is trying to create relative signals, but it does so through a mechanism (variance normalization) that catastrophically amplifies noise precisely when relative signals are most needed.

This insight operates at the level of problem framing, not algorithm design. The paper's central claim is that for open-ended tasks, the reward signal generation mechanism should be comparative by construction, not comparative by post-processing. In GRPO, the comparison is an afterthought: the judge produces absolute scores, and then the algorithm tries to extract relative information through normalization. When the absolute scores lose discriminative power, the normalization fails. ArenaRL's alternative — producing relative rankings directly through pairwise comparisons — ensures that the information content of the reward signal does not depend on the absolute score scale at all. Even if all trajectories are "very good" in absolute terms, pairwise comparisons can still determine which one is better, because the comparison does not require calibrating an absolute scale.

This is a conceptual shift with deep connections to decision theory and psychometrics. The paper explicitly cites Fürnkranz and Hüllermeier (2010), the foundational text on preference learning, which established that pairwise preference judgments are more stable and reliable than pointwise quantitative assessments. This principle has been widely applied in preference-based alignment (RLHF, DPO, Rafailov et al., 2023), but its application has been limited to offline settings where a fixed dataset of pairwise preferences is used to train a reward model or directly optimize a policy. The paper's contribution is recognizing that the same principle can and should be applied online — at each training step, generate a group of trajectories and construct pairwise comparisons among them to produce a stable relative ranking, using that ranking as the reward signal for the next policy update.

The novelty here is not the use of pairwise comparisons per se — prior work like Pref-GRPO (Wang et al., 2025b) and Writing-Zero (Jia et al., 2025) also used comparison-based signals. The novelty is in how the comparisons are organized and why the specific organization matters. Writing-Zero compares each trajectory against a single random reference, producing a binary signal that the paper argues is too coarse-grained to distinguish degrees of quality (Table 2 implicitly supports this: anchor-based ranking, which is essentially Writing-Zero with a deterministic rather than random anchor, achieves only 27.8% average win rate). Pref-GRPO uses exhaustive pairwise comparison to compute win rates, which provides rich ranking information but at prohibitive computational cost. ArenaRL's insight is that a structured tournament — specifically, a seeded single-elimination bracket — can achieve near-equivalent ranking accuracy to exhaustive comparison at a fraction of the cost, by ensuring that the most informative comparisons (between trajectories of similar quality) happen in the right order.

This framing also recasts the entire open-ended RL problem. Prior work implicitly assumed that the challenge was building a better evaluator — a judge that can produce more accurate absolute scores. ArenaRL reframes the challenge as building a better evaluation protocol — a procedure for organizing comparisons that extracts maximal information from whatever judge is available. This is a more productive framing because it suggests that progress can come from improving the tournament topology (a systems/algorithm problem) rather than from improving the judge's absolute calibration (an AI capability problem). The paper's empirical finding that Seeded Single-Elimination approaches Round-Robin accuracy (32.5% vs. 32.9%, Table 2) validates this reframing: even with a fixed, imperfect judge, the choice of comparison protocol substantially affects the quality of the resulting ranking.

The significance of this contribution extends beyond the specific tournament topologies tested. It establishes that the organization of comparisons is itself a design parameter in RL reward signal generation — a dimension that had not been systematically explored in prior work. Future work can investigate more sophisticated tournament structures (e.g., adaptive pairings based on real-time score estimates, multi-stage tournaments with different seeding strategies, or tournament designs optimized for specific noise characteristics of the judge) without changing the fundamental paradigm.

Innovation 3: The Seeded Single-Elimination Topology as a Computationally Optimal Balance Point

While the paradigm shift from pointwise to tournament-based ranking is the high-level conceptual contribution, the paper's most empirically validated innovation is the design and validation of the seeded single-elimination topology as the specific tournament structure that achieves near-optimal ranking accuracy at linear computational cost. This is an engineering contribution with theoretical implications: it demonstrates that, under the noise characteristics of LLM-based pairwise evaluation, a carefully seeded elimination tournament provides ranking information that is almost as complete as a round-robin, despite using only half as many comparisons (for N=8).

The paper's systematic comparison of five topologies (Table 2) is the strongest evidence for this claim. The progression from Round-Robin (gold standard, 32.9%, O(N²)) through Anchor-Based (27.8%, O(N)) to Seeded Single-Elimination (32.5%, O(N)) tells a clear story: the seeding phase provides a coarse but globally-informed initial ordering, and the elimination phase refines this ordering through targeted head-to-head comparisons. The gap between Anchor-Based and Seeded Single-Elimination (4.7 percentage points) represents the value of the elimination phase — the ability to distinguish among trajectories that all perform similarly against the anchor. The tiny gap between Seeded Single-Elimination and Round-Robin (0.4 percentage points) represents the minimal value of the additional comparisons that Round-Robin performs.

The theoretical insight embedded in this empirical result is that not all pairwise comparisons are equally informative. In a group of N trajectories where quality follows a roughly unimodal distribution with most trajectories clustered near the mean, comparisons between trajectories with large quality differences are highly informative (they reliably determine which is better) but comparisons between trajectories with small quality differences are noisy and add little information. The Seeded Single-Elimination topology systematically prioritizes the most informative comparisons: the seeding phase compares every trajectory against a stable anchor to establish a rough ordering, and the elimination phase pairs trajectories with appropriately large quality gaps (highest seed vs. lowest seed) in early rounds, reserving the fine-grained comparisons between similarly-ranked trajectories for later rounds when fewer comparisons remain to be made.

This is not simply "tournaments are efficient." It is a specific claim about the information geometry of pairwise evaluation under LLM judge noise: the value of a comparison is proportional to the expected quality difference between the compared trajectories (larger gaps = more reliable outcomes = more information per comparison), and the Seeded Single-Elimination topology maximizes the sum of expected quality differences across all comparisons subject to the O(N) budget constraint. The paper does not formalize this claim mathematically, but it is implicit in the pairing scheme (seed 1 vs. seed N, seed 2 vs. seed N-1, etc.) and the seeding mechanism (anchor-based pre-ranking prevents strong trajectories from meeting each other early).

The negative results on alternative topologies are equally informative. Double-Elimination (30.2%, also O(N)) should, in theory, be more robust than Single-Elimination because it gives trajectories a second chance. But without anchor-based seeding (the paper used random seeding for Double-Elimination to maintain comparable computational budget), the initial random pairings cause premature elimination of strong trajectories that the losers' bracket cannot fully recover from. This demonstrates that seeding quality matters more than bracket robustness — a well-seeded single-elimination tournament outperforms a poorly-seeded double-elimination tournament despite having a theoretically weaker structure. This is a non-obvious finding with practical implications: if you can afford one improvement to a tournament-based RL system, invest in better seeding, not in a more complex bracket.

Similarly, Swiss-System (28.3%, O(N log N)) fails because the fixed number of rounds (K ≈ log₂N) provides insufficient comparison depth — each trajectory plays only log₂N matches regardless of its quality, and the pairing algorithm (pairing trajectories with identical records) may not create enough high-quality matchups to produce a reliable ordering. This result suggests that elimination-based topologies are better suited to the noise regime of LLM judges than round-based topologies, because elimination concentrates comparisons on the most promising trajectories (the ones that keep winning) rather than spreading comparisons evenly across all trajectories regardless of quality.

The practical significance of this innovation is that it makes tournament-based RL computationally feasible for online training. At N=8 with K=8 groups per training step and hundreds of training steps, the difference between O(N²) (28 comparisons per group) and O(N) (14 comparisons per group) is the difference between infeasible and practical. At the larger group sizes the paper explores (N=16, used for Open-Travel and writing tasks), round-robin would require 120 comparisons per group versus 30 for seeded single-elimination — a 4× reduction that is essential for training with long-context trajectories (deep research trajectories can span tens of thousands of tokens).

Innovation 4: Process-Aware Pairwise Evaluation as an Optimization Signal for Agent Capabilities

A subtler but important conceptual contribution is the design of the process-aware pairwise evaluation mechanism — specifically, the decision to include chain-of-thought reasoning and tool invocation quality in the comparison rubrics, rather than evaluating only the final output. This transforms the tournament from a mere answer-scoring mechanism into a capability-reinforcement mechanism that incentivizes the policy to develop robust reasoning and tool-use behaviors, not just to produce answers that look good to a judge.

Prior work on comparison-based evaluation for LLMs (RLHF, DPO, and even Writing-Zero and Pref-GRPO) typically compares outputs — which final answer is better, or which generated text is more coherent. This is appropriate for single-turn generation tasks where the path from prompt to output is a black box. But for agent tasks, where the trajectory includes explicit reasoning steps and tool interactions, evaluating only the final answer discards information that is crucial for policy improvement. A correct final answer produced through flawed reasoning (e.g., accidentally choosing the right destination after misreading a tool output) represents a brittle capability that will not generalize; a slightly suboptimal answer produced through sound reasoning (e.g., correctly identifying constraints but making a minor arithmetic error) represents a capability that could be refined.

The process-aware rubric addresses this by instructing the judge to evaluate three dimensions: logical consistency of the chain-of-thought, precision of tool calls, and reliability of the final answer. This is not simply "more detailed evaluation" — it is a specific hypothesis about what the optimization signal should reinforce. The paper argues that this "ensures that the optimization signal reinforces the agent's intrinsic reasoning capabilities rather than merely overfitting to surface-level features of the final answer" (Section 3.3). This is a form of credit assignment at the evaluation level: the tournament does not just say "trajectory A is better than trajectory B" — the multi-dimensional rubric ensures that the reason A is judged better reflects genuine improvements in reasoning and tool use, not spurious surface features.

The bidirectional scoring protocol (evaluating each pair twice with swapped positions) is a standard technique in LLM evaluation, but its inclusion as a mandatory component of the Arena judge design reflects an important practical insight: in a tournament setting, positional bias is not just a minor annoyance — it can systematically distort rankings by advantaging whichever trajectory happens to appear first (or second) in the bracket. Since tournament pairings are determined by seeding rather than randomization, a positional bias could create a systematic advantage for trajectories that happen to fall on one side of the bracket. The bidirectional protocol eliminates this source of ranking distortion entirely, at the cost of doubling the number of judge calls per comparison. This is a principled trade-off: computational cost is increased by a factor of 2 to eliminate a systematic bias that could otherwise dominate the ranking signal.

The empirical evidence for the process-aware design is primarily indirect — the paper does not ablate the rubric structure (e.g., comparing process-aware vs. outcome-only evaluation in the tournament). However, the case study in Appendix F provides qualitative evidence: the SFT model's trajectory exhibits "a restatement tendency in its chain-of-thought" and "fails to align with the user's intent," while the ArenaRL-optimized model "proactively retrieves information about multiple target attractions, performs logically coherent route planning, and ultimately produces a persuasive, personalized itinerary." This improvement in reasoning patterns is precisely what a process-aware evaluation signal is designed to incentivize — the policy learns to produce not just better answers, but better reasoning.

This contribution is incremental rather than fundamental — process-aware evaluation is a design choice within the tournament framework, not a new paradigm — but it addresses a critical gap in prior comparison-based RL methods. Writing-Zero's binary comparisons and Pref-GRPO's win-rate-based rewards are based on final output quality alone, meaning they cannot distinguish between a well-reasoned trajectory and one that arrived at a good answer by chance. The process-aware rubric closes this gap, making tournament-based RL suitable for agent tasks where the reasoning process itself is a target of optimization.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper introduces two new benchmarks and evaluates on three existing ones. Open-Travel consists of 250 manually verified test queries across five subtasks (Direction, Search, Compare, 1-Day, M-Day) for Chinese travel planning, with 2,600 SFT and 1,626 RL training samples. Open-DeepResearch consists of 100 manually verified test queries for Chinese/English deep research tasks, with 2,662 SFT and 2,216 RL training samples. The three open-ended writing benchmarks are WritingBench (six professional/creative domains with multi-dimensional constraint evaluation), HelloBench (QA, summarization, and heuristic text generation subsets), and LongBench-write (ultra-long text generation). Full dataset statistics appear in Table 1.

• Base model(s). All experiments use Qwen3-8B-Base (Yang et al., 2025) as the backbone model, a publicly available 8-billion-parameter LLM. The paper argues this model sits in a useful capability regime for studying RL on open-ended tasks: not so powerful that the SFT baseline already solves the tasks, but capable enough that RL can produce meaningful improvements. For SFT cold-start, the base model is fine-tuned on domain-specific SFT datasets to acquire tool-use and planning capabilities.

• Metrics. The primary metrics vary by benchmark. For Open-Travel, the metric is win rate (%) — the proportion of non-tied cases where the candidate model's output is preferred over a baseline trajectory in a pairwise comparison, averaged across two independent LLM judges (Qwen3-Max and Claude-4-Sonnet) and across evaluation rubrics. For Open-DeepResearch, the metrics are valid generation rate (Val. %) — the proportion of test cases where the model successfully generates a valid final answer without context overflow — and win rate (%) conditioned on valid generations across seven evaluation rubrics (Framework, Tool Usage, Coverage, Relevance, Accuracy, Depth, Clarity). For open-ended writing, the metric is LLM-judge quality score on each benchmark's native scoring scale. The paper validates judge reliability through a human-LLM consistency analysis (Figure 4b), reporting a 73.9% overall agreement rate between LLM and human evaluations.

• Baselines. The paper compares against two categories. Closed-source models: GPT-4o (Achiam et al., 2023), Grok-4 (xAI, 2025), Gemini-2.5-pro (Team et al., 2023), and Claude-3.7-Sonnet (Anthropic, 2023). These serve as strong capability upper bounds. RL algorithms: GRPO (Shao et al., 2024) and GSPO (Zheng et al., 2025), both using standard LLM-as-Judge pointwise scoring with the exact same judge models and evaluation rubrics as ArenaRL, ensuring that performance differences are attributable to the reward signal mechanism, not the judge quality. SFT (supervised fine-tuning only, no RL) is also reported as the initialization point for all RL methods.

• Generation budget / compute accounting. The primary unit of compute is number of trajectories generated per training step, determined by the group size $N$ and number of groups $K$. For Open-Travel and writing tasks, $N=16$ and $K=8$ (128 trajectories per step); for Open-DeepResearch, $N=8$ and $K=4$ (32 trajectories per step, reduced due to longer trajectories). Tournament comparison cost is measured in number of Arena Judge calls (pairwise evaluations), with round-robin requiring $N(N-1)/2$ calls and seeded single-elimination requiring $2N-2$ calls per group. The paper compares tournament topologies under a unified configuration of $N=8, K=8$ (Table 2) and ablates group size $N \in \{2, 4, 8, 16\}$ (Figure 4a). For RL algorithm baselines, the judge evaluation cost is one pointwise scoring call per trajectory (approximately equal to a single pairwise evaluation call in the ArenaRL setting), making the per-trajectory judge cost roughly comparable across methods.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The tournament topology comparison (Table 2) appears to be evaluated directly on the Open-Travel test set, which raises a concern about topology selection being tuned to test-set performance. However, the topology selection is also justified by computational efficiency arguments independent of the exact win rates. The human-LLM consistency analysis (Figure 4b) uses a confusion matrix approach with an overall agreement rate, but no inter-annotator agreement metrics (e.g., Cohen's kappa) or confidence intervals are reported.

Main Quantitative Results

Tournament Topology Analysis

Table 2 presents a systematic comparison of five tournament topologies under a unified RL configuration ($N=8$, $K=8$) evaluated on the Open-Travel benchmark. The key finding is that Seeded Single-Elimination achieves 32.5% average win rate with $\mathcal{O}(N)$ pairwise comparisons, approaching the Round-Robin gold standard of 32.9% with $\mathcal{O}(N^2)$ comparisons, while substantially outperforming all other linear-complexity alternatives.

The results, in descending order of average win rate:

• Round-Robin (gold standard): 32.9% mean, with top subtask performance on Direction (23.3%) and Search (66.3%). Requires $N(N-1)/2 = 28$ comparisons per group.
• Seeded Single-Elimination (chosen topology): 32.5% mean, with the highest score on Search (69.9%) and competitive or better performance on Compare (22.9% vs. 23.6%), 1-Day (34.9% vs. 32.1%), and M-Day (18.1% vs. 19.0%). Requires $2N-2 = 14$ comparisons per group — exactly half of Round-Robin. Notably outperforms Round-Robin on Search (69.9% vs. 66.3%) and 1-Day (34.9% vs. 32.1%), which the paper attributes to the seeding mechanism filtering out noise that contaminates Round-Robin win-rate estimates.
• Double-Elimination: 30.2% mean, strong on Search (52.4%) and 1-Day (39.9%), but weak on Direction (12.6%) and M-Day (12.3%). Requires approximately $2N-2$ comparisons (comparable to Seeded Single-Elimination). The paper's explanation: random seeding causes premature elimination of strong trajectories, and the losers' bracket only partially recovers.
• Swiss-System: 28.3% mean, competitive on Direction (20.9%) and Search (43.0%), but weak on M-Day (11.1%). Requires $\mathcal{O}(N \log N)$ comparisons (approximately 12 for $N=8$). Fails due to insufficient comparison depth per trajectory.
• Anchor-Based Ranking: 27.8% mean. Requires only $N-1 = 7$ comparisons per group — the cheapest topology — but lacks resolution to distinguish similar trajectories, particularly visible on Search (41.3%) and 1-Day (31.1%). Notably outperforms SFT (16.4%) across all subtasks, confirming that even the simplest comparison-based approach provides useful optimization signals.

Takeaway: the 4.7 percentage point gap between Anchor-Based and Seeded Single-Elimination represents the value of the elimination phase — the ability to resolve fine-grained differences among trajectories that all score similarly against the anchor. The 0.4 percentage point gap between Seeded Single-Elimination and Round-Robin represents the minimal additional value of the $N(N-1)/2 - (2N-2) = 14$ extra comparisons that Round-Robin performs. For $N=8$, this is a 2:1 efficiency ratio with negligible accuracy loss.

It is notable that the SFT baseline (16.4%) already performs respectably on Search (29.7%) and 1-Day (20.4%), but poorly on Direction (10.6%) and M-Day (7.1%). The M-Day subtask is held out from SFT training, making it a generalization test — the fact that all tournament topologies improve M-Day performance suggests that RL with ranking-based rewards transfers to unseen task structures.

Main Benchmark Results: Open-Travel and Open-DeepResearch

Table 3 presents the headline results on the paper's two constructed benchmarks. ArenaRL is compared against four closed-source models, SFT, GRPO, and GSPO. The Seeded Single-Elimination topology ($N=16, K=8$ for Open-Travel; $N=8, K=4$ for Open-DeepResearch) is used for all ArenaRL results.

Open-Travel. ArenaRL achieves 41.8% average win rate across the five subtasks, substantially outperforming:

• SFT: 16.4% (a 25.4 percentage point improvement, roughly 2.5×)
• GRPO: 16.4% (identical to SFT — zero improvement from RL)
• GSPO: 17.2% (a 0.8 point improvement over SFT, minimal)
• All closed-source models: Claude-3.7-Sonnet achieves 31.6%, GPT-4o achieves 2.6%, Grok-4 achieves 16.8%, Gemini-2.5-pro achieves 10.6%

The per-subtask breakdown reveals that ArenaRL's advantage is particularly pronounced on Search (66.1% vs. 59.6% for the best closed-source model, Claude-3.7-Sonnet) and 1-Day (58.0% vs. 43.6% for Claude-3.7-Sonnet). On Direction, ArenaRL achieves 32.1% vs. 18.6% for Claude-3.7-Sonnet, nearly doubling the best closed-source performance. On the most challenging generalization subtask (M-Day, excluded from SFT), ArenaRL achieves 21.0% vs. 21.3% for Claude-3.7-Sonnet, essentially matching the strongest closed-source model despite being trained on an 8B-parameter open model.

The most striking finding is that GRPO and GSPO provide essentially no improvement over SFT on Open-Travel. GRPO's 16.4% average is identical to SFT; GSPO's 17.2% is marginally better. This is strong evidence for the discriminative collapse hypothesis: standard pointwise RL does not work on this benchmark despite using the same judge models and rubrics as ArenaRL. The failure is not due to poor judge quality — it is due to the pointwise scoring paradigm itself.

Open-DeepResearch. ArenaRL achieves 64.3% average win rate with 99.0% valid generation rate. The comparison is even more dramatic than Open-Travel:

• SFT: 16.7% mean win rate, but only 32.0% valid generation rate
• GRPO: 25.2% mean win rate (an improvement over SFT in win rate), but only 17.0% valid generation rate (a substantial degradation from SFT)
• GSPO: 25.2% mean win rate (identical to GRPO), but only 21.0% valid generation rate (also degraded)
• Best closed-source: Grok-4 at 34.8% mean win rate with 83.0% valid generation; Gemini-2.5-pro at 28.3% with 92.0% valid generation

The valid generation rate metric is particularly revealing. The SFT model already struggles to complete deep research tasks (only 32% of queries produce valid outputs, likely due to context overflow from long trajectories). GRPO and GSPO make this worse: their valid generation rates drop to 17% and 21% respectively, suggesting that the noisy pointwise reward optimization actively damages the policy's ability to maintain coherent long-horizon tool use. ArenaRL's 99% valid generation rate represents a near-complete recovery of task completion ability alongside a 3.85× improvement in win rate over SFT (64.3% vs. 16.7%).

Per-rubric performance on Open-DeepResearch (Table 3) shows ArenaRL is strong across all seven dimensions, with particularly high scores on Tool Usage (77.3%), Coverage (78.8%), and Clarity (61.6%). These align with the process-aware rubric's emphasis on tool invocation quality and structured output. The strongest closed-source model (Grok-4, 34.8% mean) shows a different strength profile, with higher Depth (36.1%) and Accuracy (39.2%) but weaker Clarity (17.5%), suggesting different capability emphases.

Main Benchmark Results: Open-Ended Writing

Table 4 extends the evaluation to standard open-ended writing tasks, demonstrating that ArenaRL's tournament-based ranking generalizes beyond tool-augmented agent tasks to general open-ended generation. ArenaRL achieves 80.30% average score across three benchmarks, compared to:

• SFT: 72.17%
• GRPO: 73.60%
• GSPO: 73.03%
• GPT-4o: 76.05%
• Claude-3.7-Sonnet: 76.65%
• Grok-4: 84.75%
• Gemini-2.5-pro: 85.48%

The gap between ArenaRL and the pointwise RL baselines is substantial (+6.70% over GRPO, +7.27% over GSPO), confirming that the tournament-based ranking mechanism provides benefits even on tasks without tool use. However, unlike Open-Travel and Open-DeepResearch where ArenaRL substantially outperforms most closed-source models, on writing tasks it still trails Grok-4 (84.75%) and Gemini-2.5-pro (85.48%), suggesting that for pure language generation quality, the gap between an 8B open model and frontier closed-source models is harder to close than for structured agent tasks where reasoning and tool-use strategy can compensate for raw model scale.

Domain-specific patterns are noteworthy. On WritingBench domains C (Politics & Law, 77.58%) and E (Education, 79.35%), ArenaRL closes much of the gap with frontier models. On HelloBench-Heuristic (creative reasoning, 93.78%), ArenaRL is competitive with Grok-4 (94.65%). However, on HelloBench-QA (74.82%), ArenaRL slightly trails GSPO (74.39%? — Table 4 shows GSPO at 63.97% for HelloBench-Heuristic and 81.75% for HelloBench-QA; the paper notes this subtask is "largely constrained by the model's inherent knowledge, and ArenaRL remains highly competitive under this limitation"), suggesting that tournament-based ranking cannot overcome fundamental knowledge limitations in the base model.

Scalability Analysis: Group Size Ablation

Figure 4(a) presents the ablation of group size $N \in \{2, 4, 8, 16\}$ on Open-Travel, conducted under the Seeded Single-Elimination topology:

• $N=2$: 20.8% average win rate (already above SFT's 16.4%, confirming that even minimal pairwise comparison provides useful gradients)
• $N=4$: approximately 28% (estimated from Figure 4a — the paper does not provide exact numbers for this intermediate point)
• $N=8$: 32.5% (the value from Table 2)
• $N=16$: 41.8% (the main result from Table 3)

The relationship between $N$ and performance is monotonically increasing, with the largest jump occurring from $N=8$ to $N=16$ (from 32.5% to 41.8%, a 9.3 point gain). The paper attributes this to the broader exploration space: "enlarging the candidate pool effectively broadens the exploration space, dramatically increasing the likelihood of discovering high-quality trajectories and thereby enabling the model to learn from stronger examples." The per-subtask breakdown (visible in Figure 4a) shows the 1-Day subtask benefits most dramatically from increased $N$ (34.9% at $N=8$ to 58.0% at $N=16$), while Direction shows more modest gains.

This scaling behavior has practical implications: for complex tasks with large solution spaces, increasing the group size (and thus the exploration diversity) yields substantial returns, but at the cost of linearly more pairwise comparisons (from 14 at $N=8$ to 30 at $N=16$ under Seeded Single-Elimination). The paper does not explore $N > 16$, so the saturation point of this scaling is unknown.

Direct RL Training without Cold Start

Figure 4(c) demonstrates ArenaRL's robustness to cold-start quality by training Qwen3-8B-Base directly (without SFT) on the Open-Travel Search subtask. The key finding: from a starting score of 0 (the generic model is incapable of handling the task), ArenaRL drives sustained improvement to a peak score of 71% at step 160. The improvement is not immediate — the first ~20 steps show minimal progress, suggesting the policy needs time to discover basic tool-use patterns — but accelerates after step 40 and shows a generally upward trajectory to step 160.

This result addresses a practical concern about RL for open-ended agents: if SFT data is unavailable or expensive to collect, can RL bootstrap from a generic model? The answer is a qualified yes — ArenaRL's intra-group relative ranking mechanism can provide useful gradient signals even when initial output quality is extremely low, because pairwise comparisons can still distinguish between "bad but slightly better" and "bad but slightly worse" trajectories. However, the cold-start SFT approach (Table 3) achieves competitive performance with presumably fewer RL steps (the paper does not report the number of steps for the main SFT→RL results), so cold-start remains practically beneficial for sample efficiency.

The paper frames this as "effectively mitigating RL's cold-start problem" and "demonstrating ArenaRL's capacity for self-evolution from scratch in scenarios lacking costly SFT annotated data." This is a strong claim, but the evidence is limited to one subtask (Search) on one benchmark (Open-Travel). Whether direct RL from a base model works on more complex subtasks (Direction, M-Day) or on Open-DeepResearch is not tested.

Human-LLM Consistency Analysis

Figure 4(b) presents a confusion matrix comparing LLM judge evaluations against human evaluations on the Open-Travel and Open-DeepResearch benchmarks, with an overall agreement rate of 73.9%. Most evaluation outcomes are concentrated along the diagonal, indicating that when the LLM judge declares a trajectory better, equal, or worse than the baseline, human evaluators agree in the majority of cases.

The paper uses this result to argue that "ArenaRL's performance gains do not simply stem from overfitting to the preferences of the specific judge model used during the RL phase, but instead reflect improvements that are broadly aligned with human assessments." This is a critical validity check: if the LLM judge had poor agreement with humans, the tournament-based optimization might be optimizing for judge-specific quirks rather than genuine quality.

However, the 73.9% agreement rate, while "relatively high" as the paper describes it, still implies that in approximately 1 in 4 cases, the LLM judge disagrees with human evaluators. This level of judge error could affect tournament accuracy, particularly in close matches where the true quality difference is small relative to the judge noise. The paper does not break down agreement by difficulty level, subtask, or quality tier, which would provide more insight into when the judge is reliable and when it is not.

Real-World Business Application Results

Section 6.5 reports results on real business data from the Amap (Gaode Map) ecosystem, organized into two categories:

• Deterministic POI Search: ArenaRL-tuned model achieves 75% to 83% gain in search accuracy over the baseline. The paper does not specify the baseline (presumably the pre-RL model variant) or the absolute accuracy numbers, making the magnitude of improvement difficult to contextualize. "75% to 83% gain" is ambiguous — it could mean a relative improvement (e.g., from 40% to 70–73% accuracy) or an absolute percentage point increase (e.g., from 0% to 75–83%). Given that the paper describes this as "quantifiable POI search tasks... characterized by explicit evaluation metrics," absolute accuracy would be the expected metric, but the numbers are not clearly reported.

• Open-ended Planning Tasks: On complex tasks requiring multi-step reasoning, tool invocation, and constraint satisfaction (e.g., vague intent queries like "find a quiet bar near the Bund with a river-view terrace for a date, open after 10 PM" and cross-city logistics with cost/transfer tradeoffs), the "core business metric" rises from 69% to 80%. Again, the metric is not precisely defined, and the baseline is unspecified (likely the same pre-RL model as in the POI search case).

These results are reported in a narrative format without error bars, statistical tests, or detailed per-task breakdowns, making them suggestive rather than definitive. However, they serve an important purpose: demonstrating that ArenaRL's training methodology transfers from curated benchmarks to real production data, where queries are messier, constraints are more varied, and the evaluation is tied to actual business metrics rather than LLM-judge win rates. The paper states that "substantial and consistent gains throughout the training process" were observed, suggesting the improvements are not due to cherry-picked evaluation points.

Ablation Studies and Robustness Checks

Tournament topology comparison under unified settings: Table 2 serves as the primary ablation of tournament structure, with five topologies compared at $N=8, K=8$. The key finding — that Seeded Single-Elimination achieves near-Round-Robin accuracy at half the computational cost — is the central empirical justification for the chosen topology. The negative results on Double-Elimination (30.2%, random seeding) and Swiss-System (28.3%, insufficient depth) demonstrate that more complex bracket structures do not automatically yield better rankings. The paper explicitly attributes Double-Elimination's failure to the absence of anchor-based seeding, making an implicit causal claim: seeding quality matters more than bracket robustness. An ablation testing Double-Elimination with anchor-based seeding (which would require approximately $3N$ comparisons) is not performed, leaving open the question of whether a more robust bracket with good seeding could outperform single-elimination. This is a genuine limitation — the paper's claim that "without high-quality initial seeds, its ranking fidelity falls short" is supported for the random-seeding case but does not rule out the possibility that Double-Elimination with seeding could be superior.

Group size scaling ($N \in \{2, 4, 8, 16\}$): Figure 4(a) shows monotonic improvement with increasing $N$, demonstrating that the tournament mechanism benefits from larger candidate pools. The absence of experiments with $N > 16$ prevents identification of a saturation point. Given the linear scaling of Seeded Single-Elimination comparisons ($2N-2$), larger groups would be computationally feasible (e.g., $N=32$ requires 62 comparisons per group, still manageable for offline batch processing but potentially expensive for online training). The paper does not discuss whether memory constraints (fitting 32 long-context trajectories in GPU memory) or training stability concerns prevent larger group sizes. The jump from $N=8$ to $N=16$ produces a 9.3 percentage point gain in average win rate, suggesting the scaling curve has not flattened, and $N=32$ could yield further improvements.

Direct RL without cold start: Figure 4(c) demonstrates that ArenaRL can train from a base model with zero initial task capability, achieving 71% on the Search subtask after 160 steps. This serves as a robustness check on the cold-start dependency — the main results use SFT initialization, which could mask the RL method's ability to discover capabilities from scratch. The result also implicitly ablates the necessity of SFT data, though the paper does not compare sample efficiency (how many RL steps are needed to match SFT→RL performance). A direct comparison of cold-start RL vs. SFT→RL at equivalent total compute would strengthen the claim that "ArenaRL mitigates RL's cold-start problem" — the current evidence shows it's possible, not that it's efficient relative to SFT initialization.

Process-aware vs. outcome-only evaluation: Not directly ablated. All experiments use the process-aware rubric structure. Without an ablation comparing process-aware pairwise evaluation to outcome-only pairwise evaluation (comparing only final answers, not chain-of-thought or tool calls), the contribution of process awareness to the observed gains cannot be isolated. The paper's qualitative case study (Appendix F) provides suggestive evidence — the ArenaRL model's trajectory shows improved reasoning patterns — but does not quantify how much of the improvement is attributable to evaluating the process versus a hypothetical outcome-only tournament that might achieve similar gains. This is a significant missing ablation because it would clarify whether the "process-aware" design choice is load-bearing or incidental.

Bidirectional vs. single-pass evaluation: Not ablated. All pairwise comparisons use bidirectional scoring (evaluating each pair twice with swapped positions). The paper argues this eliminates positional bias, but the magnitude of positional bias in the specific judge model (Qwen3-Max) is not quantified. An ablation comparing bidirectional to single-pass evaluation would reveal how much positional bias affects tournament rankings and whether the 2× computational cost of bidirectionality is justified. Given that bidirectional evaluation doubles the number of judge calls per comparison (from 1 to 2), this is a practically significant cost factor.

Arena Judge model choice: The paper uses Qwen3-Max as the arena judge during RL training and Qwen3-Max + Claude-4-Sonnet as dual judges for final evaluation. The choice of judge model is not ablated — the paper does not compare ArenaRL trained with different judge models (e.g., smaller open-source judges like Qwen3-8B itself, or different closed-source judges). This matters because the judge model's quality directly affects tournament accuracy, and the paper's cost analysis assumes access to a powerful closed-source judge (Qwen3-Max) for online training. Whether a smaller, cheaper judge could achieve comparable tournament accuracy is unknown. The human-LLM consistency analysis (Figure 4b, 73.9% agreement) provides some calibration on judge quality but does not address how sensitive ArenaRL's performance is to judge quality degradation.

KL penalty coefficient $\beta$ and clipping parameter $\epsilon$: Not ablated or explicitly reported. The policy optimization objective (Equation 8) includes these hyperparameters, but the paper does not state their values or show sensitivity to them. Given that the optimization objective is standard (matching GRPO/GSPO), these parameters are likely set to conventional defaults, but their impact on training stability and final performance is unexplored. For long-horizon agent tasks, KL regularization may interact differently with tournament-based rewards than with pointwise rewards, making hyperparameter sensitivity a relevant concern.

SFT data scale and quality: Not ablated. The paper constructs SFT datasets using "high-performing closed-source models as the base model to generate large-scale tool-use trajectories" (Section 5, Stage II), but does not specify which models, how trajectories were filtered, or whether SFT data quality was systematically varied. Since all RL methods (ArenaRL, GRPO, GSPO) train from the same SFT initialization, differences in SFT quality affect absolute performance but not relative comparisons. However, the interaction between SFT quality and RL effectiveness is unexplored — lower-quality SFT data might produce a weaker initialization from which RL gains are larger (or smaller, if the policy is trapped in a poor local optimum).

Negative results on Open-DeepResearch valid generation rate: GRPO and GSPO both degrade valid generation rates relative to SFT (17% and 21% vs. 32%). This is the clearest evidence for the paper's claim that pointwise RL can actively damage policy quality through noise amplification. However, the paper does not analyze the mechanism of degradation — are models producing shorter trajectories (avoiding long contexts to minimize KL penalty), hallucinating tool outputs, or repeating tokens until context overflow? Understanding the failure mode would strengthen the discriminative collapse narrative and inform mitigation strategies beyond the tournament-based solution. The paper attributes the degradation to "spurious advantages such as length bias," but this is a hypothesis, not an empirically validated mechanism.

Critical Assessment

The central claim of the paper is that tournament-based relative ranking solves discriminative collapse and enables robust RL for open-ended agent tasks where pointwise scoring fails. The experimental evidence broadly supports this claim, but with important scope limitations and methodological caveats that the paper partially acknowledges and partially leaves unaddressed.

Claim 1: Pointwise RL (GRPO, GSPO) fails on open-ended agent tasks due to discriminative collapse. The evidence for failure is strong: on Open-Travel, GRPO and GSPO achieve 16.4% and 17.2% average win rates, essentially identical to SFT at 16.4% (Table 3). On Open-DeepResearch, both methods actually degrade valid generation rates (17% and 21% vs. 32% for SFT) despite marginally improving conditioned win rates (25.2% vs. 16.7%). This is consistent with the discriminative collapse hypothesis: the policies improve enough that the pointwise judge's scores lose discriminative power, at which point noise dominates the gradient signal and causes degradation.

However, the evidence that the failure mechanism is specifically discriminative collapse (as opposed to some other RL pathology — reward hacking, inappropriate KL regularization, optimization instability, or simply insufficient training) is circumstantial. The paper's Figure 1(a) shows one illustrative example of score compression and noise, but does not provide a systematic analysis across training steps showing that $\sigma_{\text{group}}$ decreases as $\sigma_{\text{noise}}$ increases, or that the signal-to-noise ratio correlates with policy degradation. The claim that "the effective reward signal becomes dominated by noise from the reward model" is a mechanistic hypothesis that is not directly tested — it is inferred from the symptoms (stagnation, degradation) and supported by one qualitative figure. A training-dynamics analysis plotting $\sigma_{\text{group}}$, $\sigma_{\text{noise}}$, and policy performance over RL steps for both ArenaRL and the baselines would substantially strengthen this claim.

Claim 2: Seeded Single-Elimination achieves near-Round-Robin accuracy at linear cost. The evidence is clear and well-controlled: Table 2 compares five topologies under identical settings ($N=8, K=8$) and shows Seeded Single-Elimination at 32.5% vs. Round-Robin at 32.9%. The 0.4 percentage point gap (within what is likely statistical noise at a 100-sample per-bin test set, though no confidence intervals are reported) supports the efficiency claim. The 2:1 reduction in comparisons (14 vs. 28 at $N=8$) is substantial.

However, the topology comparison is conducted at a single group size ($N=8$) on a single benchmark (Open-Travel). It is not obvious that the relative ranking of topologies would remain the same at $N=16$ (the group size used for main results) or $N=4$. The paper's main results use $N=16$ under the Seeded Single-Elimination topology (Table 3), but the justification for this topology choice comes from the $N=8$ comparison in Table 2. At $N=16$, Round-Robin would require 120 comparisons per group versus 30 for Seeded Single-Elimination — a 4:1 ratio that makes the efficiency argument even stronger, but the accuracy comparison at this scale is not measured.

Additionally, the topology comparison in Table 2 appears to be evaluated on the Open-Travel test set (the same 250 samples used for final evaluation). This means the topology was selected based on test-set performance, creating a potential overfitting concern. The paper argues the selection is principled (balancing accuracy and efficiency), but a held-out validation set for topology selection would have been a cleaner methodology. This is mitigated somewhat by the fact that the topology choice is also justified by computational efficiency arguments independent of exact win rates.

Claim 3: ArenaRL substantially outperforms strong baselines on open-ended agent tasks. The evidence in Table 3 is striking: 41.8% vs. 16.4–17.2% for RL baselines on Open-Travel, and 64.3% vs. 25.2% for RL baselines on Open-DeepResearch (with 99% valid generation vs. 17–21%). ArenaRL also outperforms most closed-source models on these benchmarks, which is unexpected for an 8B-parameter model and suggests that tournament-based RL enables the policy to develop task-specific strategies that compensate for raw model scale.

However, there is a reported-data concern that must be flagged. The GRPO and GSPO results in Table 3 show identical mean win rates on Open-DeepResearch (25.2%), with identical per-rubric scores for several dimensions (Framework: 20.6% vs. 23.8%, Tool Usage: 35.3% vs. 33.3%, Coverage: 35.3% vs. 40.5%, Relevance: 23.5% vs. 16.7%, Accuracy: 23.5% vs. 21.4%, Depth: 26.5% vs. 31.0%, Clarity: 11.8% vs. 9.5%). While these are not identical between the two methods, the mean is identical and many values show suspicious patterns. The paper does not discuss this coincidence, and it could indicate a reporting error or that the two methods converged to nearly identical policies (which would itself be an interesting finding about the dominance of the pointwise reward signal over algorithmic differences). Without clarification, the reliability of the exact numbers is uncertain, though the qualitative finding (both pointwise methods are dramatically worse than ArenaRL) seems robust.

A more fundamental concern is the single model family limitation. All experiments use Qwen3-8B-Base. The paper claims ArenaRL is a general framework, but its effectiveness has not been demonstrated on models from other families (Llama, Mistral, DeepSeek), at other scales (1B, 70B), or with different base capabilities. The discriminative collapse mechanism is argued to be universal, but the tournament's effectiveness depends on the quality of the arena judge (Qwen3-Max), which may interact differently with policies from different model families. A single experiment with, say, Llama-3-8B would substantially improve confidence in the framework's generality.

Claim 4: ArenaRL generalizes to open-ended writing tasks. Table 4 shows ArenaRL achieving 80.30% average score versus 73.60% for GRPO and 73.03% for GSPO, a solid improvement. However, ArenaRL trails Grok-4 (84.75%) and Gemini-2.5-pro (85.48%), suggesting that on pure language generation tasks, the gap between an 8B model with tournament-based RL and frontier closed-source models is substantial — ArenaRL cannot close the scale gap the way it does on structured agent tasks. This is not a weakness of the paper per se, but it bounds the claim: tournament-based ranking helps, but it is not a substitute for model scale on all task types. The paper acknowledges this implicitly by reporting the gap, but does not discuss why the gap persists on writing tasks while being closed on agent tasks.

Missing experiments that would strengthen the paper:

• Process-awareness ablation: Comparing ArenaRL with process-aware rubrics to ArenaRL with outcome-only rubrics (evaluating only final answers in pairwise comparisons) would isolate the contribution of evaluating reasoning/tool-use quality. This is the single most important missing ablation because it would clarify whether the gains come from pairwise ranking per se (vs. pointwise) or from the specific design of the evaluation rubric.

• Bidirectional vs. single-pass ablation: Quantifying the positional bias of Qwen3-Max on these tasks and measuring how much ranking accuracy degrades with single-pass evaluation would justify (or challenge) the 2× cost of bidirectionality.

• Judge model robustness: Training ArenaRL with a weaker judge (e.g., Qwen3-8B itself, or a smaller open-source model) would test whether tournament-based ranking is robust to judge quality degradation. This matters for practical deployment where Qwen3-Max-level API access may not be available.

• Training dynamics analysis: Tracking $\sigma_{\text{group}}$, $\sigma_{\text{noise}}$, and policy performance over RL steps for both ArenaRL and GRPO/GSPO would provide direct evidence for the discriminative collapse mechanism. The single illustrative figure (Figure 1a) is suggestive but not systematic.

• Large-scale topology validation: Repeating the Table 2 comparison at $N=16$ on a held-out validation set would ensure the topology choice generalizes and is not overfit to the $N=8$ setting on the test set.

• Multiple model families: At minimum, one experiment replicating the Open-Travel results with a non-Qwen base model (e.g., Llama-3-8B) would strengthen the generality claim.

• Confidence intervals or statistical tests: None of the tables or figures include error bars, confidence intervals, or significance tests. With 250 test samples for Open-Travel split across five subtasks (50 per subtask), win rates have substantial sampling variance. The difference between Seeded Single-Elimination (32.5%) and Round-Robin (32.9%) is well within what could be sampling noise. Reporting confidence intervals (e.g., via bootstrap) would clarify which differences are statistically reliable.

Where the claims hold conditionally:

• The claim that tournament-based ranking solves discriminative collapse holds for the specific tasks, model, judge, and hyperparameters tested. Generalization to substantially different settings (different judge quality, different model scale, different task types) is plausible but not demonstrated.

• The claim that Seeded Single-Elimination achieves near-Round-Robin accuracy holds for $N=8$ on Open-Travel. Generalization to other group sizes and tasks is assumed but not validated.

• The claim that ArenaRL enables RL from scratch without cold-start is demonstrated for one subtask (Search), but whether it works for more complex subtasks or at the multi-task scale of the full Open-Travel benchmark is untested.

• The real-world business results (Section 6.5) are reported with insufficient detail to assess their reliability independently, though they serve a useful role in demonstrating deployment relevance.

In summary, the experimental evidence makes a compelling case that tournament-based relative ranking provides substantially better RL training signals than pointwise scoring for open-ended agent tasks, but the evidence for the specific mechanism (discriminative collapse as the cause of pointwise failure) is more suggestive than definitive, and several important ablations (process-awareness, bidirectionality, judge robustness) were not performed. The paper's contributions — the discriminative collapse concept, the tournament-based ranking paradigm, and the seeded single-elimination topology — are well-supported by the data that is presented, but the scope of the empirical validation is narrower than the paper's framing suggests, being limited to a single model family, a single judge model, and a specific set of tasks with a particular structure (tool-use planning and research).

6. Limitations and Trade-offs

Single Model Family, Single Judge Model

The assumption or constraint. All experiments use Qwen3-8B-Base as the backbone model and Qwen3-Max as the arena judge during RL training. The paper states in Section 6.1 that Qwen3-8B-Base is used as "the backbone model" with SFT cold-start on domain-specific data, and Appendix A confirms that "Qwen3-Max [is used] as the arena judge during training." The paper motivates this choice by arguing Qwen3-8B-Base is "representative," but provides no evidence that the findings transfer to other model families (Llama, Mistral, DeepSeek) or other judge models.

The consequence. Three failure modes become uncertain without cross-model validation. First, discriminative collapse may be more or less severe depending on the base model's output diversity — a model that produces more varied trajectories would maintain larger σ_group for longer, potentially reducing the advantage of tournament-based ranking over pointwise scoring. Second, the effectiveness of the arena judge depends on its reasoning quality relative to the policy being optimized; a weaker judge (e.g., a smaller open-source model) might produce noisier pairwise comparisons that degrade tournament ranking accuracy, potentially to the point where pointwise scoring with a stronger judge outperforms tournament-based ranking with a weaker one. Third, the paper uses Qwen3-Max as both the RL training judge and one of two evaluation judges (alongside Claude-4-Sonnet), creating a potential judge-model confound: the policy may be optimizing toward Qwen3-Max's specific preferences, and the evaluation partially rewards this alignment. The human-LLM consistency analysis (Figure 4b, 73.9% agreement) provides some calibration but does not isolate judge-specific bias.

What evidence exists in the paper. The paper reports no experiments with alternative base models or alternative RL-training judges. The human-LLM consistency analysis (Figure 4b) evaluates the agreement between the evaluation judges (Qwen3-Max and Claude-4-Sonnet) and human raters, but does not assess whether training with a different judge would produce a policy that generalizes differently. The Real-World Business Application results (Section 6.5) use Amap ecosystem data, which provides some domain generalization but still uses the same base model and judge.

Mitigation status. The paper does not address this limitation or suggest cross-model validation as future work. The claim that ArenaRL is a "general framework" is implicitly a claim of model- and judge-independence that the experiments do not test. A practitioner considering ArenaRL for a non-Qwen model family or with a different judge model has no direct evidence that the reported gains would transfer.

---

The Computational Cost of Difficulty Estimation (or Its Equivalent) Is Not Accounted For

The assumption or constraint. While ArenaRL does not require explicit difficulty estimation (unlike the main example paper in the prompt), it replaces pointwise scoring (one judge call per trajectory) with tournament-based ranking (2N-2 pairwise comparisons per group of N trajectories under Seeded Single-Elimination). Each pairwise comparison involves two bidirectional judge calls (evaluating the pair twice with swapped positions, Section 3.3, Equation 3), and each judge call processes two full trajectories including chain-of-thought reasoning and tool invocation logs. For N=16 (the group size used for Open-Travel main results), this means 30 pairwise comparisons × 2 bidirectional calls = 60 judge API calls per group, each processing thousands of tokens. With K=8 groups per training step (Appendix A), this is approximately 480 judge calls per RL step.

The consequence. The paper's main efficiency claim — that Seeded Single-Elimination achieves near-Round-Robin accuracy at linear cost — compares tournament topologies against each other, not against the pointwise baselines. A fair comparison of total RL training cost would account for the fact that GRPO and GSPO require one pointwise judge call per trajectory (16 calls per group, or 128 calls per training step at K=8) versus ArenaRL's approximately 480 calls per training step. The judge model (Qwen3-Max) is a large closed-source model whose API calls incur latency and monetary cost that scale with token count. This cost is not quantified or discussed in the paper. The reported 41.8% vs. 16.4% win rate advantage (Table 3) is therefore a quality-at-any-cost comparison, not a cost-normalized one.

This matters practically because the judge cost may dominate the RL training budget. The paper states (Appendix A) that RL training runs on 8 NVIDIA H20 GPUs, but this counts only the policy model's computation, not the arena judge's API inference cost. For long-horizon agent trajectories (Open-DeepResearch trajectories can span tens of thousands of tokens), the judge's per-call token count — and thus cost — is substantial. A deployment team evaluating ArenaRL would need to weigh the 2.55× improvement in win rate (41.8% vs. 16.4%) against the approximately 3.75× increase in judge API calls (480 vs. 128 per step) to determine whether the quality gain justifies the cost.

What evidence exists in the paper. The paper reports comparison cost only in terms of "number of pairwise comparisons" (Table 2), not in terms of wall-clock time, API cost, or total FLOPs. The N-2 configuration (Section 6.4, Figure 4a) shows ArenaRL achieves 20.8% win rate — still above SFT's 16.4% — with only 2 comparisons per group (4 bidirectional calls), suggesting that even minimal tournament cost yields gains on this benchmark. However, the cost scaling to the N=16 configuration used for main results is never discussed.

Mitigation status. Not addressed. The paper treats "complexity" as number of comparisons and implicitly equates this with training cost, but does not account for the fact that each ArenaRL comparison is more expensive than a pointwise scoring call (it processes two trajectories and is performed bidirectionally). No cost-normalized comparison (e.g., training budget in total judge tokens vs. performance) is provided. The paper's suggestion (Section 7) to explore "how to efficiently extend ArenaRL to multimodal agent settings" does not mention computational cost optimization as a direction.

---

Process-Aware Evaluation Is Not Ablated — Its Contribution to the Gains Is Unknown

The assumption or constraint. The paper's arena judge evaluates trajectories using a process-aware rubric that scores logical consistency of chain-of-thought, precision of tool calls, and reliability of the final answer (Section 3.3). The paper argues this "ensures that the optimization signal reinforces the agent's intrinsic reasoning capabilities rather than merely overfitting to surface-level features of the final answer," but this design choice is never tested against a simpler outcome-only pairwise evaluation (comparing only final answers, without access to intermediate reasoning or tool calls).

The consequence. The paper's central claim is that tournament-based relative ranking (not process-aware evaluation) solves discriminative collapse. But without an ablation comparing process-aware pairwise evaluation to outcome-only pairwise evaluation, it is impossible to determine how much of ArenaRL's performance advantage comes from the tournament structure versus the richer evaluation rubric. It is possible that outcome-only pairwise tournaments would achieve similar or identical gains — in which case the process-aware rubric is unnecessary complexity — or that the process-aware rubric is the primary driver of improvement, with the tournament structure playing a supporting role. Either scenario changes the interpretation of the paper's contribution.

The theoretical argument for process-awareness — that it prevents the policy from "overfitting to surface-level features" — is plausible but untested. On structured agent tasks like Open-Travel, it is conceivable that a judge evaluating only final itineraries (without seeing the planning process) could still distinguish quality effectively, since the final itinerary either satisfies constraints or doesn't. On open-ended writing tasks (Table 4), where ArenaRL also shows gains, there is no "tool use" to evaluate, making the process-awareness dimension irrelevant — yet the gains persist, suggesting that tournament-based ranking alone provides value independent of process evaluation.

What evidence exists in the paper. None. No ablation compares process-aware to outcome-only pairwise evaluation. The qualitative case study (Appendix F) shows that ArenaRL improves reasoning patterns, but this demonstrates correlation, not causation: if outcome-only tournaments also improved reasoning (because better reasoning leads to better answers), the qualitative evidence would look identical. The per-rubric breakdown for Open-DeepResearch (Table 3) shows ArenaRL strong on Tool Usage (77.3%) and Coverage (78.8%), which is consistent with process-aware evaluation reinforcing these capabilities, but a control with outcome-only evaluation is needed to attribute causality.

Mitigation status. Not addressed. The paper treats process-awareness as an integral component of ArenaRL without testing whether it is load-bearing. The ablation is straightforward — use the same tournament topology but with a rubric that evaluates only the final answer — and would substantially clarify which design choices matter.

---

Topology Selection Is Performed on the Test Set and Only at One Group Size

The assumption or constraint. The systematic comparison of five tournament topologies (Table 2) — the empirical foundation for choosing Seeded Single-Elimination — is conducted at N=8, K=8 and evaluated directly on the Open-Travel test set (the same 250 samples used for final evaluation in Table 3). The paper does not describe a held-out validation set for topology selection, nor does it report topology comparisons at the N=16 group size used for main results.

The consequence. This creates a potential test-set overfitting concern for the topology choice. Even though the paper argues the selection is principled (balancing efficiency and accuracy, Section 4.3), the specific win rates in Table 2 that motivate the choice are measured on the evaluation data. The 0.4 percentage point gap between Seeded Single-Elimination (32.5%) and Round-Robin (32.9%) could be entirely due to sampling noise (with 250 test samples split across five subtasks, each subtask has 50 samples, and subtask-level win rates have substantial variance). A different random split, or the application of these topologies to a held-out validation set, could show Round-Robin clearly ahead, or could show a different topology as optimal.

More importantly, the topology ranking may not be stable across group sizes. The main results use N=16 (Table 3), which doubles the group size from the topology comparison setting (N=8). With N=16, the Round-Robin requires 120 comparisons vs. 30 for Seeded Single-Elimination — a 4:1 ratio that changes the efficiency tradeoff. It is plausible that at N=16, Seeded Single-Elimination's accuracy advantage over alternatives changes: the seeding phase uses N-1=15 anchor comparisons, which may provide better initial ordering with more trajectories, and the elimination phase depth increases from 3 rounds (log2 8) to 4 rounds (log2 16), providing more within-tier resolution. Conversely, Double-Elimination at N=16 with random seeding might fare better or worse than at N=8. The paper provides no evidence either way.

The M-Day subtask results in Table 2 are also instructive: it is the only subtask excluded from SFT training, making it a generalization test. The topology comparison shows Round-Robin at 19.0% vs. Seeded Single-Elimination at 18.1% — a 0.9 point gap that, while small, suggests Round-Robin may provide marginally better generalization. At N=16 (Table 3), ArenaRL achieves 21.0% on M-Day, but without a Round-Robin comparison at this group size, it is unknown whether the simpler topology sacrificed generalization performance relative to the gold standard.

What evidence exists in the paper. The topology comparison is only at N=8, evaluated on the test set (Table 2). No held-out validation set is described. No topology comparison at N=16 is reported. The paper validates the N=16 Seeded Single-Elimination configuration through group size ablation (Figure 4a), but this validates the chosen topology at different group sizes, not the topology choice itself.

Mitigation status. The paper does not acknowledge this as a limitation. The topology selection is presented as a principled choice based on Table 2, without noting that the evaluation data used for selection is the same data used for final reporting. A two-fold cross-validation within the test set (similar to what the example paper in the prompt used for strategy selection) would have partially mitigated this concern. The computational efficiency argument for Seeded Single-Elimination (O(N) vs. O(N²)) provides some independent justification, but does not guarantee that Seeded Single-Elimination is the best O(N) topology for all settings.

---

The Paper Does Not Resolve the Fundamental Tradeoff Between Exploration Diversity and Ranking Accuracy

The assumption or constraint. ArenaRL's tournament mechanism assumes that within a group of N trajectories, the relative ranking produced by pairwise comparisons reflects genuine quality differences that the policy should learn from. This assumption breaks down in two regimes that the paper does not disentangle: (1) when trajectories are too diverse (early in training or on hard tasks), pairwise comparisons may be unreliable because the judge cannot meaningfully compare trajectories that take fundamentally different approaches; (2) when trajectories are too similar (discriminative collapse), pairwise comparisons lose resolution. There is a "sweet spot" of trajectory diversity where the tournament produces the most informative rankings, but the paper does not characterize where this sweet spot lies or how it shifts during training.

The consequence. The group size ablation (Figure 4a) shows that larger N monotonically improves performance up to N=16, which the paper interprets as "broadening the exploration space" leading to discovery of better trajectories. But larger N also increases trajectory diversity, potentially pushing the group into the regime where pairwise comparisons become less reliable. The fact that performance continues to improve suggests this reliability degradation has not yet bitten at N=16, but the paper does not identify the saturation point. For tasks with even larger solution spaces (e.g., open-ended research with no structural constraints), the optimal N might be smaller because the judge cannot reliably compare trajectories that take radically different approaches (e.g., one trajectory does a deep dive on one aspect while another provides broad but shallow coverage). The paper provides no diagnostic for determining appropriate group size for a new task.

The paper's direct-RL experiment (Figure 4c) is relevant here. Starting from a generic base model with zero task capability, the policy initially produces very poor trajectories. The fact that ArenaRL can bootstrap from this state (reaching 71% on Search after 160 steps) means the tournament can extract useful signal even from low-quality trajectories — but it does not reveal how many steps are wasted because early pairwise comparisons were uninformative, or whether a different exploration strategy (e.g., starting with smaller groups and expanding as quality improves) would be more sample-efficient.

What evidence exists in the paper. The group size scaling (Figure 4a) shows monotonic improvement up to N=16 but does not identify a saturation point or degradation. The direct-RL experiment (Figure 4c) shows slow initial progress (~20 steps before meaningful improvement), which could reflect the tournament struggling with low-quality trajectories, but the paper does not analyze this phase. The Double-Elimination result in Table 2 (30.2% with random seeding vs. 32.5% for Seeded Single-Elimination) provides indirect evidence that seeding quality matters: without good initial ordering, the elimination phase makes poorer matchup decisions, which could compound in high-diversity groups where the anchor itself may be low-quality.

Mitigation status. Partially addressed through the design of the anchor mechanism. The greedy-decoded anchor trajectory (T=0) serves as a stable reference point that provides a baseline quality estimate for seeding, regardless of how diverse the exploratory trajectories are. However, the anchor's quality depends on the policy's current capability — if the policy is poor (early training, hard tasks), the anchor is poor, and seeding accuracy degrades. The paper does not propose adaptive strategies (e.g., varying the exploration temperature or group size based on estimated policy capability, or using multiple anchors for different exploration strategies) and does not identify the anchor quality threshold below which the tournament becomes unreliable.

---

Open-DeepResearch Valid Generation Rate Metric Reveals a Task-Completion vs. Quality Tradeoff

The assumption or constraint. The Open-DeepResearch benchmark introduces a valid generation rate (Val. %) metric — the proportion of test cases where the model successfully generates a valid final answer without context overflow. This metric captures a binary dimension of task performance (did the agent complete the task at all?) that is orthogonal to the win rate (how good was the completed output?). The paper reports (Table 3) that ArenaRL achieves 99.0% Val. % while SFT achieves only 32.0%, and GRPO/GSPO degrade to 17.0% and 21.0% respectively.

The consequence. The dramatic gap in valid generation rates creates an interpretation problem for the win rate metric. The win rate for each method is computed conditioned on valid generations (Section 5.3): "we compute the candidate model's win rate against the baseline conditioned on valid generations." This means the win rates for SFT (16.7%), GRPO (25.2%), and GSPO (25.2%) are calculated over only 32%, 17%, and 21% of the test set respectively — the subset of queries where these methods actually produced an answer. ArenaRL's 64.3% win rate is computed over 99% of the test set. These win rates are not directly comparable because they are conditioned on different (and differently selected) subsets of the data. It is possible that the 32% of queries where SFT succeeds are systematically easier than the full test set, inflating SFT's conditioned win rate relative to what it would achieve if it could complete all queries. Similarly, the queries where GRPO/GSPO succeed (17–21%) may be an even more selected subset.

This conditioning artifact means the paper cannot cleanly decompose ArenaRL's advantage into "better task completion" versus "better quality on completed tasks." Did ArenaRL achieve 64.3% win rate because its answers were genuinely better, or because it answered more queries (including harder ones where its answers were strong), or both? The paper's narrative in Section 6.3 — that GRPO/GSPO "slightly improve the average win rate" but degrade valid generation — implicitly treats the two metrics as independent, but they are confounded by the conditioning.

What evidence exists in the paper. Table 3 reports both metrics side by side and the paper discusses them separately (Section 6.3 notes the valid generation gap). However, the paper does not report "unconditioned win rate" (treating invalid generations as losses, which would be the fairest comparison) and does not analyze whether valid generations for different methods come from systematically different subsets of the test set. The per-rubric scores for each method (Framework, Tool Usage, Coverage, etc.) are also conditioned on valid generations, so the rubric-level gaps between ArenaRL and baselines suffer from the same interpretability problem.

The valid generation rate metric captures a real phenomenon — long-horizon agent tasks have a completion dimension that short-form generation tasks lack — but the paper's evaluation protocol does not fully account for it in the reported comparison. The writing benchmarks (Table 4) do not have this issue because all methods presumably produce valid outputs for every query (writing tasks have no context-overflow failure mode).

Mitigation status. The paper does not address the conditioning artifact or report unconditioned metrics. One could argue that the valid generation rate is the primary metric and that win rate is secondary — a model that completes 99% of tasks with a 64.3% win rate is unambiguously superior to one that completes 17% of tasks with a 25.2% win rate, regardless of conditioning artifacts. This is a reasonable position that the paper could have taken explicitly but didn't. The writing benchmark results (Table 4), where this confound does not exist, provide cleaner evidence for ArenaRL's quality advantage, though the gap there is smaller (+6.70% over GRPO).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing rather than a full paradigm shift, but the reframing is sharp enough to redirect research attention and invalidate several plausible-but-incorrect intuitions about RL for open-ended tasks. The core contribution to the field's conceptual toolkit is the identification and naming of discriminative collapse as the specific mechanism by which pointwise reward scoring fails — a mechanism that is structural (inherent to the interaction between policy improvement and judge noise), not merely contingent on poor judge quality. This matters because it transforms the problem from "we need better judges" (an AI capability problem with an unclear timeline) to "we need better evaluation protocols" (an algorithm design problem that can be tackled with current tools). The paper's empirical demonstration that a carefully organized tournament over pairwise comparisons — using the same judge model that fails in pointwise mode — can recover robust RL training signals is the existence proof for this reframing.

The shift has several concrete consequences for how the field should think about open-ended RL:

First, it establishes that the reward signal generation protocol is a first-class design parameter, not an implementation detail. Prior work on RL for LLMs treated the reward function as something you plug in — if it's a ground-truth rule (math, code), use it directly; if not, substitute an LLM judge and hope it works well enough. ArenaRL treats the organization of evaluations as the primary mechanism for extracting reliable signal from an imperfect judge. This is analogous to the evolution from single-annotator labeling to multi-annotator consensus protocols in dataset creation: the insight is that aggregation structures can recover signal from noisy individual judgments. The paper's systematic comparison of five tournament topologies (Table 2) demonstrates that the choice of aggregation structure matters enormously — a 4.7 percentage point gap between Anchor-Based (27.8%) and Seeded Single-Elimination (32.5%) from the same judge model — and that the optimal structure is not obvious a priori (Double-Elimination and Swiss-System both underperform simpler designs). This opens a research area that did not previously exist: the design of evaluation topologies as an optimization problem in its own right.

Second, it partially reconciles contradictory findings in the literature about whether RL helps or hurts on open-ended tasks. The paper's results (Table 3) show that GRPO and GSPO — which have been reported as successful on math and code — provide essentially zero improvement over SFT on Open-Travel (16.4% vs. 16.4% for GRPO, 17.2% for GSPO) and actively degrade task completion on Open-DeepResearch (valid generation dropping from 32% to 17–21%). Under a naïve view where LLM judges are merely "noisy but unbiased," this degradation is puzzling — noise should cause plateauing, not deterioration. The discriminative collapse mechanism explains the deterioration: the normalization step in GRPO/GSPO amplifies judge noise when score variance shrinks, creating large-magnitude gradient updates in effectively random directions. This reconciles the apparent contradiction between positive reports of pointwise RL on tasks where trajectories remain diverse (early training, easy tasks, tasks with clear correct/incorrect distinctions) and negative results on tasks where policies quickly converge to a narrow quality band (well-initialized agents on structured planning tasks). The key variable is not "does the task have ground truth?" but "does the policy's output distribution maintain sufficient variance for the judge's noise floor?"

Third, it makes tournament-based reward generation the natural first approach to try for any new open-ended RL problem. Before this paper, a practitioner facing a new open-ended task would likely start with pointwise LLM-judge scoring (the default in existing RL frameworks like GRPO) and only explore alternatives if that failed. After this paper, the default should arguably be some form of comparison-based ranking — at minimum, anchor-based comparisons (which require only N-1 pairwise calls and already improve over pointwise scoring, per Table 2) — with pointwise scoring requiring explicit justification. The paper's evidence that even the minimal pairwise setting (N=2, Figure 4a) outperforms SFT (20.8% vs. 16.4%) suggests that the comparison paradigm provides benefits at every scale, not just at the large-group, tournament-optimized extreme. This shifts the burden of proof: pointwise scoring should now be treated as the risky choice, not the safe default.

Fourth, it redirects research investment away from judge quality improvement and toward evaluation protocol design. The paper's central finding — that the same judge (Qwen3-Max) can produce dramatically different RL outcomes depending on whether its outputs are used pointwise or in a tournament (41.8% vs. 16.4% on Open-Travel, Table 3) — implies that evaluation protocol matters more than judge quality within the quality range of current frontier models. This does not mean judge quality is irrelevant (a very poor judge would produce noisy pairwise comparisons that degrade tournament accuracy), but it suggests that the marginal return on improving judge quality in the pointwise paradigm is near zero when discriminative collapse has already set in, whereas the marginal return on switching to a tournament protocol with the existing judge is enormous. Research programs focused on building better reward models for open-ended tasks (better rubrics, better calibration, multi-dimensional scoring) are not invalidated, but they should now be evaluated in the context of tournament-based protocols — a judge improvement that yields +2% in pointwise mode might yield +10% when its improved discriminative ability is leveraged through pairwise comparisons in a tournament bracket. The paper's human-LLM consistency analysis (Figure 4b, 73.9% agreement) provides a benchmark for what level of judge quality is needed for tournament-based RL to work — and 73.9% agreement, while far from perfect, is evidently sufficient.

Fifth, it reframes the cold-start problem in RL for agents. The direct-RL experiment (Figure 4c) — training Qwen3-8B-Base from scratch on the Search subtask and reaching 71% — demonstrates that tournament-based ranking can bootstrap from a policy with zero initial task capability. This challenges the implicit assumption in much prior work that SFT cold-start is necessary for RL on complex agent tasks. The mechanism is worth understanding: even when all trajectories are poor, pairwise comparisons can still distinguish relative quality ("this bad trajectory is slightly less bad than that one"), providing gradient signals that gradually improve the policy. This is a form of curriculum learning induced by the evaluation protocol — the tournament doesn't need absolute quality judgments, only relative ones, and relative judgments remain possible at all quality levels. The practical implication is that organizations lacking expensive SFT annotation pipelines can still apply RL to agent tasks, though the paper does not quantify the sample-efficiency cost of skipping cold-start (how many more RL steps are needed to match SFT→RL performance).

Follow-Up Research This Work Enables

1. Process-awareness ablation to isolate the tournament structure's contribution. The paper's arena judge uses a process-aware rubric that evaluates chain-of-thought reasoning, tool invocation quality, and final answer reliability — but this design choice is never ablated against a simpler outcome-only evaluation (comparing only final answers). This ablation is the single most important follow-up experiment because it determines which of two interpretations of ArenaRL's success is correct: (a) pairwise comparison is the key innovation, and process-awareness is incidental (in which case the rubric structure can be simplified, reducing judge token costs and making the approach applicable to tasks without visible reasoning traces), or (b) process-awareness is load-bearing, and the tournament structure amplifies its benefits (in which case future work should focus on richer evaluation dimensions, not just comparison protocols). A clean experiment would replicate the Open-Travel main results (N=16, K=8) with two ArenaRL variants: one using the full process-aware rubric and one using a rubric that presents only the final itinerary and asks the judge to compare quality without access to reasoning traces. If outcome-only ArenaRL matches or approaches process-aware ArenaRL's 41.8% win rate, the paper's emphasis on process evaluation is overstated; if it falls significantly (e.g., toward 30%), process-awareness is confirmed as a critical component. The per-rubric Open-DeepResearch scores (Table 3) — where ArenaRL is strongest on Tool Usage (77.3%) and Coverage (78.8%) — hint that process evaluation matters, but correlation is not causation without the ablation.

2. Judge model robustness study to determine the minimum viable judge quality. ArenaRL uses Qwen3-Max as the training judge, a model that is substantially larger and more capable than the 8B policy being optimized. This creates a potential capability asymmetry: the tournament works because the judge can reliably distinguish trajectory quality, but what happens when the judge is closer in capability to the policy — or even weaker? This question matters practically because many deployment settings lack access to frontier closed-source judges, and using them for online RL training (hundreds of steps × hundreds of pairwise calls per step × thousands of tokens per call) could be prohibitively expensive. A systematic study should train ArenaRL with judges spanning a capability gradient — e.g., Qwen3-8B (same scale as the policy, evaluating itself), Qwen3-32B, Qwen3-Max (the paper's choice), and a held-out stronger model like Claude-3.7-Sonnet — and measure how final policy quality degrades as judge quality decreases. The human-LLM consistency analysis (Figure 4b, 73.9% agreement) provides a calibration point for the current judge, but a robustness curve would identify the agreement threshold below which tournament-based RL stops outperforming pointwise baselines. If a same-scale judge (Qwen3-8B evaluating Qwen3-8B) can still drive improvements, ArenaRL becomes much more practically accessible; if not, the approach is contingent on judge-policy capability gaps that limit its applicability.

3. Seeded Double-Elimination to determine whether bracket structure or seeding is the binding constraint. The paper's topology comparison (Table 2) shows Double-Elimination with random seeding underperforming Seeded Single-Elimination (30.2% vs. 32.5%), which the paper attributes to the absence of anchor-based seeding. But this conflates two variables: the bracket structure (single vs. double elimination) and the seeding quality (anchor-based vs. random). A clean follow-up would test Double-Elimination with anchor-based seeding against Seeded Single-Elimination at matched computational budgets. The hypothesis: a more robust bracket structure, when properly seeded, may produce more accurate rankings than single-elimination by giving strong trajectories a second chance after an unlucky pairwise loss (due to judge noise). At N=16, Double-Elimination with seeding would require approximately 3N comparisons (N-1 for seeding, approximately 2N for the two brackets) versus 2N-2 for Seeded Single-Elimination — a 50% cost increase that might be justified if it measurably improves ranking accuracy, particularly on hard subtasks where judge noise is likely higher. The paper's M-Day results (Table 2), where Round-Robin slightly outperforms Seeded Single-Elimination (19.0% vs. 18.1%), hint that the current topology may leave some accuracy on the table for generalization tasks, and a more robust bracket could recover it.

4. Cross-model-family replication to establish generality. Every experiment in the paper uses Qwen3-8B-Base as the backbone model. The discriminative collapse mechanism is argued to be universal — it depends on the interaction between policy convergence and judge noise, not on model-specific properties — but this universality claim is untested. A minimal replication package should reproduce the Open-Travel main results (N=16, K=8, Seeded Single-Elimination) with at least two non-Qwen base models at comparable scale: Llama-3-8B and Mistral-7B (or their successors). The key measurements are: (a) does ArenaRL improve over SFT by a similar margin across model families, or is the gain model-specific? (b) do GRPO and GSPO similarly fail to improve (consistent with universal discriminative collapse) or do they show gains on some models (suggesting model-specific properties affect collapse severity)? (c) does the optimal tournament topology change across model families? If ArenaRL's gains are consistent across models, the framework's generality is validated; if gains vary substantially, the paper's claims need to be scoped to specific model-judge interactions. This experiment is straightforward (same benchmarks, same judge, same hyperparameters, different base models) and would substantially strengthen the paper's contribution.

5. Training dynamics analysis to directly validate the discriminative collapse mechanism. The paper's central theoretical claim — that pointwise RL fails because σ_group shrinks relative to σ_noise, causing normalization to amplify noise — is supported by one illustrative figure (Figure 1a) but never systematically validated across training steps. A training-dynamics experiment would track, for both ArenaRL and GRPO/GSPO on Open-Travel, the following metrics at each RL step: intra-group variance of true trajectory utilities (σ_group, estimated via the tournament-based ranking as a proxy for ground-truth quality), noise variance (σ_noise, estimated via repeated pointwise scoring of the same trajectory with the judge), the signal-to-noise ratio (σ_group / σ_noise), the policy's average win rate on a held-out validation set, and the effective gradient norm. The predictions from discriminative collapse are: (1) for GRPO/GSPO, σ_group should decrease over training while σ_noise remains roughly constant, causing SNR to decline; (2) when SNR crosses below some threshold (~1, as suggested by Figure 1a), policy improvement should stall or reverse; (3) for ArenaRL, the rank-based advantage distribution should maintain stable variance regardless of SNR, preventing noise amplification. This experiment would transform the paper's contribution from "tournament-based ranking works better" (an empirical claim) to "discriminative collapse is the mechanism of pointwise failure, and tournament-based ranking addresses it by decoupling advantage variance from judge noise" (a mechanistic claim with predictive power). The experiment requires instrumenting the training loop to capture judge variance statistics, which is implementationally nontrivial but scientifically high-value.

6. Adaptive group sizing and dynamic tournament depth based on policy maturity. The paper's group size ablation (Figure 4a) shows monotonic improvement up to N=16, but does not explore whether the optimal N changes during training. Early in training, when the policy is poor and trajectories are diverse, smaller groups with simpler ranking protocols (e.g., anchor-based only, or Seeded Single-Elimination with N=4) might provide cleaner gradient signals than large tournaments where the anchor is low-quality and pairwise comparisons are noisy. Late in training, when the policy has converged and discriminative collapse would set in for pointwise methods, larger groups (N=16 or beyond) provide the exploration diversity needed to discover genuine improvements. A follow-up could design an adaptive tournament scheduler that varies N and the tournament topology based on estimated policy maturity: start with N=4 and Anchor-Based ranking for the first ~20% of training steps (where discriminative collapse is not yet a concern, and efficient exploration matters more than ranking resolution), then expand to N=8 with Seeded Single-Elimination for the middle 40% of training (as the policy improves and fine-grained ranking becomes valuable), and finally extend to N=16 or N=32 for the final 40% (to maintain exploration diversity as the policy saturates). The evaluation would compare sample-efficiency (final performance per total RL steps) and compute-efficiency (final performance per total judge calls) against the fixed-N=16 approach used in the paper. If adaptive scheduling matches or exceeds fixed-N performance at lower total cost, it would make tournament-based RL more practical by reducing judge API expenses during the early, low-signal phase of training.

Practical Applications and Downstream Use Cases

1. Cost-efficient fine-tuning of open-source agents for domain-specific planning tasks. Consider a travel company or logistics provider that wants to deploy an LLM agent for customer-facing itinerary planning. They have a domain-specific SFT dataset (a few thousand examples of good itineraries) and access to a powerful judge model (e.g., an internal deployment of a large LLM, or API access to a frontier model). Using standard GRPO with pointwise scoring, the paper's results (Table 3, Open-Travel) suggest they would see essentially no improvement over SFT — the agent would remain at ~16% win rate against a strong baseline, with the RL compute wasted. Using ArenaRL with Seeded Single-Elimination at N=16, the same SFT initialization and judge model can drive the agent to 41.8% win rate, with particularly strong gains on constraint-heavy subtasks like 1-Day planning (58.0%) and Search (66.1%). The practical benefit is not just higher quality — it is that the RL investment actually pays off, converting a training budget that would be wasted under pointwise RL into a meaningful capability improvement. The main implementation cost is integrating the tournament engine (Algorithm 1 in Appendix D is ~40 lines of pseudocode) and paying for the increased judge API calls (~480 per training step at N=16, K=8 vs. ~128 for pointwise), which for a one-time fine-tuning run is likely acceptable given the 2.55× quality improvement. The paper's real-world Amap results (Section 6.5, 69% → 80% on open-ended planning) provide industry validation that this transfers to production data.

2. Bootstrapping agent capabilities in domains with no SFT data. The paper's direct-RL experiment (Figure 4c) demonstrates that ArenaRL can train an agent from a generic base model with zero task-specific SFT, reaching 71% on the Search subtask after 160 steps. This has direct implications for organizations building agents for novel domains where SFT data is expensive or impossible to collect — for example, an agent that interacts with a proprietary internal API, or an agent for a newly launched product with no historical usage data. Instead of investing in manual trajectory annotation (expensive, slow, requires domain expertise), they can deploy the base model with ArenaRL and let the tournament mechanism bootstrap capability through online exploration. The key requirement is a judge model that can evaluate trajectory quality for the target domain — but this judge only needs to make relative comparisons, not absolute judgments, which is a lower bar. The paper's finding that even N=2 (the simplest pairwise setting) improves over SFT (20.8% vs. 16.4%, Figure 4a) means that even with very limited pairwise budgets, some improvement is achievable. The main risk is sample efficiency: the paper's direct-RL experiment required ~160 steps to reach 71% on a single subtask, and multi-task training would likely require more. A practitioner would need to weigh the cost of RL steps (policy model training + judge API calls) against the cost and latency of SFT data collection.

3. Improving valid generation rates for long-horizon agent tasks. The Open-DeepResearch results (Table 3) reveal a critical failure mode for long-horizon agents that is invisible in short-form benchmarks: task non-completion. The SFT model completes only 32% of deep research queries (context overflow on the remaining 68%), GRPO and GSPO make this worse (17% and 21%), and ArenaRL nearly eliminates the problem (99%). For any deployment where task completion is a hard requirement — automated report generation, regulatory compliance research, due diligence analysis — this metric matters more than answer quality conditional on completion. A model that produces brilliant reports 17% of the time and crashes 83% of the time is useless in production; a model that produces adequate reports 99% of the time is deployable. The paper attributes the completion degradation under GRPO/GSPO to "spurious advantages such as length bias" driving the policy toward patterns that cause context overflow, while ArenaRL's process-aware pairwise evaluation incentivizes efficient tool use and structured reasoning that avoids overflow. Practitioners deploying RL-trained agents for long-horizon tasks should monitor valid generation rate as a primary metric, not just output quality, and should consider tournament-based RL specifically because it appears to preserve (or even dramatically improve) task completion while pointwise RL degrades it.

4. Self-improving writing assistants for professional domains. The writing benchmark results (Table 4) show ArenaRL (80.30% average) outperforming SFT (72.17%), GRPO (73.60%), and GSPO (73.03%) across three diverse writing benchmarks. While ArenaRL does not match frontier closed-source models (Grok-4 at 84.75%, Gemini-2.5-pro at 85.48%), the 7–8 point gap over the pointwise RL baselines represents a substantial improvement for an 8B open model. For organizations building specialized writing assistants — legal document drafting, medical report generation, technical documentation — where deploying a closed-source model is infeasible due to data privacy or cost constraints, ArenaRL offers a path to extract substantially more writing quality from an open-source model than standard RL methods. The per-domain WritingBench results are instructive: on Politics & Law (77.58%) and Education (79.35%), ArenaRL closes much of the gap with frontier models, suggesting that tournament-based RL is particularly effective for structured, constraint-heavy writing domains — exactly the kind of writing that professional assistants need to produce. The implementation cost is primarily judge API calls during fine-tuning; at inference time, the trained model is a standard 8B checkpoint with no additional overhead.

When to Prefer This Method

The paper does not explicitly articulate a decision rule for choosing ArenaRL over alternatives, but the experimental results imply clear conditions where tournament-based ranking is preferable and where it is not, grounded in the paper's findings:

• Prefer ArenaRL when the policy has been initialized to a non-trivial quality level via SFT, such that trajectories are competent but not yet converged. The paper's main results (Tables 3, 4) all use SFT cold-start, and the 2.55× improvement over GRPO on Open-Travel (41.8% vs. 16.4%) is in this regime. If the policy is completely untrained (zero capability), ArenaRL can still bootstrap (Figure 4c, reaching 71% from 0%), but sample efficiency is uncharacterized — SFT cold-start likely reduces total RL steps substantially.

• Prefer ArenaRL when the task has a completion/failure dimension separate from output quality, such as long-horizon agent tasks where context overflow or tool-call errors can prevent valid output generation. The Open-DeepResearch results (Table 3) show ArenaRL achieving 99% valid generation vs. 17–32% for alternatives. Pointwise RL actively degrades completion on these tasks; ArenaRL preserves and dramatically improves it.

• Prefer ArenaRL when you have access to a judge model that is sufficiently capable to make pairwise comparisons but whose absolute scores are poorly calibrated, which describes essentially all current LLM judges. ArenaRL does not require the judge to produce well-calibrated absolute scores — only to reliably determine which of two trajectories is better. The paper uses Qwen3-Max at 73.9% human agreement (Figure 4b), and the approach works; the minimum viable judge quality is unknown but likely modest given that even N=2 pairwise comparisons improve over SFT (20.8% vs. 16.4%, Figure 4a).

• Prefer standard pointwise RL (GRPO/GSPO) when the task has verifiable ground-truth outcomes (math, code, structured prediction) where a rule-based reward function is available. The paper's entire motivation is that pointwise RL works well on such tasks; ArenaRL's tournament mechanism would be unnecessary overhead. The paper does not test ArenaRL on verifiable-reward tasks, so relative performance in that regime is unknown but likely inferior to direct ground-truth rewards.

• Prefer standard pointwise RL when judge API cost is the binding constraint and quality improvement is secondary. ArenaRL at N=16 with bidirectional scoring requires approximately 480 judge calls per training step (30 comparisons × 2 bidirectional calls × 8 groups), versus approximately 128 calls for GRPO (16 trajectories × 8 groups). If judge API budget is fixed and cannot be scaled to accommodate the tournament overhead, the pointwise approach may be the only feasible option — even if it plateaus at a lower quality level. The paper's N=2 configuration (Figure 4a, 20.8% win rate) shows that even minimal pairwise comparison with only 2 comparisons per group (16 bidirectional calls per step, competitive with pointwise cost) provides some benefit, suggesting a spectrum of cost-quality tradeoffs is available, but the paper does not characterize this spectrum explicitly.

• The 14× larger model comparison from the example paper has no direct analog here. ArenaRL does not include a FLOPs-matched comparison between a smaller model with tournament-based RL and a larger model with greedy decoding. The closed-source model comparisons (Tables 3, 4) are not compute-controlled, so they cannot inform pretraining-vs-inference tradeoffs. The paper's finding that ArenaRL-trained Qwen3-8B outperforms several much larger closed-source models on agent tasks (41.8% vs. 31.6% for Claude-3.7-Sonnet, Table 3) is suggestive that inference-time compute (in the form of tournament-based RL training) can compensate for model scale on structured reasoning tasks, but this hypothesis would require a controlled FLOPs-matched experiment to verify.