LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics

ArXiv: 2512.21010

🎯 Pitch

Current leaderboards let a math whiz with terrible safety scores tie a reliable all-rounder. This paper instead runs models through a brutal, sequential elimination tournament, revealing that some top scorers collapse under pressure while steady generalists survive—a ranking no static average can produce.

---

1. Executive Summary

This paper introduces the Competitive Swiss-System Dynamics (CSD) framework, which translates static multi-benchmark LLM results into a dynamic, competition-based ranking by simulating a multi-round sequential contest where models are paired via Swiss-System dynamics and subjected to structured elimination after each round. Evaluated on 29 advanced LLMs across 38 benchmarks spanning basic knowledge, reasoning, coding, and agent capabilities, CSD computes a statistically robust Expected Win Score (E[Sm]) using Monte Carlo simulation (N = 100,000 iterations) to eliminate pairing stochasticity, while its Failure Sensitivity Analysis — parameterized by the per-round elimination quantity Tk — distinguishes between Robust Generalists (models like Gemini-3-pro and GPT-5.1-High whose scores remain stable under increasing elimination pressure) and Aggressive Specialists (models like Qwen-Plus-0728 whose scores collapse when failure is penalized). The framework identifies four distinct performance tiers among the 29 models and demonstrates that CSD rankings are resilient to extreme score perturbations — for instance, zeroing four benchmark scores for Qwen3-Max drops its rank by only a few positions under CSD, whereas simple averaging sends it plummeting from its true position to 19th — establishing that competitive survival dynamics can recover robust rankings even when individual benchmark scores are unreliable, but only when the sequencing of benchmarks reflects the natural dependency structure of real-world deployment pipelines.

2. Context and Motivation

The Core Problem: Synthesizing a Unified Ranking from Fragmented Benchmarks

The fundamental challenge this paper tackles is deceptively simple to state but profoundly difficult to solve: given performance results for multiple LLMs across dozens of specialized benchmarks, how do you produce a single, trustworthy ranking that reflects genuine utility for real-world deployment?

This problem arises from the rapid proliferation of both LLMs and evaluation benchmarks. As the paper notes (Section 1), modern LLM evaluation pipelines "typically comprise hundreds of internal benchmarks." At this scale, manual inspection of individual metrics becomes practically impossible. A practitioner selecting a model for deployment — whether as the backbone for autonomous agents, for enterprise API procurement, or for integration into a complex software pipeline — faces a singular decision: pick one model that must handle "diverse, unpredictable workflows." The high dimensionality of the evaluation landscape demands compression into a unified ranking, but the method of compression determines whether the resulting ranking is informative or misleading.

This is fundamentally a dimensionality reduction problem with no ground truth. Unlike supervised learning, where a loss function provides an objective signal, there is no universally accepted "correct" way to weight the relative importance of, say, a 5-point gain on a mathematics benchmark versus a 3-point gain on a code generation benchmark. The tradeoff between breadth and depth, between general competence and specialized excellence, is inherently subjective — yet deployment decisions must be made nonetheless.

Why This Problem Is Important: Real-World Stakes Beyond Leaderboard Aesthetics

The paper builds its motivation not on abstract metrics but on concrete deployment scenarios where ranking failures translate directly to operational failures. Three examples in Section 2.1 ground the problem:

Supply chain automation. An LLM managing a supply chain must first parse inventory manifests (a foundational capability) before optimizing logistics routes (a complex downstream task). A model that excels at route optimization but fails at manifest parsing is not "70% useful" — it is zero percent useful because the pipeline is blocked at step one. Yet a simple average would rate such a model favorably, masking the catastrophic bottleneck.

Financial risk assessment. Loan underwriting requires simultaneous strength in numerical reasoning (for risk calculations) and regulatory text interpretation (for compliance). A model that is brilliant at one but mediocre at the other represents unacceptable operational risk. The evaluation system must identify whether a model's strengths are broad enough to cover all required competencies, not just whether it has a high average.

Complex code generation. A multi-step coding agent that correctly completes module $d_1$ but fails at $d_2$ provides zero downstream value regardless of how impressive its partial solution was. The effective utility is binary-gated by the weakest link in the chain.

These scenarios share a structure that the paper argues traditional evaluation misses entirely: real-world deployment is path-dependent. Capabilities are not independent addends; they are sequential and interdependent. A failure in a foundational task renders excellence in downstream tasks irrelevant. The paper crystallizes this in Figure 2, which contrasts the "Static Aggregation View" (where a model that fails Instruction Following with score 0 but excels at Reasoning with score 100 gets a passing 50 average) against the "Path-Dependent Reality" (where the initial failure blocks the entire pipeline, yielding effective utility of 0). This figure is more than an illustration — it is the paper's central critique of aggregation-based evaluation.

Prior Approaches and Where They Fall Short

The paper identifies two dominant paradigms in LLM evaluation and explains why neither solves the unified ranking problem adequately.

Pointwise Evaluation: The Static Benchmarking Paradigm

The most widespread approach is pointwise evaluation: run each model on each benchmark independently and report scalar scores (accuracy, F1, perplexity). Systems like HELM, GLUE, MMLU, and the benchmarks used in this paper itself (MMLU-Pro, SuperGPQA, IFEval, etc.) exemplify this paradigm. The paper acknowledges that pointwise evaluation is "essential for measuring task-specific proficiency" (Section 3), but identifies two critical limitations when one needs a holistic ranking.

First, the arbitrary weighting problem. To produce a single ranking from dozens of task-specific scores, one must combine them — and this requires assigning relative importance weights to each benchmark. How much should math count relative to coding? Is instruction following more important than factual accuracy? The paper argues that these decisions are made "based on heuristics rather than data" (Section 1), and that "the final ranking becomes highly sensitive to these manual choices." Different weightings produce different rankings, and there is no objective principle to resolve the disagreement. This is what the paper calls "the lack of an objective ground truth for task importance."

Second, the independence assumption. Even if one could agree on weights, pointwise aggregation treats all benchmark scores as independent contributions to a weighted sum. In the notation of the paper: the effective utility is modeled as $\sum_k w_k \cdot \text{score}_k$ for some weight vector $w$. But as the deployment scenarios demonstrate, capabilities are not independent — they interact through sequential dependencies. A simple weighted average cannot express the fact that failure at step 1 gates success at step 2.

Pairwise Ranking: Elo and Bradley-Terry Models

To overcome the subjectivity of pointwise weighting, the research community adopted pairwise ranking models, most prominently the Elo rating system as implemented in Chatbot Arena (Chiang et al., 2024). These models process pairwise win/loss outcomes and estimate a latent "skill parameter" for each model that best explains the observed outcomes. The paper explicitly credits the "mathematical rigor" of these approaches and their advantage over simple averaging: they produce a "universally comparable skill score that is independent of the dataset used" (Section 3).

However, the paper argues that Elo-based systems, while statistically principled, address a fundamentally different question than what deployment requires. The distinction is crucial:

Static, equilibrium skill vs. dynamic, contest-specific fitness. Elo estimates a model's average win probability against a randomly selected opponent — a property of the model's stationary distribution over all possible matchups. But deployment is not a random matchup; it is a specific, structured pipeline where earlier stages gate later ones. A model that wins 80% of its math matchups and 80% of its coding matchups might be rated identically to one that wins 95% of math and 65% of coding, yet these two models have qualitatively different risk profiles in a sequential deployment. Elo collapses them to a single dimension.

No path dependency. Elo and Bradley-Terry models treat pairwise outcomes as independent observations. There is no mechanism for the outcome of an earlier match to affect later matches — each comparison contributes independently to the skill estimate. But in real pipelines, capability $d_1$ is not independent of capability $d_k$; they are sequentially ordered, and failure at $d_1$ precludes even attempting $d_k$. Elo cannot represent this structural constraint.

No risk quantification. The paper highlights that "traditional pairwise models yield only one dimension: skill ($R$)" (Section 3). There is no second dimension that captures vulnerability — the probability or magnitude of catastrophic failure on a specific capability. A model that is excellent at 9 out of 10 tasks but abysmal at the 10th receives the same Elo as a consistently good model across all 10, yet their deployment risk is radically different. The Failure Sensitivity Analysis that CSD enables has "no equivalent in standard Elo systems" — it is "a diagnostic capability entirely absent" from the prior art.

The Limitation of Existing Leaderboard Aggregations

The paper points to platforms like the "Intelligence" Index on https://artificialanalysis.ai/ as representing the current state of practice: they rank models using aggregate scores (typically simple or weighted averages) across benchmarks. The problem, the paper argues, is that such indices "often mask critical shortcomings, treating a model with high variance (excellent in one area, poor in another) as equivalent to a consistently reliable model" (Section 1). This is not a minor statistical issue — in the context of deployment, it means that a ranked list may present unreliable models as equivalent to robust ones, leading practitioners to select models that will fail unpredictably in production.

How This Paper Positions Itself

The paper positions CSD as a paradigm shift from static aggregation to dynamic competition-based evaluation. This shift operates on multiple levels:

From weighting to structural importance. Rather than asking "how important is math relative to coding?" — a question with no objective answer — CSD asks "given the dependency structure of real-world deployment, what does competitive survival tell us about model robustness?" The "weight" of a benchmark emerges from its sequential position in the contest and the elimination pressure at that stage, not from a researcher's heuristic. Early benchmarks (foundational capabilities) carry higher structural importance because poor performance there triggers early elimination, regardless of how a researcher might subjectively rate their importance.

From independent capabilities to path-dependent fitness. CSD encodes the sequential, interdependent nature of deployment directly into the evaluation structure. The Swiss-System dynamic ensures that a model's performance history determines its future pairings, creating the same kind of "your past constrains your future" dynamic that characterizes real pipelines. The elimination mechanism explicitly penalizes catastrophic failure on any single capability, reflecting the reality that a model that fails one stage of a pipeline cannot recover through excellence at later stages.

From one-dimensional rating to multi-dimensional profiling. By parameterizing the elimination pressure ($T_k$) and tracing the resulting $\hat{E}[S_m](T_k)$ curve, CSD provides not just a ranking but a risk profile for each model. The sensitivity coefficient $\Lambda_m$ distinguishes between Robust Generalists (stable across $T_k$) and Aggressive Specialists (collapsing under pressure). This second dimension answers precisely the question that practitioners need answered: "is this model consistently good across everything I need, or does it have hidden fragility?"

A specific relationship to contest-based evaluation. The paper places itself within the "emerging field of Contest-Based Evaluation" (Section 3) but distinguishes its focus: prior work on tournament structures for AI (e.g., in game theory and multi-agent systems) emphasizes "optimizing pairing strategies for efficiency" — how to find the best model with the fewest matches. CSD, by contrast, uses the tournament structure itself "as a diagnostic tool." The goal is not efficient ranking per se but informative ranking that reveals properties (robustness, fragility, risk) invisible in static methods.

What CSD is not claiming. The paper is careful to acknowledge that CSD does not solve the problem of ground-truth ranking (Section 5.2): "the 'true' ranking of generalist LLMs remains subjective and context-dependent." CSD's ranking reflects a specific utility definition — "competitive robustness and shortfall penalty" — that the paper argues is aligned with practical deployment but is not universally applicable. The ranking is "primarily validated through its internal consistency and its superior robustness demonstrated in the sensitivity analysis, rather than by external correlation with an undisputed standard." This is an honest admission that distinguishes CSD from approaches that claim to have found the right answer.

The paper also acknowledges that CSD requires a benchmark sequence order — a design choice that carries its own subjectivity. The order used in Section 4.1 follows a "two-part rationale: moving from general to specific coverage, and progressing from fundamental to challenging tasks." This is well-motivated for the paper's deployment scenarios but is not the only possible ordering, and different orderings would produce different rankings. The paper does not hide this; it explicitly ties the sequence to the deployment dependency structure being modeled.

3. Technical Approach

3.1 Reader Orientation

The CSD framework is a computational system that takes a set of LLMs and their pairwise win rates across multiple benchmarks, then simulates a tournament — thousands of times — to produce a statistically robust ranking that penalizes catastrophic failure on individual capabilities. It solves the problem of "how to turn dozens of benchmark scores into a single trustable ranking" by replacing subjective weighting with structural competition: the importance of each benchmark emerges from its position in a sequential contest where poor performance leads to elimination, mirroring how real-world deployment pipelines actually work.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a processing pipeline:

1. Pairwise Win-rate Tensor (W): A pre-computed $M \times M \times K$ binary tensor that encodes, for every pair of models and every benchmark, which model wins. This is the sole input derived from raw evaluation data, computed once before any simulation runs.

2. Swiss-System Pairing Engine: A deterministic algorithm that, given the current cumulative scores of all surviving models, partitions them into score-homogeneous groups and pairs opponents within each group. This enforces strength-of-schedule matching — high performers face high performers, low performers face low performers.

3. Single-Instance CSD Simulator (SingleInstanceCSD): A stochastic procedure that executes one full K-round tournament, including pairing, contest execution (lookup from W), score accumulation, and structured elimination after each round. This produces one final score vector.

4. Monte Carlo Approximator: Runs N = 100,000 independent instances of SingleInstanceCSD and averages the final scores to produce the Expected Win Score $\hat{E}[S_m]$ for each model. This eliminates noise from random pairing and random elimination draws.

5. Failure Sensitivity Analyzer: Repeats the entire Monte Carlo process at multiple elimination intensities $T_k$ (e.g., $T_k \in \{0, 1, 2\}$) and computes the sensitivity coefficient $\Lambda_m$, the slope of $\hat{E}[S_m]$ against $T_k$. This yields a model-specific risk profile distinguishing Robust Generalists from Aggressive Specialists.

Information flows as follows: raw benchmark scores → pairwise win-rate tensor W → (Swiss-System pairing + W lookup + elimination) repeated N times per $T_k$ setting → averaged to produce $\hat{E}[S_m](T_k)$ curves → sensitivity coefficient $\Lambda_m$ per model → final tiered ranking with risk classification.

3.3 Roadmap for the Deep Dive

• First, how raw benchmark scores are abstracted into the Pairwise Win-rate Tensor — because this is the input representation that makes the entire framework benchmark-agnostic and eliminates subjective weighting.
• Second, the Swiss-System pairing rules and the Zero-Point Bye rule — because these are the core algorithmic choices that create path dependency and strength-of-schedule matching, distinguishing CSD from random-pairing tournaments.
• Third, the formal stochastic state transition that models one round of competition — because this is where the mathematics of expectation, grouping, and elimination are specified, and where the intractability argument originates.
• Fourth, why an analytical solution is impossible (the state-space explosion argument) — because this is the motivation for the Monte Carlo approximation, and understanding the combinatorial complexity justifies the computational investment.
• Fifth, the Monte Carlo approximation procedure and the Expected Win Score metric — because this is the output that practitioners actually use, and its statistical properties (consistency, what it measures) need to be clearly distinguished from simpler metrics.
• Sixth, the Failure Sensitivity Analysis and the sensitivity coefficient — because this is the paper's most distinctive contribution: a second evaluation dimension that static methods cannot produce.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that competitive tournament dynamics — specifically Swiss-System pairing with structured elimination — can produce LLM rankings that are more informative for deployment decisions than static aggregations, and that the failure sensitivity of a model under increasing elimination pressure provides a risk profile unavailable from any single-number metric.

---

The Pairwise Win-rate Tensor (W): Abstracting Away Raw Scores

The foundational data structure for the entire CSD framework is the Pairwise Win-rate Tensor, denoted $W$. This is an $M \times M \times K$ binary tensor where $M$ is the number of models being evaluated and $K$ is the number of benchmarks in the competition sequence.

$W(i, j, k) \in \{0, 1\}$

where $i$ and $j$ index models (1 through $M$) and $k$ indexes the benchmark used in round $k$ (1 through $K$). The entry $W(i, j, k) = 1$ means model $i$ defeats model $j$ on benchmark $d_k$; $W(i, j, k) = 0$ means model $i$ loses. By construction, $W(j, i, k) = 1 - W(i, j, k)$ — if $i$ beats $j$, then $j$ loses to $i$.

What it computes: For each pair of models $(i, j)$ and each benchmark $d_k$, the tensor records a single binary outcome indicating which model performs better. The tensor is pre-computed exactly once from the raw evaluation data before any tournament simulation begins.

Why this abstraction is critical: The paper makes an explicit design choice to abstract all raw performance scores (accuracy, F1, pass@k, perplexity, etc.) into binary win/loss outcomes. This is not merely a simplification — it is the mechanism by which CSD "inherently bypasses the critical and subjective problem of benchmark weighting" (Section 2.1.1). The paper argues that deciding whether a 10-point gain on a math benchmark is "worth more" than a 5-point gain on a coding benchmark is an irreducibly subjective exercise that makes static-weighted averages fragile. By reducing every pairwise comparison to a binary outcome — a win is a win, regardless of margin — CSD eliminates the need to calibrate score magnitudes across benchmarks. The "importance" of a benchmark is then defined not by an a priori weight chosen by the researcher, but by its sequential position in the contest and the competitive pressure at that stage.

Practical consideration: The tensor requires $\binom{M}{2} \times K$ pairwise comparisons to be fully populated. For the paper's experiments with $M = 29$ models and $K = 12$ rounds/benchmarks, this is $406 \times 12 \approx 4,872$ binary entries — a modest computation. The paper notes that this separation of W pre-computation from the iterative Monte Carlo sampling "ensures that the iterative computational cost remains relatively low, as the expensive step is not repeated within the Monte Carlo sampling" (Section 2.1.1). In other words, the heavy lifting of comparing raw benchmark scores happens once; the tournament simulation that follows only performs fast tensor lookups.

---

The Swiss-System Pairing Engine: How Opponents Are Matched

At the heart of CSD's dynamic is the Swiss-System Pairing Engine, which determines which models compete against each other in each round. The algorithm operates on the principle of strength-of-schedule matching: models are grouped by their current cumulative win score $S_m(k-1)$, and pairings are made within these score groups so that models face opponents of similar proven strength.

The pairing procedure in round $k$:

1. Grouping: All active (non-eliminated) models in round $k$ are partitioned into disjoint score groups $G_s(k) = \{m \in M_k \mid S_m(k-1) = s\}$, where $s$ is a possible cumulative score (0, 1, 2, ..., $k-1$). Let $n_s(k) = |G_s(k)|$ be the number of models in score group $s$.

2. Within-group pairing: For each score group $G_s(k)$, the SwissPairing function randomly pairs models within the group. Since all models in $G_s(k)$ have the identical cumulative score, they are considered equally strong up to this point in the tournament. The pairing produces $\lfloor n_s(k) / 2 \rfloor$ pairs.

3. Bye handling: If $n_s(k)$ is odd, one randomly selected model in the group receives a bye — no opponent for this round. Under CSD's Zero-Point Bye Rule, this model receives $I_m(k) = 0$ points for the round, not the 1 or 0.5 points typical of chess tournaments.

Why the Swiss-System over random pairing? The paper provides a two-part justification (Section 2.1.2):

• For high-performers: The dynamic strength-of-schedule matching ensures that leading models cannot achieve high ranks by defeating only weak opponents. As they accumulate wins, they are progressively matched against other high-scoring models, providing rigorous validation that their performance generalizes across strong competition.

• For low-performers: Similarly, struggling models are paired against each other, enabling "clearer differentiation at the bottom of the ranking." This is crucial for the elimination mechanism (discussed below), because it ensures that elimination decisions are based on meaningful comparisons — weak models are identified by losing to other weak models, not by losing to the strongest models (which would tell us nothing about relative weakness).

Why the Zero-Point Bye Rule? The paper makes this a design principle rather than an arbitrary choice. In traditional chess, a bye may award 1 or 0.5 points. CSD explicitly assigns 0 points because "a bye is a product of random chance (an odd number in a score group $G_s(k)$), not a competitive victory" (Section 2.1.2). The objective is to measure competitive success — points earned through actual wins — and awarding points for byes would "inflate" some models' scores through luck rather than performance. This is particularly important because the Monte Carlo simulation (Section 2.4) already handles stochasticity: by running N = 100,000 independent tournaments, the luck of receiving a bye in any single instance is averaged out. Awarding points for byes would introduce an upward bias that no amount of averaging could remove, since models in odd-sized groups would systematically benefit.

A subtle consequence: The combination of within-group pairing and zero-point byes creates a tournament structure where the only way to increase one's score is to actually defeat an opponent of similar standing. This makes the tournament purely competitive — there are no "free points," no "easy draws," and no mechanism for score inflation except winning against equally-credited opponents. This property is what gives CSD its ability to distinguish genuinely robust models from those that benefit from weak schedules.

---

The Structured Elimination Mechanism: Modeling the Cost of Failure

After each round $k$, the CSD framework applies a Structured Elimination Mechanism: exactly $T_k$ models are permanently removed from the contest. This is where CSD departs most radically from conventional tournament designs and where its alignment with deployment reality becomes explicit.

The elimination procedure:

1. Identify the minimum score group: After all models in round $k$ have been scored and their cumulative scores updated, the system identifies $G_{\min}(k) = \{m \in M_k \mid S_m(k) = \min(\vec{S}(k))\}$ — the set of all active models with the lowest cumulative score.

2. Probabilistic removal: From $G_{\min}(k)$, exactly $T_k$ models are randomly selected for elimination. The probability that any specific model $m \in G_{\min}(k)$ is eliminated is:

$P_{\text{Elim}}(m, k) = \mathbf{1}(m \in G_{\min}(k)) \cdot \frac{T_k}{\max(T_k, n_{\min}(k))}$

where $\mathbf{1}(m \in G_{\min}(k))$ is the indicator function (1 if $m$ is in the minimum score group, 0 otherwise), $T_k$ is the number of models to eliminate in round $k$, and $n_{\min}(k) = |G_{\min}(k)|$ is the size of the minimum score group.

What it computes: For any model $m$ and round $k$, this formula gives the probability that $m$ will be removed from the contest. If $m$ is not in the minimum score group, it survives with probability 1. If it is in the minimum score group, its elimination probability is $T_k / \max(T_k, n_{\min}(k))$ — when the group is larger than the elimination quota ($n_{\min}(k) > T_k$), each member faces probability $T_k / n_{\min}(k)$ of removal; when the quota meets or exceeds the group size, all members are eliminated.

Why the minimum score group? The paper's rationale is that this mechanism "models the real-world deployment-cycle reality that models exhibiting significant failures are often 'eliminated' from consideration" (Section 2.1.3). By targeting only the bottom-performing models, the elimination pressure is applied exclusively to those demonstrating the poorest relative performance. Models that are merely mediocre but not the worst survive, while those that definitively fail are removed. This is a deliberate modeling choice: in deployment, a model that is consistently second-to-last might still be useful; a model that catastrophically fails a foundational task is not.

Why $T_k$ is tunable: The parameter $T_k$ is what enables Failure Sensitivity Analysis. By varying $T_k$ from 0 (no elimination) to larger values (aggressive elimination), the framework probes how sensitive each model's expected final score is to the "penalty for failure." A model that never enters $G_{\min}(k)$ will be unaffected by any $T_k$; a model that regularly scrapes the bottom of the score distribution will see its expected score collapse as $T_k$ increases. This creates a second dimension of evaluation — risk tolerance — that is the paper's most distinctive contribution.

---

Formal Stochastic State Transition: One Round of Competition

The paper provides a formal probabilistic model of a single round's state transition (Section 2.2), which is necessary both for understanding what the Monte Carlo simulation approximates and for the intractability argument that justifies simulation over analytic computation.

State representation: At the start of round $k$, the state of the competition is $X_k = (M_k, \vec{S}(k-1))$, where $M_k$ is the set of models still active (not yet eliminated) and $\vec{S}(k-1)$ is the vector of their cumulative scores through round $k-1$.

Phase 1 — Grouping: The models are partitioned into score groups $G_s(k)$ as described above.

Phase 2 — Scoring: For each model $m$ in group $G_s(k)$, its round-$k$ score $I_m(k) \in \{0, 1\}$ is a random variable. The paper provides its conditional expectation given the state:

$E[I_m(k) \mid X_k] = \begin{cases} \frac{1}{n_s(k)-1} \sum_{j \in G_s(k) \setminus \{m\}} W(m, j, k) & \text{if } n_s(k) \text{ is even} \\ \left(1 - \frac{1}{n_s(k)}\right) \cdot \left[\frac{1}{n_s(k)-1} \sum_{j \in G_s(k) \setminus \{m\}} W(m, j, k)\right] & \text{if } n_s(k) \text{ is odd} \end{cases}$

where $n_s(k) = |G_s(k)|$ is the number of models in the score group, $W(m, j, k)$ is the pre-computed binary win-rate for $m$ against $j$ on benchmark $d_k$, and the sum is over all other models in the same score group.

What it computes: This is the expected number of points $m$ will earn in round $k$, conditional on the current state (which determines which score group $m$ is in and who else is in that group).

• If $n_s(k)$ is even: Every model in the group is paired. Model $m$ faces one randomly selected opponent from the $n_s(k) - 1$ other models in its group. Its expected score is simply its average win rate against all possible opponents in the group — the unweighted mean of $W(m, j, k)$ over $j \neq m$ in $G_s(k)$.

• If $n_s(k)$ is odd: There is a $1/n_s(k)$ probability that $m$ receives a bye (score 0 under the Zero-Point Bye Rule) and a $(1 - 1/n_s(k))$ probability that it plays. If it plays, the expectation is the same as the even case — its average win rate against the $n_s(k) - 1$ other models in the group. The factor $(1 - 1/n_s(k))$ reduces the overall expectation to account for the bye possibility.

Why this form matters: This equation reveals something subtle about CSD's design. The expected score for $m$ in round $k$ depends not on $m$'s absolute performance on benchmark $d_k$, but on its relative performance against the specific subset of models that happen to have the same cumulative score at this point in the tournament. Two models with identical raw benchmark scores could have very different expectations if one happens to be grouped with opponents it consistently beats while the other is grouped with opponents it struggles against. This is not a bug — it is the mechanism by which path dependency enters the framework. A model's early performance determines which score group it lands in, which determines which opponents it faces later, which determines its expected future score. Static aggregation assumes away these second-order effects; CSD makes them central.

Phase 3 — Elimination: Scores are updated: $S_m(k) = S_m(k-1) + I_m(k)$. The new minimum score group $G_{\min}(k)$ is identified, and elimination proceeds as described above, producing the surviving set $M_{k+1}$ and completing the transition to state $X_{k+1}$.

---

Intractability of the Analytical Solution: Why We Must Simulate

The paper's objective is to compute the Expected Win Score for each model $m$ after $K$ rounds:

$E[S_m(K)] = \sum_{k=1}^K E[I_m(k)]$

By the law of total expectation, each term expands to:

$E[I_m(k)] = \sum_{\text{all possible } X_k} E[I_m(k) \mid X_k] \cdot P(X_k)$

What this equation says: To compute the unconditional expectation of $m$'s round-$k$ score, we need to sum over every possible state $X_k$ that could occur by round $k$, multiply the conditional expectation $E[I_m(k) \mid X_k]$ (which we have a formula for) by the probability $P(X_k)$ that this particular state is reached, and sum over all states.

Why this is intractable: The paper identifies two sources of stochasticity that make $P(X_k)$ impossible to compute exactly for non-trivial problem sizes:

1. Pairing stochasticity: Within each score group $G_s(j)$ for all previous rounds $j < k$, random pairings determine which specific opponents each model faces. For a group of size $n$, the number of possible pairings grows super-exponentially.

2. Elimination stochasticity: In each previous round $j < k$, exactly $T_j$ models are randomly selected from $G_{\min}(j)$ for elimination. The number of possible survival configurations compounds across rounds.

The state $X_k$ depends on the entire history of these random events across all previous rounds. The paper states that "the number of possible 'contest histories' grows exponentially, making the direct computation of the probability $P(X_k)$ for every possible state $X_k$ infeasible for any non-trivial $M$, $K$, and $W$" (Section 2.3).

A concrete sense of scale: With $M = 29$ models and $K = 12$ rounds, even with elimination reducing the active set over time, the number of possible score vectors, group configurations, pairing realizations, and elimination trajectories is astronomically large — far beyond what could be enumerated. The pairing alone within a score group of size $n$ has $(n-1)!!$ possible configurations (the double factorial), which for $n = 10$ is already 945 possibilities, and this compounds multiplicatively across groups and rounds.

The consequence: The analytical intractability is not a weakness of the mathematical formulation — it is a fundamental property of the path-dependent stochastic process that CSD deliberately creates. The very feature that makes CSD informative (path dependency) is what makes it analytically intractable. This necessitates the Monte Carlo approximation.

---

Monte Carlo Approximation: Computing the Expected Win Score

Given the intractability of the analytical solution, CSD approximates $E[S_m(K)]$ using Monte Carlo simulation with $N = 100,000$ independent trials.

The simulation procedure (Algorithm 2 in the paper):

1. Initialize an accumulator: $\vec{S}_{\text{total}} \leftarrow \vec{0}_M$, a vector of $M$ zeros to accumulate scores across all trials.

2. Run N independent trials: For each trial $i = 1, 2, \ldots, N$:

   • Execute SingleInstanceCSD(M, D, W, K, T_k) — one complete K-round tournament — which returns a final score vector $\vec{S}^{(i)}(K)$.
   • Accumulate: $S_{\text{total}, j} \leftarrow S_{\text{total}, j} + S^{(i)}_{m_j}(K)$ for each model $m_j$.

3. Compute the estimate:

$\hat{E}[S_{m_j}] = \frac{S_{\text{total}, j}}{N}$

This is the sample mean of the final scores across all $N$ trials.

What the SingleInstanceCSD simulator does internally (Algorithm 1):

• Starts with $M_{\text{active}} = M$ and $\vec{S}(0) = \vec{0}$.
• For each round $k = 1$ to $K$:
  • Partitions $M_{\text{active}}$ into score groups $G_s(k)$ based on $\vec{S}(k-1)$.
  • Within each group, randomly pairs models using SwissPairing.
  • For each pair $(m_i, m_j)$, looks up the outcome: $I_{m_i}(k) = W(i, j, k)$, $I_{m_j}(k) = 1 - W(i, j, k)$.
  • For any model receiving a bye, assigns $I_m(k) = 0$ (Zero-Point Bye Rule).
  • Updates scores: $S_m(k) = S_m(k-1) + I_m(k)$ for all active models.
  • Identifies $G_{\min}(k)$ and randomly removes $T_k$ models from it.
  • If fewer than 2 models remain, breaks early (the contest cannot continue with 0 or 1 models).
• Returns the final score vector $\vec{S}(K)$.

Why N = 100,000? The paper does not derive this number analytically but states it directly as the simulation parameter (Section 2.4: "N = 100,000 iterations"). This is large enough that, by the law of large numbers, the sample mean $\hat{E}[S_m]$ converges to the true expectation $E[S_m]$ with high precision. The standard error of the mean scales as $\sigma / \sqrt{N}$, where $\sigma$ is the standard deviation of final scores across trials. Even for a model with high variance (e.g., $\sigma \approx 2$ wins out of 12 rounds), the standard error at $N = 100,000$ is approximately $2 / \sqrt{100,000} \approx 0.006$ wins — far below the precision needed to distinguish rankings.

What the Expected Win Score actually measures: The paper is explicit that $\hat{E}[S_m]$ "is far richer than a simple average score" (Section 2.4). It is a holistic metric that intrinsically blends three factors:

• Raw win-rate: The entries in the W tensor — $m$'s probability of beating opponent $j$ on benchmark $d_k$.
• Robustness against failure: $m$'s ability to stay out of $G_{\min}(k)$ — its probability of never being the worst model in any round, which depends on consistent performance across all benchmarks.
• Resilience to random chance: The variance in $m$'s final score due to the stochastic elements (random pairing, random elimination, byes). A model with high raw strength but unlucky pairing patterns in many trials will have a lower $E[S_m]$ than one with similar strength but more favorable pairing characteristics on average.

This distinguishes $\hat{E}[S_m]$ from simpler metrics like average accuracy or Elo rating. Average accuracy treats every benchmark independently and does not model elimination. Elo estimates an equilibrium skill parameter from a stationary pairwise distribution but does not sample from explicit tournament trajectories. CSD's metric explicitly simulates what would actually happen if models competed in this structured format, accounting for all the path-dependent interactions.

---

Failure Sensitivity Analysis (FSA): Profiling Risk Beyond a Single Ranking

The paper's most distinctive methodological contribution is the Failure Sensitivity Analysis, which transforms CSD from a ranking system into a diagnostic profiling system by varying the elimination parameter $T_k$ and observing how each model's Expected Win Score responds.

The procedure:

1. Fix a range of $T_k$ values: For the experiments, the paper uses $T_k \in \{0, 1, 2\}$, where $T_k$ is held constant across all rounds (eliminate 0, 1, or 2 models per round).

2. Run the full Monte Carlo simulation independently for each $T_k$: This produces $\hat{E}[S_m](T_k)$ — the expected final score as a function of elimination intensity.

3. Compute the Sensitivity Coefficient $\Lambda_m$:

$\Lambda_m \approx \frac{\Delta \hat{E}[S_m]}{\Delta T_k}$

where $\Delta \hat{E}[S_m] = \hat{E}[S_m \mid T_k = 2] - \hat{E}[S_m \mid T_k = 0]$ is the change in expected score when moving from no elimination to eliminating 2 models per round (and this is divided by 2 to get the per-unit slope, as shown in the paper's definition in Section 4: $\Delta E[S_m] = E[S_m \mid T_k = 2] - E[S_m \mid T_k = 0] = 2 \cdot \Lambda_m$).

What $\Lambda_m$ measures: This is the empirical derivative — the rate at which a model's expected cumulative wins decrease as the "penalty for failure" increases. A large negative $\Lambda_m$ means the model's score collapses rapidly under elimination pressure; a $\Lambda_m$ near zero means the model is largely unaffected.

The resulting model classification:

• Robust Generalist ($\Lambda_m \approx 0$): A model with a near-zero sensitivity coefficient. Its expected score is stable across elimination intensities. This means the model rarely, if ever, enters $G_{\min}(k)$ — it is never the worst performer in any round. Such a model demonstrates consistent, reliable competence across the full spectrum of benchmarks. The paper identifies Gemini-3-pro, GPT-5.1-High, GPT-5-High, GPT-5-Medium, Claude-Sonnet-4.5-Thinking, and DeepSeek-V3.2-Thinking as Robust Generalists (Section 4, Finding 2).

• Aggressive Specialist ($\Lambda_m \ll 0$): A model with a strongly negative sensitivity coefficient. At $T_k = 0$ (no elimination), it may achieve high scores due to excellent performance on its strong benchmarks. But as $T_k$ increases, its score collapses because it regularly falls into $G_{\min}(k)$ on its weak benchmarks and gets eliminated before it can accumulate points on its strong ones. This reveals the model's "fragility to 'short boards'" — the model is a specialist that excels in some areas but has critical deficiencies. The paper identifies Qwen-3-235B, Qwen-Plus-0728, and Qwen-Next-80B-Thinking as Aggressive Specialists.

Why this matters beyond ranking: The classification provides a second evaluation dimension — risk tolerance — that is directly relevant to deployment decisions. The paper frames this as answering the question: "Is it better to have a model that is excellent at 9 tasks but fails 1, or one that is good at all 10?" (Section 2.5). For a high-stakes deployment where failure on any single task is unacceptable (e.g., autonomous systems, financial compliance), the Robust Generalist is the only viable choice regardless of the Aggressive Specialist's higher peak performance. For a deployment where occasional failures can be caught by a fallback system, the Aggressive Specialist's superior performance on its strong tasks might justify the risk. CSD provides the quantitative basis for making this decision: the $\Lambda_m$ coefficient tells you exactly how much expected value the model loses as failure penalties increase.

The diagnostic power of the $\hat{E}[S_m](T_k)$ function: The paper emphasizes that "a single $\hat{E}[S_m]$ score (at a fixed $T_k$) provides a ranking, but the function $\hat{E}[S_m](T_k)$ provides a model risk profile" (Section 2.5). This is a shift from point estimation to functional estimation. The function encodes how the model behaves across a spectrum of competitive pressure levels, analogous to how a stress test reveals properties of a financial portfolio that its expected return alone cannot. A model whose curve is flat across $T_k$ has proven it can handle any reasonable failure penalty; a model whose curve drops steeply reveals hidden fragility that a single-point ranking would miss entirely.

Computational cost: Running the FSA requires independent Monte Carlo simulations for each $T_k$ value. With $N = 100,000$ trials per $T_k$ and three $T_k$ values (0, 1, 2), the total simulation count is 300,000 tournaments. Each tournament runs SingleInstanceCSD with up to $K = 12$ rounds. The paper does not report wall-clock time, but since the expensive part (W tensor computation) is done once and the tournament simulation only involves tensor lookups and random number generation, this is computationally tractable even at this scale.

---

Summary of Design Choices and Their Justifications

• Binary win/loss abstraction over continuous scores: Eliminates the need for cross-benchmark score calibration and avoids the subjective weighting problem. The tradeoff is loss of magnitude information (a narrow win is treated identically to a blowout), but the paper argues this is acceptable because the goal is ranking robustness, not precise performance estimation.

• Swiss-System pairing over random or round-robin: Provides strength-of-schedule matching that efficiently differentiates models at all performance levels. A round-robin would require $M(M-1)/2$ matches per benchmark (impractical for large $M$). Random pairing would allow strong models to inflate their scores by defeating weak opponents. Swiss-System concentrates comparisons where they are most informative.

• Zero-Point Bye Rule over standard bye scoring: Ensures that $E[S_m]$ reflects only competitive success, not luck. This is important because the Monte Carlo averaging already handles stochastic variation — the bye rule prevents systematic bias.

• Elimination from the minimum score group rather than, say, random elimination: Models the realistic penalty structure where catastrophic failure is disqualifying. Eliminating from the middle or top of the distribution would punish excellence rather than failure.

• Monte Carlo with N = 100,000 over smaller sample sizes: Provides high-precision estimates of $E[S_m]$ that are robust to the stochastic elements of individual tournaments. The computational cost is acceptable given that W is pre-computed and the simulation is lightweight.

• $T_k$ as a sensitivity parameter rather than a fixed tournament rule: Enables the Failure Sensitivity Analysis, which provides model-specific risk profiles. This is the design choice that transforms CSD from a ranking tool into a diagnostic framework.

4. Key Insights and Innovations

Innovation 1: Tournament Dynamics as an Evaluation Framework — Path Dependency Replaces Subjective Weighting

The paper's most fundamental intellectual move is not a specific algorithm but a reframing of what evaluation means. Static aggregation asks: "given scores on benchmarks A, B, and C, what weighted sum best captures overall quality?" — a question that has no objective answer because the weights are inherently subjective. CSD replaces this with an entirely different question: "given a sequence of competitive rounds where survival depends on avoiding catastrophic failure, what does a model's expected tournament outcome tell us about its deployment fitness?"

This is a shift from latent-trait estimation to process simulation. Prior work — both simple averaging and sophisticated Elo/Bradley-Terry models — treats evaluation as recovering a hidden "quality" parameter from observed outcomes. The model is assumed to have a fixed skill level, and the evaluation problem is statistical: extract the best estimate of that skill from noisy measurements. CSD rejects this framing entirely. It does not attempt to estimate any latent parameter. Instead, it simulates an explicit, path-dependent competitive process and asks: "if we ran this process many times, what would the average outcome be?" The Expected Win Score is not an estimate of a hidden variable — it is the direct expected value of a well-defined random variable under a well-defined stochastic process.

This matters because it shifts the burden of subjectivity. Static aggregation requires the researcher to decide that math is "worth" X times coding. CSD requires the researcher to decide the sequence of rounds and the elimination intensity — choices that map to concrete, defensible deployment assumptions ("in our pipeline, instruction following gates reasoning, so we put it first") rather than abstract importance weights. One can argue about whether the chosen sequence matches a specific use case, but one cannot argue that CSD "got the weights wrong" — there are no weights to get wrong. The ranking emerges from the structure, not from the coefficients.

This reframing also explains why CSD produces rankings that differ from Elo-based systems. Elo asks "who is stronger on average across all possible matchups?" CSD asks "who survives longer in this specific high-stakes contest?" These are different questions, and they should produce different answers. The paper's contribution is not to claim CSD is "better" than Elo in some universal sense, but to identify a question — competitive fitness under sequential pressure — that Elo does not answer and to provide the first framework that does.

The evidence that this reframing is substantive, not cosmetic, appears in the sensitivity experiments (Section 4.2). When four benchmark scores for Qwen3-Max are zeroed, its simple-average rank plummets from its true position to 19th — the average is dominated by the zeros because it has no structural mechanism to prevent catastrophic values from poisoning the aggregate. CSD's rank drops much less dramatically (from tier-3 to still within tier-3) because the tournament structure limits the damage: early elimination from the weak rounds prevents Qwen3-Max from accumulating many wins on its strong rounds, but it also prevents the zero scores from dragging down its expected total more than the structural penalty already would. The framework's resilience to extreme score perturbations is a direct consequence of its process-based design — structure provides robustness that coefficients cannot.

---

Innovation 2: The Failure Sensitivity Coefficient ($\Lambda_m$) as a Second Evaluation Dimension

The paper introduces a concept that has no analog in prior evaluation work: model-specific sensitivity to failure penalties as a first-class evaluation metric. Traditional methods produce exactly one number per model — an average accuracy, an Elo rating, a Bradley-Terry skill parameter — and that number must encode everything about the model's quality. CSD produces two: the Expected Win Score at baseline ($\hat{E}[S_m \mid T_k = 0]$) and the sensitivity coefficient ($\Lambda_m$), which describes how that score degrades under increasing elimination pressure.

This is not merely "reporting two numbers instead of one." It is the recognition that quality and risk are orthogonal dimensions that cannot be collapsed without loss of information. Two models can have identical expected accuracy across a set of benchmarks yet possess radically different failure profiles: one might be consistently above-average across all tasks, while the other might be brilliant at most tasks but catastrophically bad at one or two. Static metrics rate them identically; $\Lambda_m$ distinguishes them. The paper's classification scheme — Robust Generalist ($\Lambda_m \approx 0$) versus Aggressive Specialist ($\Lambda_m \ll 0$) — operationalizes this distinction.

The intellectual contribution here is the explicit parameterization of risk. Prior work on LLM evaluation has occasionally noted variance (reporting standard deviations or inter-quartile ranges), but variance is symmetric — it penalizes both upside and downside deviation equally. A model that is sometimes brilliant and sometimes terrible has high variance; so does a model that is sometimes mediocre and sometimes slightly above-average. CSD's elimination mechanism is asymmetric: it penalizes only downside deviation (being in the minimum score group). $\Lambda_m$ thus specifically measures vulnerability to catastrophic failure, not general inconsistency. This aligns with the operational reality that a model's worst-case behavior matters more than its best-case for deployment risk assessment.

The empirical demonstration in Figure 4 (plotting base performance against sensitivity coefficient) reveals that this second dimension is not redundant with the first. The top-tier models (Gemini-3-pro, GPT-5.1-High, GPT-5-High, GPT-5-Medium) cluster not just at high performance but also at near-zero sensitivity — they are both good and robust. But the paper also identifies models with high base performance and strongly negative sensitivity (Qwen-3-235B, Qwen-Plus-0728, Qwen-Next-80B-Thinking), which would be indistinguishable from the top tier under any single-dimensional metric. The fact that these clusters are visible and interpretable validates that $\Lambda_m$ captures information not present in $\hat{E}[S_m]$ alone.

This second dimension also enables a new kind of evaluation question: "at what risk tolerance does model A become preferable to model B?" A practitioner with a high-risk-tolerance deployment (where occasional failures are acceptable) might prefer an Aggressive Specialist with high peak performance at $T_k = 0$, while a low-risk-tolerance deployment (where any failure is disqualifying) might require a Robust Generalist even if its peak performance is lower. CSD's $\hat{E}[S_m](T_k)$ function provides the quantitative basis for making this tradeoff explicit, something no prior evaluation framework could do.

---

Innovation 3: The Intractability Argument as a Methodological Justification for Simulation-Based Evaluation

Section 2.3 contains a deceptively simple but intellectually important argument: that the Expected Win Score cannot be computed analytically because the path-dependent stochastic process creates a combinatorial explosion of possible tournament histories. This is not a limitation of CSD — it is presented as a fundamental property of any evaluation framework that attempts to model sequential, path-dependent competition, and it serves as the methodological justification for Monte Carlo approximation.

The significance of this argument is that it establishes simulation not as a computational convenience but as a necessity. Prior evaluation methods (averaging, Elo, Bradley-Terry) are analytically tractable precisely because they assume independence — each data point contributes independently to the final estimate. CSD deliberately breaks this independence (a model's round-k outcome depends on the entire history of who survived, who got paired with whom, and what scores accumulated), and the intractability argument proves that this is not a minor complication but a categorical shift in the computational complexity of the evaluation problem. You cannot "solve" CSD analytically for any non-trivial problem size; you must simulate.

This has implications for how the field thinks about evaluation more broadly. If deployment-relevant evaluation requires modeling sequential dependencies and structured elimination (as the paper argues it does), then the evaluation problem becomes computationally harder in a fundamental way — it transitions from a problem with closed-form solutions to one that requires Monte Carlo methods. The paper does not frame this as a disadvantage but as a feature: the difficulty of analytic solution is proportional to the richness of the evaluation structure. The fact that you cannot write down a formula for $E[S_m]$ in closed form is evidence that CSD is capturing interactions that tractable methods suppress.

The choice of $N = 100,000$ Monte Carlo trials is also methodologically significant. This is substantially more than what would be needed for convergence in a typical simulation study, and it reflects the paper's recognition that the Expected Win Score is not just a ranking device — it is a statistically rigorous quantity whose precision matters. At $N = 100,000$, the standard error is on the order of hundredths of a win (as calculated in Section 3), meaning that the reported rankings are not artifacts of simulation noise. This investment in precision signals that CSD is intended as a production-grade evaluation tool, not a proof-of-concept.

---

Innovation 4: The Sequence-as-Weight Principle — Benchmark Importance as Emergent Structure

The paper articulates — both explicitly in Section 2.1.1 and implicitly through its entire design — a principle that I'll call sequence-as-weight: in a path-dependent competitive evaluation, the relative importance of benchmarks is not a parameter to be chosen but a consequence that emerges from the tournament's sequential structure and elimination rules. A benchmark placed early in the sequence has higher structural importance than one placed late, because failure on an early benchmark triggers elimination that prevents the model from even attempting later benchmarks, whereas failure on a late benchmark only costs the round.

This principle inverts the standard approach to multi-benchmark evaluation. The standard approach says: "first decide how important each benchmark is (by choosing weights), then compute the aggregate." CSD says: "first decide the dependency structure (by choosing the sequence), then let importance emerge from the dynamics." The paper argues that the second question — "what is the dependency structure of real deployment?" — is more answerable than the first — "what is the abstract importance of math versus coding?" — because dependency structures correspond to observable, arguable facts about real-world pipelines, whereas importance weights correspond to ineffable preferences.

The empirical support for this principle is indirect but compelling: the overall ranking produced by CSD (Figure 3) aligns with community consensus while the Failure Sensitivity Analysis (Figure 4) reveals distinctions that static ranking misses entirely. If the sequence-as-weight principle were merely replacing one arbitrary choice with another, we would expect the resulting rankings to be just as sensitive to sequence choices as static rankings are to weight choices. The paper's reported robustness to severe score perturbations (Section 4.2) suggests otherwise — the structure provides stability that ad-hoc weights cannot.

This principle also explains why CSD is naturally suited to the kinds of deployment scenarios described in the motivation (supply chain automation, financial risk assessment, complex code generation). These are scenarios where dependencies are inherent to the task, not imposed by the evaluator. In supply chain automation, manifest parsing genuinely must happen before route optimization — the dependency is a fact about the world, not a choice about evaluation. CSD's sequence parameter gives the evaluator a way to encode these factual dependencies directly into the evaluation structure, whereas static weighting provides no mechanism for expressing dependency at all.

The paper does not claim that the specific 12-round sequence used in Section 4.1 is the uniquely correct one. It acknowledges that the sequence follows a "two-part rationale: moving from general to specific coverage, and progressing from fundamental to challenging tasks" — a defensible but not exclusive ordering. The methodological contribution is not the specific sequence but the existence of a framework where sequence is the primary input and where the evaluator's task is to justify the sequence rather than to justify weights. This is a more tractable and debatable form of subjective input, and that shift — from weight-justification to sequence-justification — is the paper's conceptual contribution to evaluation methodology.

---

Innovation 5: The Agentic Performance Prediction Connection — CSD as a Surrogate for Pipeline Evaluation

Section 5.1.1 makes a brief but provocative claim: that CSD rankings "could reflect (or 'predict') the ranking on a new agentic task" when the benchmark sequence is chosen to mirror the dependency structure of that task. The paper presents Example 3 (Web Navigation and Data Extraction) as an illustration: IFEval (instruction following) maps to interpreting user goals; GSM8K (reasoning) maps to devising action plans; ToolBench (function calling) maps to executing API interactions; HumanEval (code debugging) maps to validating retrieved data. If these benchmarks are sequenced to mirror the agentic pipeline, then the CSD ranking — computed using these existing benchmarks, without any data from the target agentic task — might predict performance on that task.

This is a fundamentally different vision of evaluation from the prevailing "build a new benchmark for every new capability" paradigm. The standard approach to evaluating a new agentic task is to construct a dataset for that specific task, run all models on it, and rank by score. CSD suggests an alternative: if you can decompose the agentic task into its constituent capabilities and map those capabilities to existing benchmarks, you can obtain a ranking for the new task without any new evaluation data, simply by running the CSD simulation with the appropriate sequence.

The intellectual contribution here is the modular recombination of existing evaluation data for novel tasks. This is possible because CSD's tournament structure provides a generic container for sequential capability dependencies — the same simulation engine, the same W tensor, the same Monte Carlo procedure can produce rankings for infinitely many tasks by varying only the sequence input. A single set of pairwise win-rate data across K benchmarks enables (in principle) rankings for any task whose capability dependency graph can be expressed as an ordering over those K benchmarks.

The paper is appropriately cautious about this claim, noting the "significant caveat" that "many recent advanced LLMs inevitably target or optimize for performance on open-source agentic benchmarks during their development, making it more challenging to directly apply the CSD framework to 'predict' performance on novel agentic tasks." This is an honest acknowledgment of the data contamination problem that plagues all evaluation, but it does not diminish the conceptual value of the prediction framework. The contribution is not a specific set of predictions but the architecture for prediction: the claim that competitive tournament dynamics can serve as a transfer mechanism, mapping benchmark-space rankings to task-space rankings through structural analogy.

This connects CSD to broader questions in AI evaluation about generalization and transfer. If CSD rankings on sequenced existing benchmarks correlate with performance on novel agentic tasks, then the framework provides evidence that the tournament structure captures something about general sequential competence that generalizes beyond the specific benchmarks used. If the correlation is weak, then the framework reveals that agentic tasks require capabilities not captured by existing benchmarks — also useful information. Either outcome is scientifically informative, and the paper's willingness to propose the prediction framework as a direction for future work, while transparently acknowledging its limitations, represents intellectual honesty about what CSD can and cannot currently claim.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates across 38 "recent, widely-used, and open-source benchmarks" organized into 12 sequential sub-categories (Table 1), spanning 6 high-level capability categories: basic knowledge, reasoning, instruction following, coding, agent capabilities, and factuality. The benchmarks include MMLU, MMLU-Pro, SuperGPQA, SimpleQA, ChineseSimpleQA, ArenaHard(v2), GMMLU(lite), MMMLU, IFEval, MulDimIF, EIFBench, MultiChallenge, MARS-Bench, ARC-AGI-2, ProcBench, KOR-BENCH, AIME24, AIME25, BeyondAIME, MathConstruct, GPQA(diamond), ScienceOlympiad, Phybench, PHYSICS, LiveCodeBench, CodeForces, HardEval, SWE-Bench, Multi-SWE-Bench, Terminal Bench, GAIA, HLE, BrowseComp, BrowseComp-zh, τ²-bench, FActScore, LongFact (object), and LongFact (concept). The paper also conducts single-benchmark analyses on MMLU-Pro and SuperGPQA (Section 4.2, sub-section 5.1.2). All evaluation data comes from internal evaluation runs; no public train/test splits are referenced for the CSD simulations themselves, since the framework operates on pre-computed pairwise win-rate tensors rather than raw model outputs.

• Base models. The evaluation covers 29 state-of-the-art LLMs from seven organizations: Google (Gemini-3-pro, Gemini-2.5-Pro, Gemini-2.5-Flash, Gemini-2.5-Flash.1, Gemini-2.5-Flash-Lite, Gemini-2.5-Flash-Lite-Preview-2509, Gemini-2.5-Flash-Preview-2509), OpenAI (GPT-5.1-high, GPT-5-chat, GPT-5-high, GPT-5-medium), Anthropic (Claude-Opus-4.1-nothinking, Claude-4-Sonnet-nothinking, Claude-Sonnet-4.5-nothinking, Claude-Sonnet-4.5-thinking), Alibaba/Qwen (Qwen-plus-0728, Qwen3-next-80b-a3b-thinking, Qwen3-next-80b-a3b-instruct, Qwen3-235b-a22b-instruct-2507, Qwen3-max-0923), DeepSeek (DeepSeek-V3.1-Terminus-nothinking, DeepSeek-V3.1-Terminus-thinking, DeepSeek-V3.2-Exp-nothinking, DeepSeek-V3.2-Exp-thinking, DeepSeek-V3.2-thinking), Zhipu AI/GLM (GLM-4.5, GLM-4.6), and Moonshot AI/Kimi (Kimi-K2-0711, Kimi-K2-0905). The selection is designed to represent the current frontier, including both "thinking" and "nothinking" variants where available. No justification is provided for why these specific 29 models were chosen rather than a different set, and the paper does not discuss whether this set is representative of the broader LLM landscape or biased toward high-performing commercial models.

• Metrics. The primary output metric is the Expected Win Score $\hat{E}[S_m]$, defined as the sample mean of final cumulative victories across $N = 100,000$ Monte Carlo trials of the CSD tournament (Section 2.4, Equation 5). Each individual trial produces $S_m(K)$, the number of rounds (out of $K$) that model $m$ wins before either surviving all $K$ rounds or being eliminated. The Expected Win Score is reported directly in all ranking figures. The secondary metric is the Sensitivity Coefficient $\Lambda_m$ (Section 2.5, Equation 6), defined as the empirical slope of $\hat{E}[S_m](T_k)$ over the range $T_k \in \{0, 1, 2\}$, computed as $\Lambda_m = (\hat{E}[S_m \mid T_k = 2] - \hat{E}[S_m \mid T_k = 0]) / 2$. This is reported for the Failure Sensitivity Analysis in Figure 4. For single-benchmark analyses (Section 5.1.2), the paper reports $\hat{E}[S_m]$ as a function of $T_k$ directly (Figure 6).

• Baselines. The paper primarily compares CSD against two implicit baselines: (1) simple average aggregation — computing the mean of all benchmark scores for each model — which is used as a foil in the sensitivity experiments (Section 4.2, Figure 5) to demonstrate CSD's robustness to score perturbations, and (2) Elo-based pairwise ranking as represented by Chatbot Arena (Chiang et al., 2024), discussed conceptually in Section 3. The paper does not implement or directly compare against an Elo system numerically; the comparison is qualitative and architectural (Section 3, "CSD Framework vs. Static Pairwise Ranking (Elo)"). No numeric Elo ratings are computed for the 29 models. The single-benchmark analysis (Section 5.1.2, Figure 6) implicitly compares CSD to the standard single-score-per-benchmark paradigm (accuracy on MMLU-Pro or SuperGPQA), but again without explicit numeric baseline comparisons.

• Generation budget / compute accounting. There is no "generation budget" in the LLM sense, since CSD does not generate model outputs — it simulates tournaments using pre-computed pairwise win rates from the tensor W. The computational cost is measured in terms of (1) the one-time cost of computing W from raw benchmark scores ($\binom{29}{2} \times 12 \approx 4,872$ binary comparisons) and (2) the Monte Carlo simulation cost of $N = 100,000$ independent tournament runs per $T_k$ setting. Each tournament runs SingleInstanceCSD with up to $K = 12$ rounds, involving only tensor lookups and random number generation after W is computed. For the Failure Sensitivity Analysis with three $T_k$ values (0, 1, 2), this totals 300,000 tournament simulations. The paper does not report wall-clock time, but notes that separating W pre-computation from the iterative Monte Carlo sampling "ensures that the iterative computational cost remains relatively low" (Section 2.1.1). For the single-benchmark analysis, questions are partitioned into 10 difficulty tiers (B1: >90% average accuracy; B2: 70–80%; ...; B9: <10%), and the CSD simulation is run with these as sequential rounds.

• Cross-validation / statistical protocol. No cross-validation is used because CSD does not train any model; it is a deterministic simulation procedure (with controlled stochasticity from random pairing and random elimination) applied to fixed pre-computed W tensors. The statistical protocol is the Monte Carlo approximation itself: by running $N = 100,000$ independent trials, the standard error of $\hat{E}[S_m]$ is on the order of $\sigma / \sqrt{100,000} \approx 0.006$ wins (assuming $\sigma \approx 2$), which the paper implicitly treats as sufficient for reliable ranking. No confidence intervals or bootstrap estimates are reported. The paper acknowledges in Section 5.2 that "LLM scores, especially on agent benchmarks, are sensitive to environmental factors and API stability," and therefore "the specific rankings derived via the CSD framework may exhibit fluctuations under different evaluation setups." However, no systematic analysis of this sensitivity (e.g., by perturbing the input W tensor or re-running evaluations) is conducted beyond the score perturbation experiments in Section 4.2.

Main Quantitative Results

Overall Ranking Across 38 Benchmarks

The headline finding is the four-tier ranking structure revealed in Figure 3, which plots $\hat{E}[S_m]$ for all 29 models at three elimination intensities ($T_k = 0, 1, 2$, corresponding to eliminating 0, 1, or 2 models per round). The paper reports the following tier assignments (Section 4.1, Finding 1):

• Tier 1 (top): Gemini-3-pro, GPT-5.1-High, GPT-5-High, and GPT-5-Medium. All four models "exhibit minimal degradation in score as the elimination pressure increases," which the paper interprets as "exceptional robustness and general competence." The paper explicitly notes that these four "demonstrate highly similar performance" (Section 4.1, Finding 1).

• Tier 2: Claude-Sonnet-4.5-thinking and DeepSeek-V3.2-thinking. These two models "exhibit relatively robust performance and general competence" and are "clearly demarcated" from Tier 1.

• Tier 3: Led by Qwen3-Max, "closely followed by a highly competitive group: GLM-4.6, Gemini-2.5-pro, DeepSeek-V3.2-Exp-Thinking, Gemini-2.5-Flash-Preview, and DeepSeek-V3.1-Terminus-Thinking." The paper highlights this cluster as evidence that "Chinese models (GLM, DeepSeek, Qwen) ... are effectively closing the performance gap with Gemini-2.5-Pro (and some have surpassed it), which previously defined the performance frontier (SOTA)."

• Tier 4: Contains "models with suboptimal operational or architectural trade-offs." The paper identifies this cohort as primarily consisting of "previous model iterations (e.g., GLM-4.5), lightweight or efficiency-focused versions (e.g., Gemini-2.5-Flash-Lite, Gemini-2.5-Flash), and base models lacking external 'thinking' or planning mechanisms (e.g., Kimi-K2-0905, DeepSeek-V3.2-Exp-NonThinking)."

No numeric $\hat{E}[S_m]$ values are reported in the text for any specific model; all quantitative information must be read from Figure 3, which shows scores visually on what appears to be a roughly 0–12 scale (consistent with a 12-round tournament where the maximum possible score is 12 wins). The exact values are not tabulated.

Key observations from Figure 3: The four Tier-1 models appear to cluster around $\hat{E}[S_m] \approx 9–10$ wins at $T_k = 0$, with very little visible drop at $T_k = 2$. The spread between tiers is substantial — Tier 4 models appear to achieve roughly 3–5 wins, providing clear separation. The rank ordering is largely preserved across $T_k$ values, though some Tier-3 and Tier-4 models show visible convergence or crossing at higher $T_k$, which the Failure Sensitivity Analysis (Figure 4) examines explicitly.

Failure Sensitivity Analysis: Robust Generalists vs. Aggressive Specialists

Figure 4 plots the relationship between base performance (x-axis: $\hat{E}[S_m \mid T_k = 0]$) and sensitivity coefficient (y-axis: $\Lambda_m = (\hat{E}[S_m \mid T_k = 2] - \hat{E}[S_m \mid T_k = 0]) / 2$). The paper reports the following classification (Section 4.1, Finding 2):

• Robust Generalists ($\Lambda_m \approx 0$): Gemini-3-pro, GPT-5.1-High, GPT-5-High, GPT-5-Medium, Claude-Sonnet-4.5-Thinking, and DeepSeek-V3.2-Thinking. These models combine high base performance with near-zero sensitivity — their scores are stable regardless of elimination pressure because they "rarely, if ever, fall into the $G_{\min}$ group" (Section 2.5).

• Aggressive Specialists ($\Lambda_m \ll 0$): Qwen-3-235B, Qwen-Plus-0728, and Qwen-Next-80B-Thinking. These models achieve high base performance at $T_k = 0$ but experience significant score collapse when elimination pressure increases, exposing fragility on their weaker benchmarks.

The paper does not provide numeric $\Lambda_m$ values in the text; all values must be read from Figure 4's scatter plot. The figure is described as showing that "the three tiers observed in Figure 3 are clear" in this space (Section 4.1, Finding 2), with models clustered by their combination of base performance and sensitivity. The Tier-1 models occupy the upper-right quadrant (high base performance, near-zero sensitivity), while the Aggressive Specialists appear in the lower-right (high base performance, strongly negative sensitivity).

This classification is the paper's primary empirical contribution: it demonstrates that CSD's second dimension ($\Lambda_m$) reveals information about model risk that is invisible in any single-dimensional ranking, including CSD's own ranking at a fixed $T_k$. A practitioner using only the $T_k = 0$ ranking would see Qwen-Plus-0728 and GPT-5.1-High as similarly excellent; the $\Lambda_m$ analysis reveals that they have qualitatively different risk profiles.

Sensitivity to Score Perturbations

Section 4.2 reports two perturbation experiments designed to test CSD's robustness to extreme score outliers.

Example 1: Zero scores on two benchmarks (IFEval and MulDimIF). The scores for Qwen3-Max on IFEval and MulDimIF (both in Round 3, the Basic Instruction Following sub-category) are set to zero, "potentially simulating API errors." This makes Qwen3-Max "the weakest model in the third round" (Section 4.2). The result, shown in Figure 5a:

"this extreme perturbation minimally affects the overall ranking produced by our CSD framework"

In contrast, "if we were to simply aggregate all benchmark results using the average score, the rank of Qwen3-Max would drop to 12th" (Section 4.2, Example 1). No numeric rank change is reported for CSD itself — only the qualitative claim of "minimal" effect and the comparison against simple averaging's 12th-place result.

Example 2: Zero scores on four benchmarks (IFEval, MulDimIF, AIME24, AIME25). The perturbation is expanded to zero scores on four benchmarks, placing Qwen3-Max as "the weakest model in the third round and a significantly weaker model in the sixth" (AIME24 and AIME25 constitute Round 6, the Mathematics sub-category). The result, shown in Figure 5b:

"the rank of Qwen3-Max drops to 10th but remains within the third performance tier"

The simple average baseline "plummets much further, to 19th" (Section 4.2, Example 2). The paper interprets this as "CSD framework's superior stability against widespread score anomalies."

Comparing Examples 1 and 2, the paper notes that "increasing the number of zeroed benchmark scores leads to a more significant, yet controlled, decrease in the model's rank." The mechanism is structural: zeroing early-round scores triggers elimination that prevents Qwen3-Max from accumulating points on its strong later-round benchmarks (like coding and agent tasks), but also limits the damage by preventing the zeros from dragging down a weighted average. The paper is explicit that "the purpose of this sensitivity test is not to validate the framework's behavior under widespread data failure" — for that, "the only necessary step is to re-test all results" — but rather to demonstrate robustness to "isolated, severe score perturbations (e.g., API errors affecting a few benchmarks)" (Section 4.2).

The paper also mentions in passing that "the CSD framework similarly exhibits low sensitivity to extremely high scores on several specific benchmarks," but no experimental results are reported for this direction.

Single-Benchmark CSD Analysis

Section 5.1.2 applies CSD to individual benchmarks (MMLU-Pro and SuperGPQA) by partitioning each benchmark's questions into 10 difficulty tiers based on average model accuracy and treating these tiers as sequential rounds. The workflow:

"Data-Driven Segmentation: Instead of using the datasets separately, we group all questions into ten tiers (B1: Questions with > 90% average accuracy, B2: 70%–80%, ..., B9: < 10%). CSD Input: We use B1, B2, ..., B10 as the sequential input."

Figure 6 shows $\hat{E}[S_m]$ for models on SuperGPQA (Figure 6a) and MMLU-Pro (Figure 6b) at $T_k = 0$ and $T_k = 2$. The paper reports:

"the Kimi models (light green) experience a substantial performance decline on both benchmarks. This suggests that they are less competitive on easier questions compared to harder ones, indicating a lack of robustness across these two benchmarks. Similar cases happen on Claude nonthinking models and GLM models too. In comparison, Gemini-3-pro, GPT-5.1-high, and Claude-Sonnet-4.5-thinking are quite robust on these two QA benchmarks."

No numeric values are reported in the text; all comparisons are visual from Figure 6. The interpretation — that Kimi models perform disproportionately poorly on easy questions relative to hard ones — is notable because a standard accuracy score would treat all questions equally and would not reveal this pattern of "failing easy questions while succeeding on hard ones." CSD's sequential structure (easy tiers first) penalizes this pattern because poor performance on early (easy) rounds triggers elimination or low cumulative scores that cannot be fully recovered by strong later-round performance. This is the same structural principle that distinguishes Robust Generalists from Aggressive Specialists in the cross-benchmark analysis, now applied to within-benchmark difficulty variation.

The paper's expectation framing is explicit: "a model failing on empirically easy questions (low score on B1) but succeeding on complex ones (high score on B3) suggests some instability or unreliable reasoning, possibly due to stochasticity or data contamination. By applying the CSD mechanism, such models will incur a severe performance drop ($\Delta E[S_m]$), resulting in a lower final rank" (Section 5.1.2). This is presented as a feature of CSD rather than a bug — it "inherently favors robust generalists over models that exhibit large performance variance" within a single benchmark.

Ablation Studies and Robustness Checks

The paper contains limited systematic ablation studies in the traditional sense (varying one design choice while holding others constant). However, several analyses function as implicit ablations:

Elimination intensity ($T_k$) variation: This is the central ablation, varying the penalty-for-failure parameter across $T_k \in \{0, 1, 2\}$. The results appear in Figure 3 (overall ranking stability across $T_k$), Figure 4 (sensitivity coefficient computation), and Figure 6 (single-benchmark sensitivity). Models are observed to fall into two categories — those whose rank is stable across $T_k$ and those whose rank degrades — validating that $T_k$ is not merely a monotonic scaling factor but reveals heterogeneous model behavior. Critically, the paper uses only $T_k = 0$ and $T_k = 2$ for the sensitivity coefficient calculation, with $T_k = 1$ shown in figures but not used in the $\Lambda_m$ formula. The choice to use a two-point slope rather than a regression over all three points is not justified.

Score perturbation magnitude: Examples 1 and 2 (Section 4.2) ablate the number of zeroed benchmarks (2 vs. 4), showing that CSD rank degradation is "controlled" — scaling sub-linearly with the number of perturbations, unlike simple averaging where degradation is roughly proportional. However, only two perturbation magnitudes are tested, both on the same model (Qwen3-Max), and only in the zero-score direction. There is no systematic sweep of perturbation magnitudes, no testing on other models, and no testing of positive perturbations (inflated scores).

Single-benchmark vs. multi-benchmark CSD: Sections 4.1 (38 benchmarks) and 5.1.2 (single benchmarks with difficulty tiers) demonstrate CSD operating at two different granularities — cross-benchmark and within-benchmark. Both reveal similar patterns (Robust Generalists vs. models with sensitivity to early-round performance). This is not a controlled ablation but a demonstration of generality across aggregation scales.

Thinking vs. non-thinking variants: The 29-model set includes both "thinking" and "nothinking" variants from the same model families (e.g., DeepSeek-V3.1-Terminus-nothinking vs. DeepSeek-V3.1-Terminus-thinking; Claude-Sonnet-4.5-nothinking vs. Claude-Sonnet-4.5-thinking). Figure 3 shows that thinking variants consistently rank above their non-thinking counterparts, with Claude-Sonnet-4.5-thinking in Tier 2 while Claude-Sonnet-4.5-nothinking falls in Tier 4, and DeepSeek-V3.2-thinking in Tier 2 while DeepSeek-V3.2-Exp-NonThinking falls in Tier 4. This pattern is not explicitly analyzed but serves as a validation that CSD rankings align with known capability improvements from reasoning enhancements.

Absent ablations that would strengthen the paper:

• Sequence order: The 12-round sequence in Table 1 follows a specific "general-to-specific, fundamental-to-challenging" rationale, but no alternative sequences are tested. The paper's central claim — that sequence-as-weight replaces subjective weighting with structural importance — implicitly requires that the specific sequence matters. If CSD rankings were invariant to sequence order, the framework would lose its claimed advantage over static methods. But invariance is never tested. A natural ablation would be reversing the sequence (hard-to-easy), randomizing it, or placing coding before reasoning, and observing how rankings change.
• Number of rounds (K): The paper uses exactly 12 rounds corresponding to 12 sub-categories. No analysis of how rankings change with fewer or more rounds is provided.
• Elimination target (minimum score group vs. alternatives): The paper eliminates from $G_{\min}(k)$ only. An alternative would be eliminating from other score groups or using a probabilistic elimination function that is not strictly tied to being the worst. No such comparison is made.
• Monte Carlo trial count (N): $N = 100,000$ is used throughout without justification of why this particular number or evidence that smaller $N$ produces comparable rankings. A convergence analysis (plotting $\hat{E}[S_m]$ stability as a function of $N$) would establish whether the computational investment is necessary.
• Pairing mechanism: The Swiss-System is asserted to be superior to random pairing, but no experiment compares CSD rankings under Swiss-System pairing vs. random pairing vs. round-robin. This is a significant gap given that the pairing mechanism is one of the framework's three core components.
• Binary vs. score-margin win/loss: The W tensor uses binary outcomes (win/loss), discarding performance margin. No comparison to a version of CSD using continuous win probabilities or score differences is reported.

Critical Assessment

Does CSD actually solve the subjective weighting problem, or does it relocate it?

The paper's central claim is that CSD "intrinsically solves the subjective weighting problem plaguing aggregated leaderboards" (Section 7) by replacing researcher-chosen weights with structural importance that emerges from benchmark sequence and elimination dynamics. The experiments provide evidence that CSD rankings are robust to score perturbations (Section 4.2), which is consistent with the claim that structure provides stability that ad-hoc weights cannot. However, the experiments do not test whether CSD rankings are robust to the new subjective input it introduces: the benchmark sequence itself.

The sequence choice in Table 1 — foundational knowledge → instruction following → reasoning → coding → agent capabilities → factuality — is explicitly motivated by a "two-part rationale: moving from general to specific coverage, and progressing from fundamental to challenging tasks." This is a defensible ordering, but it is not the only possible ordering, and the paper provides no evidence that alternative orderings would produce similar rankings. If a practitioner believes that coding capability is more foundational than reasoning (a plausible perspective for software engineering deployments) and reorders the sequence accordingly, would the rankings change substantially? The paper cannot answer this because no sequence ablation is run.

The claim that the weight of a benchmark "emerges from the tournament dynamics" is true in a mechanical sense — early benchmarks have higher structural importance because failure there triggers early elimination — but this is a direct consequence of the researcher's sequence choice, not an independent property of the benchmarks. The structural importance of IFEval in the CSD framework is high not because IFEval is objectively more important than AIME24, but because the researcher placed it in Round 3 rather than Round 6. The paper's framing — that CSD replaces subjective weighting with "structural importance" — is therefore somewhat misleading. It replaces "how important is math relative to coding?" with "should math come before or after coding in the sequence?" The latter question may be more answerable (as the paper argues), but it remains a subjective choice that the evaluator must justify. The experiments do not demonstrate that this choice is less consequential than weighting choices would be.

This does not invalidate CSD — a framework that requires a defensible sequence is still an advance over one that requires defensible weights, because sequence maps to concrete deployment dependencies — but it means the claim of having "solved" the subjective weighting problem is overstated. The paper has replaced one form of subjective input with another, and the experiments do not quantify how sensitive the output is to this input.

How well-supported is the Robust Generalist vs. Aggressive Specialist classification?

The Failure Sensitivity Analysis (Figure 4) is the paper's most distinctive empirical contribution. The identification of Gemini-3-pro, GPT-5.1-High, GPT-5-High, GPT-5-Medium, Claude-Sonnet-4.5-Thinking, and DeepSeek-V3.2-Thinking as Robust Generalists, and Qwen-3-235B, Qwen-Plus-0728, and Qwen-Next-80B-Thinking as Aggressive Specialists, is based on the sensitivity coefficient $\Lambda_m$ computed over the limited range $T_k \in \{0, 2\}$.

Several concerns arise:

The range of $T_k$ is narrow and arbitrary. The paper uses $T_k \in \{0, 1, 2\}$. At $K = 12$ rounds, eliminating 2 models per round removes at most 24 models over the full tournament — which, with 29 starting models, means the tournament could still complete all 12 rounds. But the choice of $T_k = 2$ as the "high elimination pressure" endpoint is not motivated by any deployment consideration. Would the classification change at $T_k = 3$ or $T_k = 4$? The paper does not test this. The sensitivity coefficient is essentially a two-point slope estimate, and its stability across a broader range of $T_k$ values is unknown.

The $\Lambda_m$ values are not reported numerically. All classification must be inferred from Figure 4's scatter plot, which lacks grid lines and axis tick labels that would enable precise reading. The paper states which models are Robust Generalists and which are Aggressive Specialists, but does not provide the numeric thresholds used for these categories. Is there a natural gap in the $\Lambda_m$ distribution, or is the classification based on a somewhat arbitrary cutoff? The scatter plot suggests a continuum rather than discrete clusters, which would imply that the Robust Generalist / Aggressive Specialist binary is an interpretive overlay rather than a statistically grounded taxonomy.

Only 29 models are analyzed. The entire classification is based on one set of 29 commercially available models. There is no evidence that the patterns observed (clustering into four tiers, separation into Robust Generalist / Aggressive Specialist) would replicate on a different model set or at a different point in time as models evolve. The paper's acknowledgment that "LLM scores ... are sensitive to environmental factors and API stability" (Section 6) and that "the specific rankings ... may exhibit fluctuations under different evaluation setups" implicitly undermines the stability of the classification, but no quantification of this sensitivity is provided.

The identification of Aggressive Specialists as specifically Qwen models raises a question that the paper does not address: are these models genuinely "aggressive specialists" (implying a tradeoff where high peak performance on some benchmarks comes at the cost of catastrophic weakness on others), or are they experiencing benchmark-specific evaluation artifacts (API errors, prompt sensitivity, evaluation setup differences) that the perturbation experiments only partially control for? The score perturbation experiments (Section 4.2) demonstrate CSD's robustness to simulated failures, but they do not investigate whether the observed sensitivity of Qwen models in Figure 4 reflects genuine capability gaps or measurement noise in the underlying evaluation data. This distinction matters for deployment: one wants to avoid models with real capability gaps, but one doesn't want to penalize models for noisy evaluation data.

Do the score perturbation experiments demonstrate what they claim?

The perturbation experiments (Examples 1 and 2) are presented as evidence of CSD's "superior robustness to outlier data" (Section 4.2). However, these experiments test a narrow and somewhat artificial scenario:

• Only one model is perturbed. Qwen3-Max is the sole target of both perturbation experiments. There is no demonstration that CSD is robust to perturbations on any model, or that the degree of robustness is uniform across models. A model at the boundary between tiers might be more sensitive to perturbations than a model securely in the middle of a tier.

• Only zero-score perturbations are tested. The experiments simulate "API errors" that produce scores of zero. But real measurement errors are more likely to produce scores that are lower (but not zero) or higher (overestimation) than ground truth. The paper mentions in passing that CSD "similarly exhibits low sensitivity to extremely high scores," but no experiment supports this.

• The perturbation magnitude is extreme. Setting a score to zero is the most extreme possible perturbation — it makes the model the absolute worst on that benchmark. Real API errors or evaluation artifacts are more likely to produce scores that are anomalously low but not catastrophically so. The experiments demonstrate that CSD handles the worst case well, but do not characterize behavior under more realistic, moderate perturbations.

• The "simple average" baseline is a straw man. The paper compares CSD against simple averaging, which is known to be sensitive to outliers. A more informative comparison would be against robust aggregation methods — median, trimmed mean, or weighted averaging with outlier detection — which are standard in practice. The paper does not claim that practitioners literally compute unweighted means of all benchmark scores; it critiques the principle of aggregation. But the perturbation experiment's rhetorical force depends on the contrast with a naive baseline that few serious evaluation pipelines would use.

More fundamentally, the perturbation experiments test a scenario (isolated measurement errors on a few benchmarks) that the paper's own motivation suggests is not the primary problem CSD solves. The motivation emphasizes sequential dependencies and the need to penalize catastrophic failure on foundational capabilities — but in the perturbation experiments, the zero scores represent measurement artifacts, not genuine model failures. CSD handles them similarly (eliminating the model early, preventing it from accumulating later-round wins), but the interpretation differs: in the motivation, early elimination is a desirable penalty for genuine capability gaps; in the perturbation experiment, early elimination is an unfortunate consequence of bad data that CSD fortunately limits. The experiments do not disentangle these two interpretations.

Are the single-benchmark results (Section 5.1.2) convincing?

The application of CSD to single benchmarks by partitioning questions into difficulty tiers is an interesting extension, but the experimental support is thin. Figure 6 shows results for only two benchmarks (SuperGPQA and MMLU-Pro), both QA datasets. The paper claims that Kimi models "experience a substantial performance decline" and that this "suggests that they are less competitive on easier questions compared to harder ones." However:

The difficulty tier boundaries are empirical, not intrinsic. The tiers are defined by average model accuracy across all 29 models: B1 has >90% average accuracy, B2: 70–80%, ..., B9: <10%. This means a question's tier assignment depends on which models are in the evaluation set. Add or remove models, and the tier boundaries shift. This is reasonable for the within-benchmark CSD simulation, but it means the claim that a model "fails easy questions" is relative to the aggregate performance of the specific 29-model set, not an absolute statement about question difficulty.

No comparison to baseline ranking methods is provided. The paper does not report what a standard accuracy-based ranking on MMLU-Pro or SuperGPQA would produce, nor does it quantify how much the CSD-based ranking differs. The claim that CSD "provides a more nuanced and robust ranking" (Section 5.1.2) is therefore asserted rather than demonstrated — there is no evidence that the CSD ranking captures something the standard ranking misses, or that the standard ranking would give substantively different model selection guidance.

The sample size is generous (full benchmark question sets), but only two benchmarks are tested. The paper does not explain why MMLU-Pro and SuperGPQA were chosen, whether the pattern observed (Kimi models underperforming on easy questions) generalizes to other benchmarks, or whether models other than Kimi, Claude nonthinking, and GLM show interesting within-benchmark sensitivity patterns. The analysis reads as an existence proof (CSD can be applied to single benchmarks) rather than a systematic evaluation.

The absence of any external validation

The paper is candid that CSD rankings cannot be validated against a "ground-truth ranking" because "the 'true' ranking of generalist LLMs remains subjective and context-dependent" (Section 5.2). This is intellectually honest, but it means that all of CSD's claimed advantages — better alignment with deployment reality, more informative risk profiling, robustness to perturbations — are supported only by internal consistency arguments and the perturbation experiments. The paper does not:

• Correlate CSD rankings with any external measure of model utility (e.g., user preference data from production deployments, downstream task success rates, or even alternative ranking systems like Chatbot Arena Elo).
• Demonstrate that CSD's Robust Generalist classification predicts actual deployment outcomes (lower failure rates, fewer pipeline blockages) compared to Aggressive Specialists with similar average performance.
• Show that practitioners making model selection decisions based on CSD rankings would make systematically better choices than those using existing leaderboards.

The paper acknowledges this limitation explicitly (Section 5.2: "it is challenging to perform a definitive external validation of the CSD ranking against an objective 'best' list"), but the absence of external validation means that the paper's claims — while internally coherent — remain unverified against any external criterion. The ranking "is primarily validated through its internal consistency and its superior robustness demonstrated in the sensitivity analysis, rather than by external correlation with an undisputed standard." Whether CSD's notion of "competitive robustness" actually corresponds to deployment success is an open question that the experiments do not address.

The agentic performance prediction claim is entirely untested

Section 5.1.1 proposes that CSD rankings could "predict" performance on novel agentic tasks by sequencing benchmarks to mirror the task's capability dependencies. Example 3 sketches a mapping for web navigation and data extraction: IFEval → GSM8K → ToolBench → HumanEval. The paper presents this as an "interesting and important extension direction" (Section 5.1.1) but provides zero experimental evidence that such predictions work. No agentic task is evaluated, no CSD-predicted ranking is compared against actual task performance, and no correlation is reported. The section is entirely prospective — it describes what could be done, not what was done. This is not a weakness of the experiments per se (the paper does not claim to have tested this), but it means that one of the paper's most provocative claims — that CSD can serve as a "surrogate for pipeline evaluation" — has no experimental support whatsoever.

6. Limitations and Trade-offs

The Benchmark Sequence Replaces One Subjective Input with Another — and Sequence Sensitivity Is Untested

The assumption or constraint. CSD's central claim — that it "intrinsically solves the subjective weighting problem plaguing aggregated leaderboards" (Section 7) — rests on the premise that benchmark importance emerges from sequential position rather than from researcher-chosen coefficients. However, this means the evaluator must choose the sequence order of benchmarks ($d_1, d_2, \ldots, d_K$). The paper explicitly acknowledges this: the 12-round sequence in Table 1 follows a "two-part rationale: moving from general to specific coverage, and progressing from fundamental to challenging tasks" (Section 4.1). This is a defensible ordering, but it is not the only possible ordering, and the framework provides no mechanism for determining the "correct" sequence — any more than static aggregation provides a mechanism for determining the "correct" weights.

The consequence. The structural importance of early-round benchmarks is a direct mechanical consequence of the evaluator's sequence choice, not an objective property of the benchmarks themselves. IFEval has high structural importance in the paper's CSD ranking not because instruction following is inherently more important than mathematics for all deployment contexts, but because the researcher placed it in Round 3 rather than Round 6. A practitioner deploying an LLM for a code-generation pipeline — where instruction following might be less critical than algorithmic reasoning — would likely want a different sequencing, and that different sequencing would produce a different CSD ranking. The framework provides no guidance on how sensitive rankings are to sequence perturbations.

The paper has essentially traded one subjective choice (how to weight benchmarks) for another (in what order to sequence them). The claim that sequence choice is "more answerable" than weight choice because it maps to "concrete deployment dependencies" (Section 2.1.1) is plausible but unverified: there is no evidence that practitioners agree more on sequence than on weights, or that sequence disagreements produce less ranking variance than weight disagreements.

What evidence exists in the paper. None. The paper conducts no sequence ablation. It never tests whether reversing the order (hard-to-easy), randomizing it, placing coding before reasoning, or any alternative sequence would produce substantially different rankings. This is perhaps the single most important missing experiment in the paper: if CSD rankings are largely invariant to sequence choice, then the entire "sequence-as-weight" principle — the paper's core conceptual contribution — is empirically vacuous. If rankings are highly sensitive to sequence, then the evaluator's sequence choice carries exactly the same burden of justification that weight choices carry in static aggregation.

The sensitivity experiments in Section 4.2 perturb scores (zeroing benchmarks for one model) but leave the sequence unchanged. They demonstrate that CSD is robust to measurement error given a fixed sequence, but they say nothing about robustness to sequence specification error. These are fundamentally different forms of sensitivity, and only the second addresses whether CSD solves the arbitrary weighting problem or merely relocates it.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not propose methods for validating sequence choices, and does not suggest sequence sensitivity analysis as future work. The conclusion mentions "the formal integration of sequential dependencies for agentic performance prediction" (Section 7) as a future direction, but this is about designing sequences for specific tasks, not about measuring how rankings vary under sequence perturbations. A natural mitigation — reporting CSD rankings under multiple plausible sequences and showing that qualitative conclusions (tier assignments, Robust Generalist vs. Aggressive Specialist classifications) are stable — is entirely absent.

---

No External Validation Against Any Deployment Outcome or Alternative Ranking System

The assumption or constraint. CSD rankings are presented as providing "a more reliable metric for selecting LLMs destined for complex, multi-stage deployment" (Section 7). However, the paper provides no evidence that CSD rankings correlate with any external measure of model quality. The authors are transparent about this:

"the 'true' ranking of generalist LLMs remains subjective and context-dependent" (Section 5.2)

"it is challenging to perform a definitive external validation of the CSD ranking against an objective 'best' list. The ranking is primarily validated through its internal consistency and its superior robustness demonstrated in the sensitivity analysis, rather than by external correlation with an undisputed standard." (Section 5.2)

The consequence. All of CSD's claimed advantages — better alignment with deployment reality, more informative risk profiling, the Robust Generalist vs. Aggressive Specialist distinction — are supported solely by internal arguments (the mathematical structure of the framework) and perturbation experiments (showing that rankings are stable when individual scores are corrupted). There is no demonstration that:

• A model classified as a Robust Generalist by CSD actually experiences fewer catastrophic failures in production pipelines than a model classified as an Aggressive Specialist with similar base performance.
• CSD's ranking order (Gemini-3-pro > GPT-5.1-High > Claude-Sonnet-4.5-thinking > Qwen3-Max > ...) better predicts downstream task success rates than, say, Chatbot Arena Elo ratings or simple average accuracy.
• A practitioner who selects models based on CSD tiers would make systematically better deployment decisions than one who uses existing leaderboards.

This absence of external validation is not merely a missing experiment — it means the paper's core claim (that CSD provides a better ranking for deployment) is an untested hypothesis. The framework is internally coherent and mathematically well-specified, but whether its notion of "competitive robustness under sequential elimination" actually corresponds to real-world deployment success is entirely unknown. The sensitivity experiments demonstrate that CSD rankings are robust to measurement noise, but robustness to noise does not imply accuracy — a ranking can be stable and wrong.

What evidence exists in the paper. Section 5.2 acknowledges this limitation explicitly and at length, which is to the authors' credit. However, the paper provides no quantitative evidence that even begins to bridge this gap. There is no correlation analysis with Chatbot Arena Elo scores (despite Elo being discussed extensively in Section 3 as the primary pairwise alternative), no comparison with production deployment outcomes from any of the organizations whose models are ranked, and no user study where practitioners rank models and CSD's ranking is compared to their judgments. The single-benchmark analysis (Section 5.1.2, Figure 6) provides within-benchmark validation that some models perform disproportionately poorly on easy questions, but this is an observation about model behavior, not validation that CSD's ranking of those models is correct in any external sense.

Mitigation status. The authors are transparent about the limitation but offer no mitigation beyond conceptual argument. They note that "the primary goal of CSD is to penalize weaknesses and reward resilience" and that "our ranking may deviate from baselines that prioritize raw peak performance," and characterize this as "a deliberate methodological choice rather than an error" (Section 5.2). This is a fair defense of the design, but it does not address whether the design achieves its stated goal of better deployment selection. The paper does not propose external validation studies as future work — the conclusion focuses on "agentic performance prediction" and "correlation between CSD rankings and real-world task failure rates" (Section 7), acknowledging the gap but not committing to closing it.

---

The Failure Sensitivity Analysis Uses Only 2–3 Elimination Levels with No Justification, and Classification Thresholds Are Not Defined

The assumption or constraint. The Failure Sensitivity Analysis, which the paper presents as its most distinctive contribution ("a diagnostic capability entirely absent in standard Elo systems," Section 3), computes the sensitivity coefficient $\Lambda_m$ as a two-point slope over the range $T_k \in \{0, 2\}$:

$\Lambda_m \approx \frac{\hat{E}[S_m \mid T_k = 2] - \hat{E}[S_m \mid T_k = 0]}{2}$

The paper then classifies models into Robust Generalists ("near-zero slope") and Aggressive Specialists ("highly negative slope") based on this coefficient. The choice of $T_k = 2$ as the upper endpoint, the use of exactly two points for the slope estimate, and the thresholds for classification are all presented without systematic justification.

The consequence. Several issues cascade from this underspecification:

1. The slope may be non-linear. With only three $T_k$ values tested ($0, 1, 2$) and only two used in the slope calculation, the paper cannot distinguish between a model whose expected score drops linearly with $T_k$ and one whose score is stable at $T_k = 1$ but collapses sharply at $T_k = 2$. Figure 3 shows $\hat{E}[S_m]$ at all three $T_k$ values, and some models appear to show non-linear degradation (relatively stable from $T_k = 0$ to $T_k = 1$, then dropping at $T_k = 2$), but this is not analyzed.

2. The elimination intensity range is narrow and arbitrary. At $K = 12$ rounds with 29 starting models, eliminating $T_k = 2$ models per round removes at most 24 models — the tournament still completes all 12 rounds for the surviving 5 models. $T_k = 2$ is therefore not a "high-pressure" regime in any absolute sense; it merely applies more pressure than $T_k = 0$. A deployment where failure on any task is catastrophic might correspond to a much higher equivalent $T_k$ (e.g., eliminating all models in $G_{\min}(k)$ regardless of group size). The paper does not explore whether the Robust Generalist classification holds at higher elimination intensities.

3. Classification is based on visual inspection of a scatter plot. The paper states which models are Robust Generalists and which are Aggressive Specialists (Section 4.1, Finding 2), but provides no numeric $\Lambda_m$ values and no classification threshold. Figure 4 shows a scatter plot without grid lines or precise axis labels, making it impossible to determine exactly where the boundary between categories lies. The distribution appears continuous rather than bimodal, suggesting that the Robust Generalist / Aggressive Specialist distinction is an interpretive overlay rather than a statistically grounded clustering. A model with $\Lambda_m = -0.3$ and one with $\Lambda_m = -0.5$ might both be called "Aggressive Specialists," but their risk profiles could be meaningfully different.

4. Sensitivity coefficient comparability across models is assumed but not established. $\Lambda_m$ is an absolute slope (expected wins per unit $T_k$), but models have different base performance levels. A drop of 0.5 wins per $T_k$ unit means something very different for a model with $\hat{E}[S_m \mid T_k = 0] = 10$ (5% relative degradation) versus one with $\hat{E}[S_m \mid T_k = 0] = 3$ (17% relative degradation). The paper does not report relative sensitivity or normalize $\Lambda_m$ by base performance, which could change which models appear most sensitive.

What evidence exists in the paper. Figure 3 (overall ranking at $T_k = 0, 1, 2$) and Figure 4 (base performance vs. sensitivity coefficient) contain the relevant data, but the analysis is primarily qualitative. The paper reports the $\Lambda_m$ calculation formula (Section 4, Equation 7) and names specific models as Robust Generalists and Aggressive Specialists, but does not report numeric $\Lambda_m$ values, confidence intervals on the slope estimate, or any formal test for whether a model's $\Lambda_m$ is significantly different from zero. The Monte Carlo simulation with $N = 100,000$ provides precision on $\hat{E}[S_m]$ at each $T_k$ (standard error ~0.006 wins), which in principle enables statistical testing of whether $\Lambda_m$ differs from zero, but no such test is performed.

Mitigation status. Partial. The paper demonstrates that $\Lambda_m$ varies substantially across models and that this variation is interpretable (Section 4.1, Finding 2), which is sufficient to establish the existence of the phenomenon. However, the paper does not provide the tools a practitioner would need to use FSA confidently — numeric sensitivity scores with uncertainty estimates, validated classification thresholds, or evidence that the classification is stable under broader $T_k$ ranges. The authors do not acknowledge the narrow $T_k$ range or the classification threshold problem as limitations.

---

The Pairwise Win-Rate Tensor Discards Performance Magnitude, and the Binary Abstraction Is Never Validated

The assumption or constraint. The entire CSD framework operates on the Pairwise Win-rate Tensor $W(i, j, k) \in \{0, 1\}$, which binarizes all model comparisons: model $i$ either beats model $j$ on benchmark $d_k$ or loses. The paper argues this is a feature, not a bug, because it "inherently bypasses the critical and subjective problem of benchmark weighting" (Section 2.1.1) — a win is a win regardless of margin, so there is no need to calibrate score magnitudes across benchmarks.

The consequence. The binarization discards all information about the magnitude of performance differences. Two scenarios produce identical $W(i, j, k) = 1$ entries:

• Model $i$ scores 95% accuracy and model $j$ scores 94% accuracy (a narrow win).
• Model $i$ scores 95% accuracy and model $j$ scores 12% accuracy (a blowout).

In the CSD framework, these are treated identically. This has several consequences:

1. CSD cannot distinguish between closely-matched models and models with large capability gaps. In a score group where all models have similar raw performance, the binary win/loss outcomes will be near 50% for all pairs, and CSD's pairing stochasticity will dominate — the ranking within that group will be largely random. In a score group with clear performance stratification, binary outcomes will be near 0% or 100%, and the ranking will be stable. CSD provides no signal about when it is in the first regime versus the second, because the W tensor's binary entries look the same in both cases (just with different underlying win probabilities).

2. The Expected Win Score may conflate narrow wins against strong opponents with blowout wins against weak opponents. A model that consistently ekes out 51% win rates against Tier 1 opponents could accumulate the same $E[S_m]$ as a model that dominates Tier 3 opponents with 90% win rates but loses to Tier 1 opponents. CSD's Swiss-System pairing partially mitigates this by matching models within score groups (so Tier 1 models mostly face Tier 1 opponents), but to the extent that score groups mix models with different true capability levels, the binary abstraction obscures the difference between narrow and decisive victories.

3. The binary abstraction may amplify noise in the underlying evaluation data. If two models have nearly identical raw scores on a benchmark, the binary win/loss outcome can flip based on a single-question difference — a highly unstable signal. The Monte Carlo simulation with $N = 100,000$ trials averages over pairing stochasticity but does not average over the binary threshold: the W tensor is fixed once computed, and any noise in the raw scores that happens to flip a binary outcome becomes a permanent feature of the tournament simulation. The paper's sensitivity experiments (Section 4.2) address extreme score perturbations (zeroing entire benchmarks) but do not address boundary perturbations (small score changes near the win/loss threshold).

What evidence exists in the paper. None. The paper does not:

• Compare CSD rankings using binary W against a version using continuous win probabilities (e.g., $W(i,j,k)$ as the actual win rate rather than its thresholded binary version).
• Report how many of the 4,872 binary comparisons in the tensor are "close calls" (near 50% win rate) versus decisive (near 0% or 100%), which would indicate how much information the binarization discards.
• Analyze whether the Robust Generalist / Aggressive Specialist classification is sensitive to the binarization — for instance, whether an Aggressive Specialist's sensitivity arises from genuinely catastrophic failures on some benchmarks or from being narrowly on the losing side of many close comparisons.

The paper also does not discuss the possibility of ties. In many benchmarks, two models may achieve identical scores — a genuine tie that the binary W tensor cannot represent (it forces a win for one model and a loss for the other). The paper's W construction presumably breaks ties arbitrarily (or uses some deterministic tie-breaking rule), but this is never stated. In benchmarks where many models achieve ceiling performance (common for saturated benchmarks), arbitrary tie-breaking could introduce substantial noise into the W tensor that propagates through the CSD simulation.

Mitigation status. Not addressed. The paper presents binarization as a principled design choice that solves the weighting problem (Section 2.1.1), but does not acknowledge the information loss or test whether it matters empirically. A natural mitigation — computing CSD with both binary and continuous-probability W tensors and comparing the resulting rankings — is not performed. The paper also does not discuss the sensitivity of the W tensor to tie-breaking rules or near-threshold score differences.

---

The Agentic Performance Prediction Claim Has No Experimental Support

The assumption or constraint. Section 5.1.1 proposes that CSD could "predict" model rankings on novel agentic tasks by sequencing existing benchmarks to mirror the task's capability dependencies. The paper presents Example 3 (Web Navigation and Data Extraction) with a specific mapping: IFEval → GSM8K → ToolBench → HumanEval. The text states:

"if we set the corresponding sequence of benchmarks, the overall ranking given by our CSD framework could reflect (or 'predict') the ranking on the new agentic task."

The consequence. This claim — that CSD can serve as a surrogate for task-specific evaluation without requiring any data from the target task — is among the most provocative in the paper, but it is presented with zero experimental evidence. No agentic task is evaluated. No CSD-predicted ranking is compared against actual task performance. No correlation between CSD rankings and any agentic benchmark (even existing ones like GAIA, HLE, or BrowseComp, which are already in the 38-benchmark suite) is reported.

This matters because the claim, if true, would represent a paradigm shift in LLM evaluation: instead of building new benchmarks for every new capability, evaluators could decompose tasks into constituent capabilities, map them to existing benchmarks, and compute CSD rankings as predictions. But if the claim is false — if CSD rankings on sequenced benchmarks do not correlate with actual agentic task performance — then the framework's utility is limited to the benchmarks it explicitly includes, and the "agentic performance prediction" framing is misleading.

The paper itself identifies a "significant caveat" that partially undermines even the conceptual motivation:

"since many recent advanced LLMs inevitably target or optimize for performance on open-source agentic benchmarks during their development, making it more challenging to directly apply the CSD framework to 'predict' performance on novel agentic tasks. This occurs because the input benchmarks themselves may be subject to data contamination or overfitting." (Section 5.1.1)

This caveat acknowledges a fundamental obstacle — if models are trained on the very benchmarks used as CSD inputs, then CSD rankings may reflect training set memorization rather than generalizable sequential competence — but it does not address the more basic question of whether the prediction would work even without contamination.

What evidence exists in the paper. None whatsoever. Section 5.1.1 is entirely prospective. It describes a conceptual mapping (Example 3), asserts that CSD rankings "could reflect" agentic task rankings, and flags data contamination as a concern. There is no experiment, no correlation analysis, and no comparison between CSD-predicted rankings and actual agentic task performance. The paper's own agentic benchmarks (GAIA, HLE, BrowseComp, BrowseComp-zh, τ²-bench) appear in the 38-benchmark suite (Table 1, Rounds 10–11), so in principle the paper could have tested whether CSD rankings computed without those benchmarks predict rankings on those benchmarks — a leave-one-out prediction experiment. This is not attempted.

Mitigation status. The paper explicitly labels this as an "extended application" and a direction for "future work" (Sections 5.1.1, 7). The claim is hedged with "could reflect (or 'predict')" and the caveat about data contamination. However, the paper does not acknowledge the complete absence of experimental support for what is presented as one of the framework's key extended applications. A reader who encounters the agentic prediction framing in Section 5.1.1 might reasonably assume some validation exists; the paper should state more clearly that this is entirely speculative.

---

Difficulty Estimation Within Single Benchmarks Uses Circular Empirical Tiering with No Robustness Check

The assumption or constraint. Section 5.1.2 applies CSD to individual benchmarks by partitioning questions into difficulty tiers based on average model accuracy across the 29 evaluated models. Concretely:

"we group all questions into ten tiers (B1: Questions with > 90% average accuracy, B2: 70%–80%, ..., B9: < 10%)"

The tier boundaries are thus empirically defined relative to the specific set of 29 models being evaluated, not by any intrinsic property of the questions.

The consequence. This creates a form of circularity: the "difficulty" of a question is defined by how the 29 specific models perform on it, and then those same models are evaluated on their ability to handle "easy" versus "hard" questions as defined by their own collective performance. This is not necessarily invalid — it is a form of item response theory where difficulty is estimated from the sample — but it means:

1. The difficulty tiers are not portable. If a different set of models were evaluated, the tier assignments would shift. A question that 90% of the current 29 models answer correctly might be answered correctly by only 50% of a weaker model set, moving it from B1 to B4. The claim that Kimi models "are less competitive on easier questions compared to harder ones" (Section 5.1.2) is relative to the difficulty-as-defined-by-this-specific-model-set, not to an absolute notion of question difficulty.

2. Ceiling and floor effects are unaddressed. The "easiest" tier (B1: >90% average accuracy) contains questions that nearly all models answer correctly. On these questions, the binary win/loss outcomes in the W tensor will be near-random (close to 50% for all pairs), because any model that also answers correctly will tie with any other model that answers correctly. The CSD simulation on B1 may therefore be dominated by the arbitrary tie-breaking in the W tensor construction, making the apparent "poor performance" of some models on easy questions an artifact of how ties are resolved rather than a genuine capability signal.

3. The tier boundaries (e.g., 90%, 70–80%) appear ad-hoc. The paper does not explain why these specific thresholds were chosen, whether results are sensitive to the number of tiers (10) or the boundary placements, or whether alternative segmentation schemes (e.g., equal-size bins, standard-deviation-based binning) would produce different conclusions.

What evidence exists in the paper. Figure 6 shows results for two benchmarks (SuperGPQA and MMLU-Pro) using the 10-tier scheme. The paper reports that Kimi, Claude nonthinking, and GLM models show "substantial performance decline" under elimination pressure while Gemini-3-pro, GPT-5.1-high, and Claude-Sonnet-4.5-thinking are "quite robust." However, the paper does not:

• Report what fraction of questions fall into each tier.
• Analyze whether the "performance decline" is concentrated in specific tiers (e.g., are Kimi models failing on B1 questions that everyone else answers correctly, or on B2–B3 questions where meaningful differentiation exists?).
• Compare against a baseline ranking method (e.g., simple accuracy) to demonstrate that CSD adds information beyond what a standard score would show.
• Test alternative tiering schemes to assess robustness.

Mitigation status. Not addressed. The paper presents the single-benchmark application as a demonstration of CSD's flexibility (Section 5.1.2) rather than as a fully validated method, but it does not discuss the circularity of empirical tiering, the tie-breaking issue in high-accuracy tiers, or the sensitivity to tier boundary choices. A natural mitigation — comparing CSD rankings with tiers defined by human expert difficulty ratings (if available) or by item-level properties (question length, required reasoning steps) rather than by model accuracy — is not considered.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a genuinely new evaluation paradigm — competitive tournament dynamics as a diagnostic framework — rather than an incremental improvement to existing aggregation methods. The shift is not "here is a better weighting scheme for your leaderboard" but "the question you are asking — what weighted sum best captures model quality? — is the wrong question for deployment decisions." The paper argues that deployment-relevant evaluation must model path dependency and elimination pressure, not just average capability, and CSD provides the first concrete framework that does this at scale.

The magnitude of this shift is best understood by comparing what CSD enables versus what prior methods could do:

Prior to CSD: An evaluator with 38 benchmark scores for 29 models had two choices. They could produce a one-dimensional ranking (average, Elo, Bradley-Terry) that collapses all variation into a single number, silently treating a model with catastrophic weakness on one benchmark as equivalent to a consistently competent model if their averages coincide. Or they could manually inspect 38 numbers per model and attempt to form a holistic judgment — a cognitively impossible task at scale. Neither approach provided a principled way to answer the deployment-critical question: "if I put this model into a sequential pipeline where early failure blocks later stages, which model should I choose?"

After CSD: The same evaluator can compute not just a ranking but a risk profile for each model — the $\hat{E}[S_m](T_k)$ function and its derivative $\Lambda_m$ — that explicitly quantifies vulnerability to catastrophic failure. The Robust Generalist vs. Aggressive Specialist distinction provides an interpretable second dimension that directly answers the pipeline-suitability question without requiring the evaluator to specify importance weights or manually scan dozens of metrics. This is not a refinement of existing methods; it is a new type of evaluation output that no prior framework could produce.

The work also reconciles a latent tension in evaluation practice. The field has long recognized that simple averaging is fragile (sensitive to outliers, requires ad-hoc weights) and that Elo-based systems are statistically principled but one-dimensional. Yet these two approaches — statistical rigor vs. deployment relevance — have been treated as an inherent tradeoff. CSD suggests a third path: keep the statistical rigor (Monte Carlo with N = 100,000 trials, controlled stochasticity) but change the structure of the evaluation to encode deployment constraints directly. The key insight is that evaluation structure — not just estimation quality — determines whether a ranking is useful. A perfectly estimated Elo rating answers a question that may not be the right one for deployment; a CSD simulation with a well-chosen sequence answers the deployment question directly, even if the estimation is approximate (Monte Carlo rather than analytic).

The work makes certain research directions more attractive and others less so:

More attractive:

• Structured evaluation design. CSD demonstrates that evaluation structure — the sequence, the elimination rules, the pairing mechanism — can encode assumptions that would otherwise require subjective weights. This suggests that future evaluation frameworks should invest more effort in formalizing the structure of deployment pipelines and less effort in optimizing weighting schemes for static aggregation.
• Risk-aware model selection. The Failure Sensitivity Analysis shows that model risk (vulnerability to catastrophic failure) is a measurable, distinguishable dimension from model quality (average performance). This opens the door to risk-aware deployment strategies where models are selected based on risk tolerance, not just expected performance.
• Simulation-based evaluation. The paper's intractability argument (Section 2.3) provides a methodological justification for simulation-based evaluation that extends beyond CSD. Any evaluation framework that wants to capture path-dependent interactions — and the paper argues convincingly that deployment-relevant evaluation should — will face similar combinatorial complexity and will need Monte Carlo methods. CSD provides a template for how to do this rigorously.

Less attractive:

• Marginal improvements to weighting schemes. If CSD's central argument — that path dependency matters and weights cannot capture it — is correct, then efforts to find better weighting formulas for static aggregation are addressing a fundamentally limited question. No weighting scheme, however sophisticated, can model the fact that failure at step 1 gates success at step 2. The structural limitations of aggregation are not fixable by better coefficients.
• Purely one-dimensional ranking. The Robust Generalist vs. Aggressive Specialist distinction demonstrates that one number per model is insufficient for deployment decisions. Future evaluation work that produces only a scalar score per model will need to justify why the second dimension (risk) can be safely ignored — a burden that did not exist before this paper made risk quantification concrete.

However, the paper does not resolve the fundamental question it implicitly raises: does CSD's notion of "competitive robustness under sequential elimination" actually correspond to real-world deployment success? This remains an untested hypothesis. The paper provides internal validation (rankings are stable under score perturbations, the Failure Sensitivity Analysis reveals interpretable model differences) but no external validation against any deployment outcome, user preference signal, or even alternative ranking system. This is the critical gap between CSD as a compelling conceptual framework and CSD as a validated deployment tool. As discussed in the Limitations section (Section 6), the paper acknowledges this but does not even propose specific validation experiments. The landscape has therefore changed — a new evaluation paradigm has been introduced, with a new type of evaluation output (risk profiles) — but the magnitude of the change depends on whether future work can connect CSD rankings to external outcomes, a connection this paper does not establish.

---

Follow-Up Research This Work Enables

1. External validation of CSD rankings against production deployment outcomes. The single most important missing piece in this paper is any evidence that CSD rankings predict real-world model utility better than existing methods. A strong follow-up would partner with an organization that deploys LLMs in sequential pipelines (e.g., customer support automation, where models must handle intent classification → information retrieval → response generation → escalation decisions sequentially) and compare: (a) CSD rankings using benchmarks that map to pipeline stages, (b) Elo ratings from Chatbot Arena, (c) simple average accuracy across relevant benchmarks, and (d) actual pipeline success rates in production. The key measurement is whether CSD's Robust Generalist classification — models with $\Lambda_m \approx 0$ — identifies models that experience fewer pipeline-blocking failures in deployment compared to Aggressive Specialists with similar base performance. A negative result (CSD ranking correlates no better with deployment success than Elo or averaging) would not invalidate CSD's conceptual framework but would suggest that path dependency in evaluation tournaments does not capture deployment path dependency as effectively as the paper hypothesizes.

2. Sensitivity of CSD rankings to benchmark sequence order. The paper's central claim — that CSD replaces subjective weighting with structural importance derived from benchmark sequence — is empirically unvalidated because the paper never varies the sequence. A targeted follow-up would take the same 38 benchmarks and 29 models and run CSD under systematically varied sequences: (a) the original "foundational-to-specialized" order from Table 1, (b) the reverse order (specialized-to-foundational), (c) a random permutation, (d) several domain-specific orders (e.g., coding-first for software engineering deployment, reasoning-first for scientific deployment). The key questions: Do tier assignments (especially Tier 1 vs. Tier 2–3) remain stable across sequences? Does the Robust Generalist vs. Aggressive Specialist classification persist, or do some models switch categories depending on sequence? If rankings are largely invariant to sequence, then CSD's claimed advantage over weighting — that it replaces an arbitrary choice with a principled one — is weakened, because the sequence choice wouldn't matter much in practice. If rankings are highly sensitive, then the paper's framing needs refinement: sequence choice is consequential, and the paper must provide guidance on how to validate sequences for specific deployment contexts (something it currently does not do).

3. Continuous win-probability CSD vs. binary win/loss CSD. The paper binarizes all model comparisons into $W(i, j, k) \in \{0, 1\}$, discarding performance margin. A natural follow-up would implement a variant of CSD where $W(i, j, k)$ is the actual win probability (or the continuous score difference normalized to [0,1]) and compare the resulting rankings, tier assignments, and $\Lambda_m$ classifications against the binary version. The specific hypothesis to test: models whose Aggressive Specialist classification in the binary CSD arises from being narrowly on the losing side of many close comparisons (rather than from genuinely catastrophic failures) should shift toward Robust Generalist in the continuous version, because small performance differences that flip binary outcomes become small continuous differences that don't dramatically affect expected scores. This would distinguish between "the model genuinely fails on some benchmarks" and "the model is slightly weaker than peers on some benchmarks, and binarization amplifies this into apparent failure" — a distinction with direct deployment implications. The continuous version would also naturally handle ties (identical scores map to 0.5 win probability), eliminating the tie-breaking arbitrariness that the binary version must resolve silently.

4. Convergence analysis of Monte Carlo CSD and efficient approximation. The paper uses N = 100,000 trials without justification. A methodological follow-up would systematically analyze how $\hat{E}[S_m]$ and $\Lambda_m$ estimates converge as a function of N, reporting both the standard error of $\hat{E}[S_m]$ (which should scale as $1/\sqrt{N}$) and — more importantly — the stability of the ranking order (do the top-5 models ever swap positions as N increases?). This matters because the paper positions CSD as a practical evaluation tool, and practitioners need to know whether N = 1,000 or N = 10,000 would suffice. Additionally, the paper's W tensor pre-computation "once, then simulate" design suggests that variance reduction techniques (common random numbers, stratified sampling of pairing configurations, antithetic variates) could substantially reduce the required N while maintaining ranking precision. A follow-up that identifies the minimum N for ranking stability at the 29-model, 12-round scale would make CSD adoption more practical. If N = 5,000 produces identical tier assignments as N = 100,000, the computational barrier drops substantially.

5. Failure Sensitivity Analysis across a full $T_k$ range with statistical testing. The paper's FSA uses only $T_k \in \{0, 1, 2\}$ and classifies models by visual inspection of a scatter plot. A rigorous follow-up would: (a) sweep $T_k$ from 0 to the maximum possible (where all but the winner are eliminated each round), computing $\hat{E}[S_m](T_k)$ at each point with confidence intervals from the Monte Carlo replicates; (b) fit a parametric or non-parametric curve to each model's $\hat{E}[S_m](T_k)$ function; (c) perform formal hypothesis tests for whether a model's sensitivity is significantly different from zero (using the Monte Carlo standard errors, which at N = 100,000 should provide sufficient power); (d) identify whether the $\hat{E}[S_m](T_k)$ curves naturally cluster into discrete categories (Robust Generalist, Aggressive Specialist, and possibly others — e.g., "Fragile at low $T_k$ but resilient at high $T_k$," which would indicate a model that occasionally falls into $G_{\min}$ but survives when elimination pressure is moderate) or form a continuum. The paper's current binary classification may be an oversimplification, and a full sweep would reveal whether the Robust Generalist / Aggressive Specialist distinction is a genuine taxonomy or a convenient narrative overlaid on a continuous distribution.

6. Leave-one-benchmark-out prediction of CSD rankings. The paper proposes but never tests that CSD can predict model rankings on held-out benchmarks. The 38-benchmark suite includes benchmarks from distinct capability categories (math, coding, agent, factuality). A targeted experiment: remove all math benchmarks (AIME24, AIME25, BeyondAIME, MathConstruct) from the CSD sequence, compute CSD rankings on the remaining 34 benchmarks, then compare the resulting model ranking against actual performance on the held-out math benchmarks. Does CSD's ranking order predict math performance better than (a) a simple average of all non-math benchmarks, (b) Elo ratings computed from non-math benchmarks, or (c) a domain-specific baseline (e.g., average of reasoning benchmarks only)? Repeat for coding benchmarks, agent benchmarks, and factuality benchmarks. This would test whether CSD's tournament structure captures something about general sequential competence that transfers across capability categories, or whether CSD rankings are primarily driven by the specific benchmarks included and do not generalize to held-out tasks. This is a weaker version of the agentic performance prediction claim (Section 5.1.1) and is directly testable with the paper's existing data, making it a natural and high-priority follow-up.

---

Practical Applications and Downstream Use Cases

1. Enterprise LLM procurement with risk-aware selection. An organization evaluating LLM APIs for integration into a multi-stage customer support pipeline — where the model must handle intent classification → policy lookup → response generation → escalation judgment in sequence — faces the exact deployment scenario CSD is designed for. Using CSD, the procurement team can: (a) map their pipeline stages to benchmark categories (e.g., instruction following → IFEval for intent classification; factuality → FActScore for policy lookup; reasoning → GPQA for escalation judgment); (b) sequence these benchmarks in pipeline order; (c) run CSD on all candidate models; and (d) select based on the Failure Sensitivity Analysis rather than raw average scores. The paper's finding that Qwen-Plus-0728 achieves high base performance but collapses under elimination pressure ($\Lambda_m \ll 0$, classified as Aggressive Specialist) while GPT-5.1-High maintains stable performance ($\Lambda_m \approx 0$, Robust Generalist) means the procurement team can explicitly quantify the risk of choosing the cheaper, higher-peak model versus the more expensive, consistently reliable one. Without CSD, this risk is invisible — the two models might have similar average scores — and the procurement decision reduces to price negotiation. With CSD, the risk dimension becomes a quantifiable input to the decision.

2. Continuous monitoring of model versions for regression detection. When model providers release updated versions (e.g., Gemini-2.5-Flash → Gemini-2.5-Flash.1; GPT-5-medium → GPT-5-chat), practitioners need to know whether the update improves general competence or creates new failure modes (a "specialization" that sacrifices breadth for depth). CSD's two-dimensional output — base performance $\hat{E}[S_m \mid T_k = 0]$ and sensitivity coefficient $\Lambda_m$ — provides a natural monitoring dashboard. If a model update increases base performance but makes $\Lambda_m$ significantly more negative (shifting from Robust Generalist toward Aggressive Specialist), the update has introduced fragility — the model is now better on its strong tasks but more vulnerable to catastrophic failure on weak ones. This pattern, which would be invisible in a simple average score (since the base performance improved), would be immediately flagged by CSD's FSA. The paper's Tier 4 observation that "lightweight or efficiency-focused versions" concentrate in the lowest tier (Section 4.1, Finding 1) suggests that this monitoring is practically relevant: model compression and optimization often produce models with worse CSD profiles than their larger counterparts, even when average scores remain comparable.

3. Internal evaluation pipeline design for LLM development organizations. An organization developing LLMs (like the paper's affiliation, ByteDance Seed) that maintains hundreds of internal benchmarks for model evaluation faces the exact scaling problem the paper addresses. CSD provides a principled way to compress multi-benchmark results into an interpretable ranking without requiring per-benchmark importance weights, which are particularly difficult to calibrate across hundreds of heterogeneous tasks. The 100,000-trial Monte Carlo simulation is a one-time computational cost (the W tensor is updated only when new evaluation results arrive), and the output — tier assignments, $\hat{E}[S_m](T_k)$ curves, $\Lambda_m$ values — can be tracked across model iterations as a standardized health dashboard. The paper's demonstration that CSD rankings are stable under score perturbations (Section 4.2: zeroing four benchmark scores for Qwen3-Max shifts its rank by only a few positions under CSD versus a 19th-place collapse under simple averaging) is particularly relevant: in large-scale internal evaluation where API instability and measurement noise are common, CSD's structural robustness means that ranking decisions are not dominated by noisy benchmarks.

4. Difficulty-stratified analysis of model weaknesses for targeted improvement. The single-benchmark CSD analysis (Section 5.1.2) — partitioning questions into difficulty tiers and running CSD with those tiers as sequential rounds — provides a diagnostic that standard accuracy reporting cannot: it identifies models that perform disproportionately poorly on easy questions while excelling on hard ones. The paper reports this pattern for Kimi models on SuperGPQA and MMLU-Pro (Figure 6). For a model development team, this is actionable: a model that fails easy questions but succeeds on hard ones may have issues with basic instruction following, formatting, or output parsing that are masked by strong reasoning capabilities on complex problems. Standard accuracy would report a single number (say, 75% on MMLU-Pro) with no indication that this represents 60% on easy questions and 90% on hard ones — a variance pattern that CSD's difficulty-tier analysis captures through the performance drop $\Delta E[S_m]$. The team can then prioritize fixing the basic failure modes, which are likely cheaper to address (better output parsing, improved instruction tuning) than further improving already-strong reasoning capabilities, and the CSD difficulty-tier analysis provides a quantitative before/after metric for whether the fix worked.

---

When to Prefer This Method

The paper does not provide an explicit decision framework comparing CSD against named alternatives (Elo, static averaging), but the conceptual arguments in Sections 2 and 3, combined with the experimental limitations discussed in Section 6, imply conditional preferences that can be extracted:

Prefer CSD over static aggregation or Elo when:

• The deployment pipeline has sequential dependencies where failure at an early stage blocks later stages (supply chain automation, multi-step agentic workflows, financial compliance pipelines). CSD's path-dependent tournament structure encodes this directly; static aggregation assumes independence.
• The evaluation goal includes risk assessment as well as quality assessment — you need to know not just "which model is best on average" but "which model is unlikely to catastrophically fail on any single capability." The Failure Sensitivity Analysis provides this second dimension; Elo and averaging produce only one.
• The benchmark suite is large and heterogeneous, making manual weight specification infeasible or highly contested. CSD replaces weight choices with sequence choices, and sequence design can be grounded in deployment dependency structures rather than abstract importance judgments.
• The evaluation data is noisy or subject to occasional measurement errors (API instability, evaluation environment variation). The paper's perturbation experiments (Section 4.2) demonstrate that CSD rankings degrade gracefully under score corruption, while simple averaging can collapse (Qwen3-Max drops to 19th under averaging vs. remaining in Tier 3 under CSD with four zeroed benchmarks).

Prefer Elo or Bradley-Terry when:

• The evaluation data consists of human preference judgments from pairwise comparisons (Chatbot Arena style), where the independent-pairwise assumption is reasonable and path dependency is not a concern. Elo was designed for this setting; CSD requires a fixed benchmark sequence, which does not naturally arise from crowd-sourced pairwise votes.
• The deployment context has no sequential dependency structure — each task is independent, and failure on one task does not affect others. In this case, CSD's elimination mechanism may penalize models for failures that have no deployment consequence, and a simpler skill-estimation method would be more appropriate.
• The evaluation goal is universal skill estimation (a single number that predicts win probability against any opponent on any task), not deployment-specific fitness. Elo estimates a latent skill parameter; CSD estimates expected tournament outcome under a specific structure. The former is more portable; the latter is more context-specific.

Prefer CSD with caution due to unvalidated assumptions when:

• The benchmark sequence order is not clearly motivated by deployment dependencies. The paper's sequence sensitivity is untested; if you cannot justify why IFEval should precede AIME24 rather than follow it, CSD's structural importance assignments may encode arbitrary rather than principled prioritization.
• The binary win/loss abstraction may discard critical information. If performance margins matter (a model that consistently beats opponents by 20% is materially different from one that beats them by 1%), CSD's binary W tensor loses this signal, and continuous-score methods (weighted averaging with proper calibration) may be more informative.
• The verification of Robust Generalist classification against actual deployment outcomes has not been performed. If you are making high-stakes deployment decisions (safety-critical systems, large financial commitments), you should supplement CSD rankings with deployment-specific testing rather than relying on CSD's risk classification alone, since the paper provides no evidence that $\Lambda_m$ predicts real-world failure rates.