Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models

🎯 Pitch

You can replace the neural networks at the heart of game-solving AI with LLMs that write commented Python code—and the resulting agents are not just interpretable but competitive with a 27B-parameter RL baseline. The key is matching the refinement mechanism to your goal: evolutionary optimization produces the hardest-to-exploit policies, while targeted feedback loops maximize raw performance.

1. Executive Summary

This paper introduces Code-Space Response Oracles (CSRO), a framework that replaces the deep reinforcement learning oracles in Policy-Space Response Oracles (PSRO) with Large Language Models (LLMs) that generate multi-agent policies as human-readable source code. The method is validated on Repeated Rock-Paper-Scissors and Repeated Leduc Hold'em poker, using Gemini 2.5 Pro as the code-generating LLM. CSRO's oracle operates through three refinement mechanisms of increasing power—ZeroShot (single-pass generation), LinearRefinement (an intra-iteration feedback loop that regenerates policies until they achieve non-negative utility against the current meta-strategy), and AlphaEvolve (a distributed evolutionary system that mutates programs across multiple threads)—with AlphaEvolve producing the lowest exploitability policies (25.2 ± 20.3 PopExpl in RRPS; 4.4 ± 0.6 in Leduc) and LinearRefinement achieving the highest aggregate score against the external population (122.1 ± 9.8 AggScore in RRPS, competitive with a 27B-parameter Gemma 3 baseline at 126.0). The policies are inherently interpretable—commented Python classes whose strategic logic can be directly inspected—establishing that LLM-based code generation can produce game-theoretic equilibria comparable to mature baselines while yielding verifiable, compositional strategies only when the refinement mechanism is matched to the evaluation criterion (evolutionary optimization for exploitability minimization, targeted feedback loops for population return).

2. Context and Motivation

The Core Problem: Multi-Agent Learning Produces Opaque, Uninterpretable Policies

The fundamental tension this paper addresses is between performance and interpretability in multi-agent reinforcement learning. Over the past decade, game-theoretic approaches to multi-agent learning — particularly the PSRO family of algorithms — have achieved remarkable success in computing approximate Nash equilibria for increasingly complex games. AlphaStar (Vinyals et al., 2019) reached Grandmaster level in StarCraft II; DeepNash (Perolat et al., 2022) achieved expert-level play in Stratego; and numerous other systems have demonstrated superhuman performance in domains ranging from poker to autonomous driving.

But there is a catch: every one of these successes relies on deep reinforcement learning oracles that produce neural network policies. These policies are, in the authors' words, "black-box" models — massive matrices of floating-point weights that resist human inspection or understanding. When such an agent makes a decision in a high-stakes setting, there is no straightforward way to answer the question why did it do that? You can query the network, observe its outputs, and run counterfactual analyses, but you cannot open it up, read a clear explanation of its strategic logic, and verify that its reasoning is sound.

This is not merely an aesthetic concern. The authors identify three concrete consequences of policy opacity (Section 1):

Strategy verification is impossible. If a multi-agent system is deployed in cybersecurity, financial markets, or military simulations, stakeholders need to verify — before deployment — that the learned strategies satisfy safety constraints, legal requirements, or ethical guidelines. Opaque neural policies cannot be verified in this way; they can only be tested empirically, which provides no guarantees.
Debugging is prohibitively difficult. When a neural policy fails — exploits a loophole in the environment, produces an unintended side effect, or collapses against an unexpected opponent — there is no straightforward way to diagnose why the failure occurred or to fix the specific component of the strategy responsible.
Trust is undermined. In any domain where human operators must work alongside or oversee autonomous agents (air traffic control, medical decision support, military command), the inability to explain decisions erodes trust and blocks adoption, regardless of the agent's empirical performance.

The paper frames this as a fundamental limitation of the standard PSRO paradigm, not an implementation detail. The authors explicitly state that "this lack of interpretability prevents strategy verification, and forms a significant barrier to deploying such agents in high-stakes, real-world applications where explainability is crucial" (Section 1). The choice of the word "barrier" is deliberate — it implies that interpretability is not a nice-to-have feature but a hard requirement for certain deployment contexts that the current approach cannot satisfy at all.

A Secondary Problem: Sample Inefficiency

The paper also highlights a second, more practical limitation of RL-based oracles: sample inefficiency. The authors note that "training these RL oracles is often sample-inefficient, requiring millions or billions of game simulations to converge" (Section 1). This matters for several reasons:

Computational cost: Training a single best response in PSRO can require enormous amounts of environment interaction. Since PSRO trains a new oracle at every iteration — and the meta-game itself requires evaluating all policy-policy matchups — the total sample budget grows combinatorially.
Simulator fidelity: In domains where the environment is a learned or approximate simulator (e.g., robotics, economic modeling), running millions of training episodes may be infeasible or may introduce sim-to-real transfer problems.
Iterative development: When researchers want to experiment with different meta-solvers, population sizes, or game variants, the RL training time becomes a bottleneck on the research cycle itself.

CSRO addresses this by leveraging the LLM's pretrained knowledge of logic, planning, and strategy. The oracle does not need to learn the game's rules or basic strategic concepts through trial and error — these are already embedded in the model's weights from pretraining on code and text corpora. The oracle's job is reduced to reading a description of the current meta-game (opponent strategies, game rules) and synthesizing a novel counter-strategy through in-context reasoning rather than environment interaction. This is a fundamentally different scaling regime: the number of LLM calls grows linearly with the number of iterations (e.g., $K = 20$ in the experiments), not with the number of environment steps needed to train a neural network from scratch.

Where Prior Approaches Fall Short

The paper identifies specific limitations in three categories of prior work:

1. Standard PSRO and its variants (the dominant paradigm). PSRO (Lanctot et al., 2017) and extensions like Pipeline PSRO (Mcaleer et al., 2020), $\alpha$ -rank-based meta-solvers (Omidshafiei et al., 2019), and others have focused almost exclusively on improving convergence speed and scalability — how to compute equilibria faster, with larger populations, on more complex game trees. The authors are careful to note that CSRO is "orthogonal to these meta-solver improvements" (Section 5). The meta-solver (the component that computes the equilibrium mixture $\sigma$ over the current policy set $P$ ) is unchanged; CSRO changes the nature of the oracle itself. But critically, no prior PSRO variant has addressed the opacity problem. The policies remain neural networks regardless of how sophisticated the meta-solver becomes. This means that even as PSRO scales to harder problems, the interpretability barrier becomes more acute — larger networks are even harder to understand.

2. LLM-guided strategic reasoning (emerging but limited). Several recent works have explored using LLMs for game-theoretic reasoning, but with significant limitations relative to CSRO's goals:

LLM-PSRO (Bachrach et al., 2025) is the most direct precursor. It demonstrated that an LLM could generate code for game-playing agents within a PSRO loop. However, the authors identify three critical gaps that CSRO addresses (Section 5): (a) LLM-PSRO uses "open-loop best-of-N sampling" — the LLM generates a batch of candidates and the best is selected, with no iterative refinement. This means there is no mechanism to systematically improve a losing policy based on feedback from the meta-game. (b) LLM-PSRO includes opponent source code directly in the prompt without abstraction, which limits scalability to games with large policy populations (context windows are finite). (c) LLM-PSRO only evaluates policies against the internal population (self-play); it provides no evidence that the generated policies generalize to hold-out opponents or external baselines. The authors explicitly note this: "the work only includes inter-population evaluation and does not show performance against external populations or baselines."
Game-theoretic prompting methods (Gemp et al., 2024) integrate LLMs with solvers like CFR and PSRO, but the LLM's role is to generate natural language prompts containing high-level strategic instructions to another game-playing LLM. The policies themselves are implicit in the LLM's weights — they are not extractable, inspectable, or reusable as standalone artifacts. The authors draw a sharp distinction: "the best responses produced by their method are natural language prompts, whereas CSRO outputs executable code."
Game of Thoughts / PSROLM (Kempinski et al., 2025) uses LLMs within an "outer loop" for iterative strategic reasoning inspired by cognitive hierarchy theory. The LLM refines its play against prior strategies through level- $k$ reasoning. However, the authors identify two fundamental differences: (a) PSROLM generates actions directly (the LLM's output is a move, not a policy), which means there is no persistent, inspectable policy artifact that can be analyzed, reused, or composed. (b) PSROLM lacks an intra-iteration refinement loop — the policy is whatever the LLM produces on a single pass, with no systematic hardening against the current meta-game before being added to the population.

A unifying limitation across all prior LLM-game-theory work: None of these approaches produces policies that are simultaneously (1) executable as standalone programs, (2) human-readable and commented, (3) refined through closed-loop feedback against the meta-game, and (4) validated against external, standardized evaluation populations. CSRO aims to satisfy all four criteria simultaneously.

3. The Chinchilla-based LLM agent baseline (Lanctot et al., 2023). This work demonstrated that a pretrained LLM could play Repeated Rock-Paper-Scissors by predicting opponent actions from a textual history prompt and playing the best response — achieving strong aggregate scores (155.2 for a 70B model). While this approaches interpretability (the LLM's predictions can be inspected), it has two critical drawbacks from CSRO's perspective: (a) Cost at inference time: the LLM is queried on every turn of every game — 1,000 model calls per match. In contrast, CSRO generates a policy once and executes it as cheap Python code. (b) No strategic synthesis: the LLM predicts moves, but does not produce a persisting, composable strategic artifact that can be analyzed, combined with other policies, or deployed independently of the LLM. The paper explicitly frames this efficiency difference as a key trade-off (Section 6): "the baseline LLM agents are invoked on every turn, i.e., requiring 1000 model calls for a single game. In contrast, LinearRefinement generates a complete, reusable policy where the number of LLM calls only grows linearly with the number of iterations."

How This Paper Positions Itself

CSRO is positioned not as an incremental improvement to PSRO's meta-solver or as yet another way to use LLMs for game playing, but as a fundamental reframing of the best-response computation itself. The key conceptual move is stated clearly in Section 2.2: "This reframes the best response computation from a process of numerical optimization to one of programmatic reasoning and generation." Rather than training a neural network to approximate $\arg\max_{\pi} \mathbb{E}_{\pi' \sim \sigma} [u(\pi, \pi')]$ through gradient descent on millions of sampled trajectories, CSRO asks an LLM to write a program that embodies a strategy designed to beat the current opponent mixture.

This reframing carries several implications that define the paper's identity:

The unit of optimization shifts from weight space to code space. The oracle's output is not a tensor of floating-point numbers but a Python class with an act method, internal state variables, and — crucially — docstrings and comments explaining the strategy. This makes interpretability a structural property of the method rather than a post-hoc explanation exercise. The policy is its own explanation.
The role of pretrained knowledge is made explicit. The LLM's pretraining on code corpora, game theory discussions, and strategic analysis is treated as a first-class resource — not as suspicious "contamination" to be worried about, but as a capability to be leveraged. The paper acknowledges this directly: "the LLM's pre-training data likely contains knowledge of game strategies for a classic game like Rock-Paper-Scissors" (Section 6). But the authors argue this is not simply pattern retrieval: "the oracle must synthesize a novel best response to a specific, dynamically generated mixture of programmatic opponents provided in-context. The success of this process demonstrates a sophisticated capability for in-context strategic reasoning and code generation."
Scalability is addressed through context abstraction. Unlike LLM-PSRO, which includes full opponent source code in the prompt, CSRO introduces a context abstraction mechanism (Section 2.4.1): opponent strategies can be summarized in natural language by another LLM call, and only the most relevant opponents (filtered by equilibrium support or top-k) are included. This is not just an engineering convenience — it is what enables CSRO to scale beyond the small populations where full source code fits in a context window. The ablation in Table 7 (Supplementary Materials) partially validates this: in the ZeroShot setting, providing opponent strategies as natural language descriptions yields better results than providing code (AggScore of 63.5 vs. -54.3), suggesting that textual summaries can be more tractable inputs for single-pass generation even at moderate population sizes.
Validation is against standardized external populations. The paper explicitly positions itself as providing the first rigorous external validation of code-generating PSRO oracles. The evaluation population in RRPS consists of 43 hand-coded bots from international competitions (Billings, 2000a,b) — a diverse set spanning Markov models, frequency analyzers, pattern matchers, and neural-like systems. This allows measurement of not just self-play convergence but generalization — can the CSRO-generated policy exploit strategies it was never explicitly trained against? The three metrics (PopReturn, PopExpl, AggScore) from Lanctot et al. (2023) capture this multi-dimensional evaluation, distinguishing between an agent that beats most opponents but is catastrophically exploited by one (high PopReturn, high PopExpl, low AggScore) and one that is genuinely robust.

The Connection Between Interpretability and Strategic Sophistication

A subtle but important thread in the paper's motivation is the hypothesis that interpretability and strategic sophistication are not in tension — that by generating code, the LLM can actually produce more sophisticated strategies than it could through pure action prediction. The qualitative analysis (Section 4.3) provides evidence for this: the best RRPS policy contains a second-order theory of mind component that explicitly "infers the opponent's likely predictive model by observing which of its own experts is currently most successful" and then "simulates the opponent's prediction of its own move and plays the corresponding counter-move." This is a level of strategic reasoning that would be difficult to specify in a reward function but emerges naturally from a code-generation prompt that asks the LLM to "reason about these opponents, and come up with a strategy that can exploit them" (Supplementary Materials Listing 3).

Similarly, the best Leduc policy synthesizes a dynamic expected value calculation driven by opponent modeling, where the agent tracks the opponent's folding probability and adjusts its bluffing/value-betting threshold accordingly. Against an AlwaysCall opponent, it learns to stop bluffing and only raise with strong hands; against an AlwaysFold opponent, it learns that raising is always profitable regardless of hand strength. This transparent adaptation — directly readable in the policy's _calculate_action_ev method — demonstrates a synthesis of poker theory, opponent modeling, and adaptive execution that the code-generation paradigm enables.

The paper thus makes an implicit argument: code is not just a more interpretable representation of policy; it is a more expressive medium for strategy that enables compositional, modular, and hierarchically-organized behavior that would be difficult to discover through pure numerical optimization.

3. Technical Approach

3.1 Reader Orientation

CSRO is a system that builds a population of game-playing strategies by repeatedly asking a large language model to write Python code that defeats the current collection of known strategies, rather than training neural networks through millions of simulated games. It solves the twin problems of policy opacity and sample inefficiency in multi-agent reinforcement learning by reframing the hardest step—computing a "best response" to an opponent's strategy—as a code generation task where the output is not a matrix of weights but a commented, executable program whose logic can be read, understood, and verified by humans.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interconnected components, arranged in an outer-inner loop structure:

The Meta-Game — a payoff matrix $U$ recording how every policy in the current population $P$ performs against every other policy. This is computed by actually playing the policies against each other in the game environment and recording the outcomes.
The Meta-Solver — a standard game-theoretic equilibrium solver that takes the payoff matrix $U$ and computes a symmetric equilibrium mixture $\sigma$ over the current policies. This mixture tells us which opponent strategies currently matter most and with what weight they appear.
The Prompt Constructor — a module that builds a structured prompt for the LLM containing the game rules, the API specification for the policy function, and crucially, descriptions (or source code) of the opponent strategies that are currently active in the equilibrium mixture $\sigma$ . It applies filtering (e.g., top-5 by probability mass) and optional natural-language summarization to keep the prompt within context limits.
The LLM Oracle — an LLM (Gemini 2.5 Pro in all experiments) that receives the constructed prompt and generates a candidate policy as executable Python code. This is the core replacement for the deep RL oracle in standard PSRO.
The Intra-Iteration Refinement Loop — a feedback mechanism that evaluates the candidate policy against $\sigma$ , and if the policy is losing (expected utility < 0), feeds the performance results back to the LLM for regeneration. This loop continues until the policy achieves non-negative utility or exhausts a refinement budget $M = 10$ . An alternative instantiation, AlphaEvolve, replaces this loop with a distributed evolutionary search.

Information flows as follows: at each iteration $k$ , the meta-game matrix is computed from all previous policies → the meta-solver produces equilibrium mixture $\sigma$ → the prompt constructor builds a prompt describing the game and the relevant opponent strategies → the LLM oracle generates a candidate policy → the refinement loop evaluates, provides feedback, and regenerates until the policy is satisfactory → the final policy is added to the population $P$ , and the cycle repeats for $K = 20$ iterations.

3.3 Roadmap for the Deep Dive

First, the standard PSRO formulation (Section 2.1 of the paper), because CSRO inherits the outer-loop structure exactly and only changes the oracle—understanding what the oracle must compute is prerequisite to understanding how it computes it.
Second, the code policy abstraction (Section 2.2), because it defines the output format that all oracle variants must produce and establishes why a Python class is the right representation for interpretable, stateful strategies.
Third, the prompt construction mechanism (Section 2.4.1), because the prompt is the primary interface between the meta-game state and the LLM's reasoning—its content determines what strategic information the oracle has access to.
Fourth, the intra-iteration refinement loop in detail (Section 2.4.2), covering ZeroShot, LinearRefinement, and AlphaEvolve as three points on a spectrum of optimization power, since this is the main technical distinction from prior LLM-PSRO work.
Fifth, the complete CSRO algorithm as pseudocode and operational narrative, tying all components together into the full iterative procedure.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that best-response computation in multi-agent games can be reframed as a program synthesis task, where an LLM generates executable code policies that are iteratively refined against the current meta-game equilibrium, and that the choice of refinement mechanism—zero-shot generation, linear feedback loops, or evolutionary search—determines the trade-off between exploitability minimization and population return maximization.

The PSRO Outer Loop: What the Oracle Must Compute

CSRO adopts the standard PSRO iterative structure unchanged (Section 2.1). The paper focuses on two-player symmetric zero-sum games, though the method directly extends to general settings. In a symmetric game, both players share the same strategy set $\Pi$ and the same utility function $u$ . For any two strategies $\pi, \pi' \in \Pi$ , if the first player plays $\pi$ and the second plays $\pi'$ , the first player receives $u(\pi, \pi')$ and the second receives $u(\pi', \pi) = -u(\pi, \pi')$ by the zero-sum condition.

PSRO maintains a single population of policies $P = \{\pi_1, \pi_2, ..., \pi_k\}$ . At each iteration $k$ , the algorithm performs three steps:

Compute the meta-game payoff matrix $U$ : the entry $U_{ij}$ is the expected utility of policy $\pi_i$ against policy $\pi_j$ , estimated by actually playing them against each other in the game environment for some number of episodes.
Compute the meta-equilibrium $\sigma$ : solve for a symmetric equilibrium mixture over the rows/columns of $U$ . In a symmetric game, this means finding a probability distribution $\sigma$ over the current policies such that no player can improve their expected payoff by unilaterally deviating to any other policy in $P$ . This is a standard linear programming or regret-minimization problem over the empirical game matrix.
Compute a best response $\pi^*$ to the opponent meta-strategy $\sigma$ :

$\pi^* \in \arg\max_{\pi \in \Pi} \mathbb{E}_{\pi' \sim \sigma} \left[ u(\pi, \pi') \right]$

where $\pi^*$ is any policy that maximizes expected utility, $\Pi$ is the space of all possible policies (not just the current population), $\pi$ is the candidate policy being optimized, $\sigma$ is the equilibrium mixture over the current population $P$ , $\pi'$ is an opponent policy sampled from $\sigma$ , and $u(\pi, \pi')$ is the utility when $\pi$ plays against $\pi'$ .

What it computes: the oracle's job is to find a policy that achieves the highest possible expected payoff against a mixture of opponents, where each opponent $\pi'$ appears with probability $\sigma(\pi')$ . Operationally, this means the oracle must consider multiple opponent strategies simultaneously and produce a policy that performs well against their weighted combination—not just a single exploitative counter to one opponent.

Why this form: the best-response objective is the workhorse of game-theoretic equilibrium computation because it guarantees that adding $\pi^*$ to the population strictly expands the empirical game in a direction that reduces exploitability. If $\pi^*$ achieves positive expected utility against $\sigma$ , the current mixture was not a true Nash equilibrium and the new policy improves the approximation. If no such $\pi^*$ exists (or it achieves zero expected utility), an approximate equilibrium has been found. This objective is also why balancing exploitation against strong opponents versus maximizing return against weak opponents matters: $\sigma$ assigns higher weight to strong opponents that are in the equilibrium support, so the oracle naturally prioritizes robustness over exploitation of easily-beaten strategies.

In standard PSRO, step 3 is implemented by a deep RL algorithm that trains a neural network through extensive environment interaction. CSRO replaces this with an LLM that generates code. The rest of the technical approach is entirely about how to implement step 3 using an LLM while preserving the convergence properties of the outer loop.

The Code Policy Abstraction

The central innovation described in Section 2.2 is the redefinition of what a policy is within the PSRO framework. In standard PSRO, a policy $\pi$ is a function approximator—typically a neural network—that maps observations to action probabilities. In CSRO, a policy is a code policy: a stateful Python class whose act method takes an observation dictionary and returns an action string.

A code policy is not just a more interpretable representation of the same mathematical object as a neural network. It differs in three structurally important ways:

Statefulness. A neural network policy in PSRO may be recurrent (e.g., LSTM-based) and thus maintain internal state across time steps, but this state is implicit in the network's hidden activations—it cannot be inspected, saved, or reasoned about independently. A code policy maintains explicit, named instance variables (e.g., self.opponent_history: List[str], self.opponent_model: Dict) that represent the agent's memory of past interactions. The act method can read and update these variables at each call. This means the policy's memory architecture is itself part of the generated code and can be read directly.

Compositionality. A code policy can be an arbitrary program that internally decomposes its strategy into subroutines, data structures, and conditional logic. The best policy from the RRPS experiments (Supplementary Materials Listing 1) contains 32 separate "expert" predictors (Markov models of orders 1 through 8, reactive models, joint history models, periodic predictors, meta-predictors), each implemented as a separate method or factory-generated closure, with a weighted voting system that aggregates their predictions. This compositional structure—building a complex strategy from modular, independently interpretable components—is natural in code but extremely difficult to induce in neural network training.

Transparency of logic. The act method's control flow is the policy's decision procedure. A reader can trace exactly how an observation leads to an action: which data structures are updated, which subroutines are called, what conditions are checked, and how the final action is selected. This is qualitatively different from attempting to interpret a neural network by analyzing its weights, gradients, or saliency maps.

The paper does not formally define the space of code policies (which would require specifying the Turing-complete programming language and observation space), but operationally, a code policy conforms to a specific API. The act method receives an observation: dict[str, Any] and returns a str. For RRPS, the observation contains my_action and opponent_action from the previous round (both None on the first round). For Leduc Poker, the observation is a nested dictionary containing the player's hand, legal actions, public state (round, pot size, chips, public card), and action history structured by round. The exact format is specified in the prompt (Supplementary Materials Listing 4 for Leduc).

A crucial design choice is that the code policy is executed directly in the game environment—the Python interpreter runs the policy's act method at each decision point, without any LLM involvement at inference time. This is what the paper means by "complete, reusable policy" (Section 6). The LLM is called during policy generation (once per iteration, or a few times if refinement is needed), but during actual gameplay, the policy runs as ordinary Python code. This is the source of the computational efficiency advantage over LLM agents that query the model on every turn.

Prompt Construction: The Interface Between Meta-Game and LLM

At each iteration, the CSRO oracle must convey the current strategic situation—what opponents are in the equilibrium mixture and how they play—to an LLM that has no persistent memory of previous iterations. Section 2.4.1 of the paper describes how this is accomplished through cross-iteration strategic adaptation, implemented in the construct_prompt function (Algorithm 1, line 5).

The prompt is not static; it is dynamically regenerated at every iteration to reflect the evolving meta-game. It contains four categories of information:

1. Game rules and objectives (static across iterations): A natural language description of the game mechanics, valid actions, winning conditions, and the number of rounds. For RRPS (Supplementary Materials Listing 3), this includes the three valid moves, the cyclic win relation (Rock beats Scissors, Scissors beats Paper, Paper beats Rock), and the goal of maximizing wins over 1000 rounds. For Leduc Poker (Supplementary Materials Listing 4), this is substantially more detailed—a 65-line specification covering blinds, raise sizing (2 units preflop, 4 units postflop), betting caps (two raises per round), the dealing procedure, and the full hand ranking system (pairs beat high cards, higher pairs beat lower pairs, ties split the pot).

2. API specification (static across iterations): A precise signature for the policy class or function, including the exact structure of the observation dictionary the act method will receive. For Leduc, this includes three example observations (two for act, one for receive_outcome) showing the full JSON structure with all keys, nested dictionaries, and example values. This is essential because the LLM must generate code that correctly accesses observation fields—a single KeyError means the policy is non-executable.

3. Opponent strategy descriptions (dynamic, iteration-dependent): This is the component that communicates the meta-game state. The construct_prompt function (implicit in Algorithm 1, line 6 referencing $\sigma$ and $P$ ) extracts the opponent policies that have non-zero probability in the current equilibrium mixture $\sigma$ . The paper explores two representation formats:

Source code: The full Python source code of each opponent policy is included verbatim in the prompt. This gives the LLM complete information about opponent logic but consumes large amounts of context and may be overwhelming when the population is large.
Natural language description: Another LLM call is used to summarize each opponent's strategy in prose, e.g., "This bot uses a 5th-order Markov model to predict the opponent's next move and plays the counter-move." The code policy's own docstrings and comments facilitate this summarization, since they already contain the intended strategy description.

The paper also introduces filtering mechanisms to manage prompt length when the equilibrium mixture includes many policies with small probabilities:

Top-k filtering: Include only the $k$ opponents with the highest probability in $\sigma$ . The paper uses Top 5 (Supplementary Materials Table 7).
Minimum support filtering: Include only opponents with $\sigma(\pi) \geq \tau$ for some threshold $\tau$ . When the equilibrium concentrates probability on a single policy (as often happens in PSRO), this may include only one opponent.

4. Task directive (static): A clear instruction to generate a best response, optionally including a prompt for self-explaining code: "implement an agent class called Agent that represents a player's strategy in a game of Repeated Rock Paper Scissors" (Listing 3) or "your task is to iteratively improve the provided bot in repeated leduc poker" (Listing 4). The Leduc prompt (used with AlphaEvolve and LinearRefinement) additionally includes detailed SEARCH/REPLACE block formatting rules for proposing code modifications.

Design rationale for prompt structure: The separation of static game information from dynamic opponent information is deliberate. The game rules and API are the same at every iteration and could theoretically be provided once in a system prompt. But the opponent descriptions change every iteration as new policies are added and the equilibrium shifts. By regenerating the full prompt each iteration, the LLM receives a self-contained task specification that requires no persistent memory of the meta-game history—all relevant context is in the prompt. This is important because the LLM is stateless between calls (each call to the Gemini 2.5 Pro API is independent).

The criticality of opponent conditioning: The paper includes a crucial ablation (Table 7, "ZeroShot (no opponent input)") where the oracle receives no information about opponent strategies—only the game rules and API. This variant achieves a high PopReturn (135.3 ± 10.2) but an extremely high PopExpl (614.2 ± 60.8), leading to a catastrophic aggregate score of -478.9 ± 70.2. This demonstrates that opponent conditioning is "the most important component of the prompt for generating robust strategies" (Section 4.1). Without knowledge of what opponents are doing, the LLM generates strategies that exploit weak opponents but are themselves highly exploitable.

Section 2.4.2 describes the second major innovation of CSRO relative to LLM-PSRO: an inner feedback loop that iteratively improves a candidate policy within a single PSRO iteration before adding it to the population. The paper presents three variants that form a spectrum of computational cost and optimization power. All three share the same input (a prompt describing the game and opponents) and the same output (an executable code policy), but differ in how the generation process is structured.

ZeroShot

ZeroShot is the simplest variant and corresponds directly to the case where the refinement loop (Algorithm 1, lines 9–14) is never entered. The LLM receives the prompt and generates a complete policy in a single pass. The policy is added to the population immediately, regardless of its performance against $\sigma$ .

This is the fastest and cheapest option—one LLM call per PSRO iteration—but provides no guarantee that the generated policy is actually a best response. The LLM might misinterpret the opponent descriptions, generate syntactically incorrect code, or produce a strategy that is logically coherent but underperforms against the specific opponent mixture. The paper's results show that ZeroShot (with description input) achieves a moderate aggregate score of 63.5 ± 11.4 in RRPS (Table 1), significantly below LinearRefinement and AlphaEvolve, confirming that single-pass generation leaves substantial performance on the table.

LinearRefinement

LinearRefinement introduces a conditional feedback loop that iteratively improves a losing policy. The mechanism works as follows (inferred from Algorithm 1, lines 9–14, and Section 2.4.2 description):

The LLM generates an initial candidate policy $\pi'$ from the prompt.
The policy is evaluated against the current equilibrium mixture $\sigma$ to obtain its expected utility $u = \mathbb{E}_{\pi' \sim \sigma}[u(\pi', \pi')]$ . This evaluation is done by actually playing $\pi'$ against opponents sampled from $\sigma$ in the game environment.
If $u \geq 0$ (the policy is at least break-even against the mixture), the policy is accepted and the refinement loop terminates. This is the success condition.
If $u < 0$ (the policy is losing on average), the prompt is updated with performance feedback. The update_prompt function (Algorithm 1, line 11) modifies the prompt to include information about the current policy's performance—what utility it achieved, and possibly qualitative feedback about its weaknesses. The specific contents of this feedback are not detailed in the paper but likely include the numerical utility value and potentially the specific opponent sub-strategies that the policy struggled against.
The LLM generates a new candidate policy from the updated prompt. This new policy replaces the previous candidate, regardless of whether it is better or worse (the algorithm as written does not explicitly retain the best-so-far, though the text says "the program is updated if the score is increased").
Steps 2–5 repeat until either $u \geq 0$ is achieved or the refinement budget $M = 10$ is exhausted (the counter $j$ reaches $M$ ).

What this achieves: LinearRefinement provides a safety net that prevents demonstrably losing policies from entering the population. In standard PSRO, a weak best response can still contribute to the meta-game (it provides more data about the payoff landscape), but it may also dilute the population with low-quality strategies. In CSRO, where each policy is a discrete, human-readable program, a losing policy could be confusing to interpret—the generated comments might claim a strategy that doesn't actually work against the opponents. LinearRefinement ensures that only policies that at least break even against the current meta-game are retained, which maintains both the quality of the population and the trustworthiness of the generated explanations.

Why not always use more refinement budget? The budget $M = 10$ balances two concerns. Too few refinement steps risk leaving easily-fixable weaknesses unaddressed. Too many steps risk overfitting the generated policy to the specific evaluation matches (since each evaluation uses a finite sample of opponent episodes) or wasting computation on policies that the LLM fundamentally cannot improve in the current prompt format. The paper does not ablate over $M$ , so the choice of 10 is a hyperparameter without sensitivity analysis.

The paper calls this "linear refinement" because all computations occur in a single thread—the generate-evaluate-feedback-regenerate cycle is sequential, each step depending on the previous.

AlphaEvolve

AlphaEvolve is the most powerful of the three oracle mechanisms and represents a fundamentally different approach to optimization. Rather than sequentially refining a single policy, AlphaEvolve runs a large-scale distributed evolutionary search over the space of code policies. The key properties (Section 2.4.2, drawing on Novikov et al., 2025; Romera-Paredes et al., 2024) are:

Population-based search with diversity maintenance. AlphaEvolve maintains multiple subpopulations of candidate programs, which are evolved independently. Programs are clustered into different subpopulations to ensure diversity—preventing the entire search from collapsing to a single strategy type. Each subpopulation explores a different region of the program space.

LLM-driven mutation. Each evolution thread continuously samples past programs from its subpopulation and prompts the LLM to generate modifications. The LLM acts as a mutation operator: given a program and a prompt describing the desired improvement, it produces a new program that differs from the parent by a small, semantically meaningful change. This is fundamentally different from random search over code—the LLM's mutations are guided by its understanding of programming and game strategy, so they are more likely to be improvements or meaningful variations than random character-level changes.

Score-guided selection. The fitness of each program is evaluated using an estimate of the expected utility against the opponent meta-strategy $\sigma$ :

$\text{fitness}(\pi) \approx \mathbb{E}_{\pi' \sim \sigma}[u(\pi, \pi')]$

Programs with higher fitness are more likely to be selected as parents for the next generation of mutations. This creates a selection pressure toward policies that perform well against the equilibrium mixture.

Independent subpopulation evolution. By evolving subpopulations independently, AlphaEvolve maintains multiple hypotheses about what constitutes a good strategy. This is crucial in game-theoretic settings where there may be multiple qualitatively different best responses (e.g., an aggressive exploitative strategy versus a balanced Nash strategy). The paper's results show that AlphaEvolve consistently produces the lowest exploitability policies (25.2 ± 20.3 in RRPS, 4.4 ± 0.6 in Leduc)—suggesting that the evolutionary pressure toward fitness against $\sigma$ naturally favors policies that are robust against the strongest opponents in the mixture, since those opponents dominate the fitness evaluation.

Why AlphaEvolve is suitable for best-response computation: The paper explicitly argues that AlphaEvolve is "perfectly suitable for our purpose of computing best response programs" because the score function—expected utility against $\sigma$ —is exactly the quantity that the best response should maximize (Equation 1). The evolutionary search directly optimizes the objective that defines the oracle's task, without the approximation inherent in sequential refinement or the single-shot limitation of ZeroShot. Moreover, the distributed nature of AlphaEvolve allows it to explore many candidate strategies in parallel, increasing the probability of finding a genuinely strong best response.

The relationship between AlphaEvolve and the other oracles: The three mechanisms form a spectrum of trade-offs between computational cost and optimization power. ZeroShot is cheapest (one LLM call) but weakest. LinearRefinement costs up to $M + 1 = 11$ LLM calls per iteration (initial generation plus up to 10 refinements) and produces better policies. AlphaEvolve is the most expensive—it makes many LLM calls across multiple evolution threads and generations—but produces the strongest optimization results for exploitability minimization. This spectrum means that practitioners can choose the oracle mechanism based on their computational budget and performance requirements.

The Complete CSRO Algorithm

Algorithm 1 of the paper presents the full CSRO procedure as pseudocode, which ties all components together. Here is the algorithm in operational detail, with the concrete hyperparameters used in the experiments:

Input: the game $G$ (a symmetric zero-sum game), the maximum number of iterations $K = 20$ , and the maximum refinement budget $M = 10$ . The game $G$ is not just an abstract specification—it includes the environment simulator (OpenSpiel in the experiments) that can execute policies and return payoffs.

Initialization (line 1): The policy set $P$ is initialized with a single initial policy $\pi_{\text{initial}}$ . For RRPS, this initial policy is not specified in the main text but is likely a simple random or heuristic strategy. For Leduc poker, the initial policy is the heuristic strategy shown in Supplementary Materials Listing 5—a rule-based bot that always raises with a King preflop, calls with a Queen, and calls (or folds if forced) with a Jack, and plays straightforwardly postflop based on hand strength relative to the public card.

Iteration loop (lines 2–16): For $k = 1$ to $K = 20$ :

Step 1: Compute the meta-game payoff matrix (line 3). This is the operation $U \leftarrow \text{compute\_payoff\_matrix}(P)$ . For each pair of policies $(\pi_i, \pi_j)$ in the current population $P$ , the system executes them against each other in the game environment and records the average utility. In a symmetric game, $U_{ij} = -U_{ji}$ by the zero-sum condition, and the diagonal $U_{ii}$ (a policy against itself) is zero in expectation for symmetric zero-sum games (though finite-sample estimates may show small non-zero values).

The number of evaluation episodes per matchup is not specified in the paper, but it must be large enough to produce reliable utility estimates—especially for the repeated games (1000 rounds per episode for RRPS, 100 hands for Leduc), where variance can be substantial.

Step 2: Compute the meta-equilibrium (line 4). This is the operation $\sigma \leftarrow \text{compute\_meta\_equilibrium}(U)$ . The paper does not specify the exact equilibrium solver used (possibilities include linear programming, replicator dynamics, or regret minimization), but for symmetric two-player zero-sum games, the problem reduces to finding a mixed strategy that maximizes the minimum payoff—a standard linear programming formulation.

The output $\sigma$ is a probability distribution over the current $|P|$ policies. The equilibrium computation is done on the empirical game matrix, not on the full game tree—this is the standard PSRO approximation that makes the approach tractable for large games.

Step 3: Construct the prompt (line 5). This is the operation $\text{prompt} \leftarrow \text{construct\_prompt}(G, \sigma, P)$ . As described in the prompt construction section above, this builds a structured text prompt containing game rules, API specification, and opponent descriptions filtered by the equilibrium $\sigma$ . The specific filtering and input format (code versus description) are hyperparameters that the paper sweeps over (Table 7).

Step 4: Generate a candidate policy (line 6). This is the operation $\pi' \leftarrow \text{llm\_oracle}(\text{prompt})$ . The LLM (Gemini 2.5 Pro) receives the prompt and generates Python code. The exact temperature and generation parameters are not specified in the paper. If the generated code is syntactically invalid (fails to parse), an implicit error-handling step would be needed—the paper mentions "error-handling and regeneration logic" in the limitations (Section 6) but does not detail it.

Step 5: Evaluate the candidate (lines 7). This is the operation $u \leftarrow \text{evaluate\_policy}(\pi', \sigma, P)$ . The new policy $\pi'$ is executed against opponents sampled from $\sigma$ to estimate $\mathbb{E}_{\pi' \sim \sigma}[u(\pi', \pi')]$ . At this stage, $\pi'$ has not yet been added to $P$ , so the evaluation is against the existing population only.

Step 6: Intra-iteration refinement (lines 8–14). The refinement loop runs while $\text{not terminated}(u, j, M)$ . The termination condition depends on the oracle variant:

ZeroShot: The loop never runs (implicitly, terminated is always true initially).
LinearRefinement: The loop terminates when $u \geq 0$ (the policy achieves non-negative utility against $\sigma$ ) or when $j \geq M = 10$ (budget exhausted). While the loop runs, the prompt is updated with performance feedback (line 11), the LLM generates a new candidate (line 12), and the new candidate is evaluated (line 13). The counter $j$ increments (line 10).
AlphaEvolve: The refinement stage is replaced entirely by the distributed evolutionary search. The initial LLM call (line 6) generates a seed program or population, and the evolutionary process (not explicitly shown in Algorithm 1 because it replaces the inner loop) iteratively improves the population using LLM-driven mutations and score-based selection. The final output is the best program found by the evolutionary search.

Step 7: Add to population (line 15). The final policy $\pi'$ (after refinement, if applicable) is added to the population: $P \leftarrow P \cup \{\pi'\}$ . At this point, the policy becomes available as an opponent for future iterations.

After all iterations (line 17): Return the final population $P$ and the equilibrium mixture $\sigma$ computed from the final meta-game matrix. The equilibrium mixture $\sigma$ over the final population is the algorithm's output strategy—it specifies how to mix among the discovered policies to achieve an approximate Nash equilibrium.

Why this structure enables convergence: The algorithm preserves the theoretical properties of PSRO. At each iteration, if the oracle produces a policy that achieves positive expected utility against the current equilibrium $\sigma$ , then the new equilibrium after adding this policy will be closer to a true Nash equilibrium of the full game. The algorithm inherits PSRO's convergence guarantees (to approximate equilibria) because the outer loop structure is unchanged. The only question is whether the LLM oracle can actually produce policies with positive utility—and the experimental results demonstrate that it can, particularly with iterative refinement.

Filtering and Input Format: Managing Context Complexity

The paper's supplementary material (Table 7) reports results for an extensive sweep over input formats and filtering strategies, revealing important interactions between these design choices and the oracle's performance.

Input format options:

Code: The full source code of opponent policies is included in the prompt. This gives the LLM maximum information but at maximum context cost.
Description: Natural language summaries of opponent strategies are included instead of code. These summaries are themselves generated by an LLM analyzing the opponent code. The paper does not detail the summarization prompt, but the existence of docstrings and comments in CSRO policies makes this feasible.
No opponent input: An ablation where the prompt contains only game rules and API—no information about opponents at all. This serves as a lower bound.

Filtering options:

Top 5: Include only the 5 opponents with highest probability in the equilibrium mixture $\sigma$ .
Min support: Include only opponents with probability above some threshold (not explicitly specified, but implied to be small enough to exclude near-zero-probability strategies).
No filter: Include all opponents in the support of $\sigma$ .

Key findings from the sweep (Table 7):

The best LinearRefinement performance (AggScore 122.1 ± 9.8) comes from Code input with Top 5 filtering. The best ZeroShot performance (AggScore 63.5 ± 11.4) comes from Description input with no filtering. This suggests an interaction effect: iterative refinement can leverage detailed code-level information because the refinement loop provides multiple opportunities to digest and respond to it, whereas single-pass generation performs better with summarized, higher-level descriptions.

The paper hypothesizes (Section 4.1) that "generating a best response from a large set of source code in a single pass is a more complex task for which a textual summary provides a more tractable input." This is plausible: reading and synthesizing a counter-strategy to multiple complete programs requires a level of multi-document comprehension that may exceed the LLM's single-pass capacity, whereas refining iteratively allows the LLM to focus on specific weaknesses identified through evaluation feedback.

The Top 5 filter consistently outperforms Min support across most variants. The paper's explanation is that "the meta-game equilibrium often concentrates its probability mass on the most recent, single-best counter-policy," so Min support filtering may provide only one opponent as context—"leading to overfitting, where the oracle generates a narrow policy that is highly effective against that one opponent but brittle against the wider, low-probability population." The Top 5 filter ensures a minimum diversity of opponents in the prompt, encouraging more generalizable strategies.

Summary of Design Choices and Their Justifications

Python as the policy language: Python is both the language the LLM is most proficient in (due to its prevalence in pretraining data) and the language of the game environment (OpenSpiel has Python bindings). This eliminates any translation step between the LLM's output and the execution environment.

Class-based policy representation: The Agent class (RRPS) and RepeatedLeducPokerBot class (Leduc) encapsulate both strategy logic and memory state. The act method's signature is fixed, making it easy to integrate into the game simulator. The class structure also encourages modularity—sub-methods for different strategic components, which aids both interpretability and the LLM's ability to generate structured code.

Stateful rather than stateless policies: In repeated games, the agent's history of past interactions is the primary source of information about opponent tendencies. A stateless policy (mapping only the current observation to an action) cannot model or exploit opponent patterns. The code policy's explicit state variables make this memory transparent.

Separate receive_outcome and restart methods for Leduc: In the repeated Leduc setting, the agent needs to learn across multiple hands while also knowing when a new hand starts (to reset hand-specific state but retain opponent-modeling state). The restart method signals hand boundaries; the receive_outcome method provides the final game state for learning. This separation is necessary because the agent must distinguish between inter-hand learning and intra-hand decision-making.

Equilibrium mixture $\sigma$ as the opponent representation: Rather than providing the LLM with an unstructured list of all previous policies, CSRO uses the equilibrium mixture to identify which opponents currently matter. This is consistent with the game-theoretic foundation: if an opponent has zero weight in the equilibrium, it means that even if it were available, no rational player would use it. Filtering by equilibrium support is thus a theoretically-motivated form of opponent selection.

Code policy as both strategy and documentation: The prompts explicitly instruct the LLM to generate "self-explaining code with detailed comments and a docstring describing the intended strategy" (Section 2.2). This means interpretability is not a post-hoc analysis but a first-class objective during generation. The LLM is asked to articulate its strategic reasoning in the code's documentation, creating a policy whose comments are part of the strategic artifact rather than an external explanation.

4. Key Insights and Innovations

Innovation 1: Reframing Equilibrium Computation as Program Synthesis, Not Numerical Optimization

The paper's most fundamental conceptual move is not architectural but ontological: it changes what kind of object a "best response" is. For the entire history of PSRO and its variants (Lanctot et al., 2017; Mcaleer et al., 2020; Vinyals et al., 2019), a best response has been understood as the solution to a numerical optimization problem — find parameters $\theta$ that maximize expected utility against a mixture of opponents, using gradient descent on millions of sampled trajectories. The output is a weight tensor. The process is opaque; the artifact is opaque.

CSRO reframes the best response as a program synthesis task: given a specification (game rules, opponent descriptions, API), produce source code that implements a strategy satisfying the specification. This is not a different optimization algorithm for the same objective — it is a different kind of objective, one where the search space is a discrete set of syntactically valid programs in a Turing-complete language, the search operator is an LLM's in-context reasoning, and the fitness signal is closed-loop evaluation feedback.

Why does this matter beyond the obvious interpretability benefit? Because it shifts the unit of analysis from parameter vectors to program structure. A neural policy's "strategy" is distributed across millions of weights and can only be interrogated through behavioral testing. A code policy's strategy is an explicit control flow: if-then-else branches, modular subroutines, named data structures, and human-readable comments. This makes strategy compositional — components can be understood, verified, and modified independently — in a way that neural networks fundamentally resist. The paper demonstrates this concretely: the best RRPS policy contains a _predict_meta_imitation method implementing second-order theory of mind, with a docstring explaining its logic. You can read that method in isolation and understand exactly what game-theoretic concept it implements. No saliency map or probing classifier can provide equivalent insight into a neural network's internal computations.

This reframing also changes the scaling regime of best-response computation. Deep RL oracles scale with environment complexity — more states, longer horizons, richer observations mean more training samples. LLM-based oracles scale with the LLM's pretrained knowledge and context window capacity — they don't need to learn the game from scratch because they already understand concepts like "bluffing," "Markov model," "equity calculation," and "counter-strategy" from pretraining. The number of environment interactions is reduced to evaluation only (determining whether the generated strategy actually works), not exploration. The paper quantifies this efficiency gap implicitly: LinearRefinement uses at most 11 LLM calls per PSRO iteration (one initial generation plus up to 10 refinements), compared to the millions of game simulations needed to train an IMPALA-based neural oracle (the PSRO-IMPALA baseline, which the authors had to sweep hyperparameters over — Table 3).

The paper explicitly distinguishes this from prior work that used LLMs for game-playing (Lanctot et al., 2023; Kempinski et al., 2025; Gemp et al., 2024): those approaches query the LLM at decision time to produce actions or strategic prompts. CSRO queries the LLM at design time to produce a standalone policy artifact. This is a categorical difference: the LLM is not the player; the LLM is the policy designer. The player is ordinary Python code that runs without any LLM involvement at inference time — "the number of LLM calls only grows linearly with the number of iterations (e.g., K = 20 in our experiments)" (Section 6), not with the number of game turns (which would be 1,000 per episode in RRPS).

This is a fundamental shift, not an incremental improvement over LLM-PSRO or any prior PSRO variant. It redefines the nature of the oracle, not just its implementation, and opens a research direction where program synthesis tools (LLMs, program synthesizers, evolutionary code search) are the primary mechanism for strategy discovery in multi-agent systems.

Prior LLM-PSRO work (Bachrach et al., 2025) treated the oracle as a single-pass generator: prompt the LLM, get a policy, add it to the population. The paper identifies this as a critical weakness — there is "no inner feedback loop to iteratively refine the candidate policy" (Section 5). CSRO introduces a hierarchical optimization structure: an outer loop that builds the population (PSRO iterations) and an inner loop that hardens each candidate before it enters that population (intra-iteration refinement).

This is not a minor engineering addition. It represents a diagnostic insight about what can go wrong when an LLM generates game strategies from a prompt alone. The LLM can produce syntactically valid, logically coherent, and well-commented code that nonetheless loses against the opponent mixture. The failure modes are not random bugs — they are strategically plausible but empirically wrong: the LLM might overestimate the effectiveness of a particular tactic against the described opponents, fail to account for interaction effects between opponent strategies in the mixture, or produce a strategy that works against one opponent but is catastrophically exploited by another. Without a feedback loop, these failures are invisible — the policy enters the population with a convincing docstring and mediocre performance.

The refinement loop converts the oracle from an open-loop text generator into a closed-loop optimization system where the generated policy is validated against the exact objective it is supposed to maximize (Equation 1). This is the same principle that makes standard PSRO's RL oracle work — the oracle maximizes expected utility through interaction, not through introspection — but applied to the program generation domain. The paper's results quantify the value of this feedback: LinearRefinement achieves an AggScore of 122.1 ± 9.8 versus 63.5 ± 11.4 for ZeroShot in RRPS (Table 1). That is a ~2× improvement from adding the inner loop.

The three refinement mechanisms (ZeroShot, LinearRefinement, AlphaEvolve) are not just points on a performance curve. They represent different philosophies about how to use an LLM as an optimizer:

ZeroShot treats the LLM as an oracle in the classical sense — an all-knowing entity whose single utterance is authoritative. This is the default assumption in most LLM-as-reasoning-engine work.
LinearRefinement treats the LLM as a coachable expert: it can produce a good first draft, but benefits from seeing its mistakes and trying again. This aligns with how human programmers work — write, test, debug, revise.
AlphaEvolve treats the LLM as a mutation operator in an evolutionary search, where the LLM's creativity provides the variation and the score function provides the selection pressure. This is a fundamentally different role — the LLM is not the decision-maker but the generator of hypotheses that are tested externally.

The paper's finding that AlphaEvolve minimizes exploitability while LinearRefinement maximizes aggregate score (Table 1) is not just a hyperparameter tuning result. It suggests that different refinement mechanisms optimize for different properties of the equilibrium. Evolutionary search with explicit diversity maintenance (AlphaEvolve's subpopulations) naturally favors robustness — the fitness function is expected utility against the mixture, which is dominated by the strongest opponents with the highest equilibrium weight. This drives exploitability down. The linear feedback loop, by contrast, accepts any policy that achieves $u \geq 0$ and terminates — it produces policies that are "good enough" rather than maximally robust, which leaves room for exploitation of weaker opponents in the broader population and yields higher PopReturn.

This is a novel diagnostic concept: the inner-loop optimization criterion shapes the strategic character of the discovered equilibrium, not just its quality. Future work on LLM-driven equilibrium computation should choose refinement mechanisms based on what kind of equilibrium properties they value, not just raw performance.

Innovation 3: Demonstrating That Programmatic Policies Can Achieve Game-Theoretic Soundness Against Standardized External Baselines

The paper provides something that prior LLM-PSRO work explicitly lacked: external validation against standardized, independently-curated evaluation populations. LLM-PSRO (Bachrach et al., 2025) only reported self-play performance — how well policies performed against the population they were trained within. The Chinchilla-based LLM agent (Lanctot et al., 2023) was evaluated against the 43-bot RRPS population, but it queried the LLM at every turn, making it a different class of approach (online LLM reasoning vs. offline policy synthesis).

CSRO is the first to show that code policies generated by an LLM within a PSRO loop can achieve competitive or superior performance against hand-crafted, competition-grade heuristic strategies that the policies were never explicitly trained against. In RRPS, CSRO-LinearRefinement (AggScore 122.1) is competitive with a 27B-parameter Gemma 3 agent (AggScore 126.0) that uses 1,000 LLM calls per game. In Leduc Hold'em, CSRO-AlphaEvolve achieves an AggScore of 44.9 ± 4.1, exceeding the CFR+ Nash equilibrium baseline (39.8) and dramatically outperforming PSRO-IMPALA (-45.0). These are not self-play numbers — they are generalization metrics against external opponents (the 43 RRPS competition bots; a CFR+ Nash agent plus AlwaysCall and AlwaysFold for Leduc).

Why does external validation matter? Because self-play evaluation in iterative equilibrium computation can be misleading. A PSRO population may achieve low exploitability against itself — the equilibrium mixture is computed from the empirical game matrix of the population's internal matchups. But this tells you nothing about whether the policies generalize to strategies not represented in the population. The 43-bot RRPS population tests exactly this: can the CSRO-discovered strategy beat a diverse set of heuristic strategies spanning Markov models, frequency analyzers, neural-like systems, and cognitive architectures, many of which use qualitatively different approaches than anything the PSRO loop would have generated? The positive result (LinearRefinement and AlphaEvolve both achieve positive AggScore, meaning they beat the average bot while being less exploitable than the worst-case bot's exploitability) suggests that the LLM's pretrained strategic knowledge generalizes beyond the specific code patterns it might have memorized from its training data.

The PSRO-IMPALA baseline provides a crucial comparison point. Its catastrophic performance (AggScore -532.1 in RRPS, -45.0 in Leduc) is not a failure of PSRO per se — it reflects the extreme sample inefficiency of training a recurrent neural policy from scratch in a repeated game with a 1000-round horizon and partial observability. The LLM oracle, by contrast, starts with a strong prior about what constitutes a reasonable strategy and refines from there. This is not a head-to-head algorithmic comparison (the two oracles use vastly different computational resources and priors) but rather a demonstration that pretrained strategic knowledge can substitute for millions of environment interactions when the domain is within the LLM's knowledge distribution.

The paper explicitly acknowledges the pretraining-data concern — "the LLM's pre-training data likely contains knowledge of game strategies for a classic game like Rock-Paper-Scissors" (Section 6) — but argues that "the oracle must synthesize a novel best response to a specific, dynamically generated mixture of programmatic opponents provided in-context," which goes beyond pattern retrieval. The Leduc results strengthen this argument: Leduc Hold'em is a synthetic research game with no real-world competitive tradition comparable to RRPS, making it less likely that the LLM has memorized specific strategies. Yet CSRO-AlphaEvolve achieves exploitability of 4.4 ± 0.6 against CFR+, demonstrating genuine strategic synthesis rather than recall.

This contribution is incrementally novel in its specific empirical findings but diagnostically important because it establishes the validity of the program-synthesis approach to equilibrium computation against an external standard, which no prior work had done. Without this validation, CSRO would be an interesting demo of LLM code generation within a self-play loop — interesting, but unproven. With it, CSRO is a credible alternative to deep RL oracles for domains where external strategy diversity is a relevant evaluation criterion.

Innovation 4: Verifying That Interpretability Enables the Discovery of Compositional, Higher-Order Strategic Reasoning

The paper makes an argument that goes beyond "code policies are easier to read than neural networks." The qualitative analysis (Section 4.3) demonstrates that the code-generation paradigm enables strategic structures that would be difficult to specify as optimization objectives but emerge naturally when the LLM is asked to "reason about these opponents, and come up with a strategy that can exploit them" (Supplementary Materials Listing 3, line 115).

The best RRPS policy implements second-order theory of mind — it not only models the opponent's behavior (first-order: "what will the opponent do next?"), but models the opponent's model of its own behavior (second-order: "what does the opponent think I will do next?"). This is implemented as a _predict_meta_imitation method that identifies which of its own predictive experts is currently most successful — the assumption being that the opponent is likely using a similar predictive model — and then simulates the opponent's prediction of its own next move, playing the counter to that prediction. This is not a vague analogy to theory of mind; it is an explicit algorithmic implementation that can be read, verified, and tested.

Similarly, the best Leduc policy synthesizes dynamic expected value calculation that adapts to opponent tendencies. It tracks the opponent's folding probability through a statistical model updated via receive_outcome, computes equity (probability of winning at showdown) weighted by the opponent's observed action patterns, and then calculates the expected value of raising as $P(\text{fold}) \times \text{pot} + P(\text{call}) \times (\text{equity} \times \text{pot\_after\_call} - \text{cost\_to\_raise})$ . Against AlwaysCall, the learned folding probability approaches zero, causing the EV calculation to simplify to a pure value-betting strategy. Against AlwaysFold, the folding probability approaches 100%, causing the EV of raising to equal the pot size regardless of hand — a pure bluffing strategy. This adaptation is not an emergent property of a black-box network; it is directly encoded in the policy's source code and verifiable by inspection.

The significance of this finding is not that LLMs can write code with theory-of-mind components — that is an expected capability given their pretraining. The significance is that the code-generation paradigm makes these higher-order reasoning structures naturally expressible and discoverable within the PSRO framework. A deep RL oracle optimizing expected utility against $\sigma$ might also learn second-order opponent modeling (recurrent networks can in principle represent arbitrary Turing-complete computations), but you would have no way to know that it had done so, or to verify that its implementation was correct. The interpretability of CSRO policies is not just a post-hoc benefit for human stakeholders; it enables a different class of strategic reasoning to be explicitly targeted and verified during the generation process.

This also suggests something about the compositional expressiveness of code versus weight matrices. Modular strategic components — an ensemble of 32 experts, a theory-of-mind meta-predictor, an expected value calculator with dynamic opponent modeling — are natural to specify, compose, and debug in code, but extremely difficult to induce through scalar reward signals in weight space. The code-generation paradigm may therefore be capable of producing more sophisticated strategies than RL-based oracles in domains where strategic complexity comes from composition of interpretable sub-strategies rather than from massive pattern recognition (e.g., recognizing thousands of StarCraft unit configurations from pixels, where neural networks excel).

This is a diagnostic insight rather than an incremental technical contribution. It reframes interpretability not as a constraint that must be traded off against performance, but as a capability enabler that makes certain forms of strategic reasoning accessible to the optimization process. The paper's qualitative evidence for this is suggestive but not conclusive — they show that the best policies contain these structures, not that RL oracles fail to discover equivalent structures. But the conceptual argument is clear and testable: if strategic sophistication in a domain comes from compositional reasoning (as in poker, negotiation, and many multi-agent settings), then optimizing in code space may be fundamentally more effective than optimizing in weight space, independent of the interpretability benefits.

Innovation 5: Context Abstraction as a Principled Scalability Mechanism for Multi-Agent LLM Systems

The paper introduces a specific technical mechanism — using LLM-generated natural language summaries of opponent strategies, combined with equilibrium-support-based filtering, to keep prompts within context limits — but the conceptual contribution is broader than the mechanism itself. It establishes context abstraction as a first-class design consideration for LLM-based multi-agent systems where the number of interacting components (policies) grows over time.

The problem is structural and unavoidable: in any iterative population-based method, the number of policies in the population grows linearly with iterations. At iteration $k$ , there are $k$ policies, each with potentially hundreds of lines of source code. Including all of them in the prompt is impossible beyond small $k$ — the context window of even the largest models (Gemini 2.5 Pro supports 1M tokens, but including 20+ full bot implementations plus game rules and API specifications would be unwieldy and expensive). LLM-PSRO (Bachrach et al., 2025) did not address this; it relied on full source code inclusion, implicitly limiting the approach to small populations.

CSRO's solution has two components that are independently motivated:

Equilibrium-support filtering is theoretically grounded: if an opponent has zero (or near-zero) probability in the equilibrium mixture, then a rational player would never use it, and the oracle does not need to consider it when computing a best response. This is not a heuristic shortcut — it follows directly from the definition of the best-response objective (Equation 1), where opponents are weighted by $\sigma(\pi')$ . Opponents with $\sigma(\pi') \approx 0$ contribute negligibly to the expected utility and can be omitted with bounded error.

Natural language summarization is practically motivated but interacts with the code-policy paradigm in a non-obvious way. Because CSRO policies contain docstrings and comments describing their intended strategy (the prompt explicitly requests "self-explaining code"), summarizing a policy is not a lossy compression of its source code — it is extracting the intended strategy that the policy's author (the LLM at a previous iteration) explicitly wrote down. This means the summary is likely to capture the strategic essence, even if it misses implementation details. The paper's finding that Description input outperforms Code input in ZeroShot settings (AggScore 63.5 vs. -54.3, Table 7) supports this: the strategic essence may be more informative than the raw code for single-pass generation, where the LLM cannot iteratively digest detailed implementations.

The conceptual contribution is not the specific mechanisms (filtering by probability, summarizing with another LLM call) but the recognition that context budget is a resource that must be allocated strategically in multi-agent LLM systems. As populations grow, you cannot include everything. You must decide what information about the opponent population is most decision-relevant and provide only that. Equilibrium-support filtering and strategic summarization are principled answers to this question, grounded in game theory (support filtering) and the program-synthesis paradigm (summarization of self-documenting code). Future systems that use LLMs to reason about larger sets of interacting agents will need similar abstractions — this paper provides both a concrete implementation and a conceptual framework for designing them.

This is an incremental contribution in its technical mechanism but a foundational insight for scaling LLM-based multi-agent systems. Without it, CSRO (and any similar approach) would be limited to small populations where full source code fits in context — a severe restriction that would prevent application to the complex, many-opponent settings that PSRO was designed to handle.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on two game environments implemented in OpenSpiel (Lanctot et al., 2019). The first is Repeated Rock-Paper-Scissors (RRPS), where players compete for 1,000 consecutive rounds of the standard stage game. The second is Repeated Leduc Hold'em poker, where players play 100 hands of Leduc Hold'em, with the dealer role alternating between rounds. The RRPS evaluation population consists of 43 hand-coded, heuristic strategies from the international RRPS programming competitions (Billings, 2000a,b), spanning simple stateless bots (randbot, rockbot), Markov chain predictors (markov5), frequency analyzers (freqbot2), neural-like systems (sunNervebot), and the competition winners iocainebot and greenberg. The Leduc evaluation population consists of three opponents: a CFR+-computed Nash equilibrium strategy for the single-hand game (Tammelin, 2014), and two heuristic strategies with easy-to-detect patterns (AlwaysCall, AlwaysFold).
Base model(s). All CSRO experiments use Gemini 2.5 Pro (et al., 2025) as the code-generating LLM oracle. The PSRO-IMPALA baseline uses a deep LSTM-based recurrent network trained with the IMPALA algorithm (Espeholt et al., 2018). The LLM Agent baseline adapts the approach from Lanctot et al. (2023), substituting the original Chinchilla models for Gemma 3 models (Team, 2025) at scales of 270M, 1B, 4B, and 27B parameters. The authors argue that Gemini 2.5 Pro is a state-of-the-art model for code generation, and that PaLM 2-S* from the original Chinchilla-based approach is replaced to "benchmark a more recent and accessible model on this task."
Metrics. The paper adopts three metrics from Lanctot et al. (2023). Population Return (PopReturn) measures average performance against the evaluation population P: PopReturn(π) = E_{π′∼P}[u(π, π′)]. Higher is better — it captures generalization capability. Within Population Exploitability (PopExpl) measures worst-case performance: PopExpl(π) = −min_{π′∈P} u(π, π′). Lower is better — it captures how badly a single opponent can exploit the policy. Aggregate Score (AggScore) balances both: AggScore(π) = PopReturn(π) − PopExpl(π). Higher is better — a policy can have high PopReturn but also high PopExpl, meaning it beats most opponents but is catastrophically exploited by one; AggScore penalizes this. For Leduc Hold'em, metrics are computed against P = {CFR+, AlwaysCall, AlwaysFold}.
Baselines. Four baselines are compared against CSRO variants:
1. PSRO-IMPALA: A standard PSRO algorithm where the best response oracle is a deep LSTM-based network trained with IMPALA. The recurrent architecture makes it suitable for partially observable repeated games. Hyperparameters (learning rate, hidden sizes, unroll length, entropy cost, batch size, max gradient norm) were swept (Table 3), with best configurations reported in Tables 4 and 5.
2. LLM Agent (Gemma 3): Based on Lanctot et al. (2023), the LLM receives a textual history of previous actions and predicts the opponent's next action, playing the best response to that prediction. Results are reported for 270M, 1B, 4B, and 27B parameter Gemma 3 models (Table 6).
3. Tabular Q-learning (QL, R=10): From Lanctot et al. (2023), a tabular Q-learning agent with recall length of 10 rounds. Reported results are from the original paper.
4. Contextual Regret Minimization (ContRM): From Lanctot et al. (2023), a contextual regret minimizer trained via self-play. Reported results are from the original paper.
5. CFR+: For Leduc Hold'em only, a CFR+ solver run for 10,000 iterations using weighted averaged strategy. This serves as a Nash equilibrium baseline for the single-hand game, applied to every hand of the repeated game.
Generation budget / compute accounting. For CSRO, compute is measured in LLM API calls. ZeroShot uses one call per PSRO iteration. LinearRefinement uses up to M + 1 = 11 calls per iteration (initial generation plus up to 10 refinement steps). AlphaEvolve uses many calls across multiple evolution threads and generations, but the exact count is not specified. Critically, at inference time, the generated policy runs as ordinary Python code with no LLM involvement — a single game of RRPS (1,000 rounds) requires zero LLM calls. This stands in contrast to the LLM Agent baseline, which requires 1,000 model calls per game (one per turn). The paper explicitly frames this efficiency difference: "the baseline LLM agents are invoked on every turn, i.e., requiring 1000 model calls for a single game. In contrast, LinearRefinement generates a complete, reusable policy where the number of LLM calls only grows linearly with the number of iterations (e.g., K = 20 in our experiments)" (Section 6). For PSRO-IMPALA, compute is measured in environment interactions during training (millions of game simulations), making direct FLOPs comparison impractical — the authors instead compare final policy quality after training budgets that were "sufficient for convergence" (implicitly determined by the hyperparameter sweep).
Cross-validation / statistical protocol. All CSRO experiments are run for K = 20 iterations. For LinearRefinement, the refinement budget is M = 10. Results for AlphaEvolve are averaged over 3 seeds; all other CSRO variants are averaged over 5 seeds. The LLM Agent baseline results (Table 6) are averaged over 16 games per bot. Standard deviations are reported for all CSRO and PSRO-IMPALA results. No explicit cross-validation or train/test split is used within the PSRO loop itself — the population P grows cumulatively, and the final equilibrium mixture σ is what is evaluated against the external bot population. This means the meta-equilibrium is computed on the same policies used to generate it, which is standard PSRO practice (the empirical game matrix is the entire available data). The external validation against the 43-bot RRPS population and the Leduc hold-out opponents serves as the generalization test.

Main Quantitative Results

Repeated Rock-Paper-Scissors: Population-Level Performance

The primary results for RRPS are summarized in Table 1 and detailed exhaustively in Table 7 (Supplementary Materials). The headline finding is that CSRO variants achieve competitive or superior performance to all baselines except the largest LLM Agent, while producing interpretable, reusable code policies rather than requiring per-turn LLM queries.

CSRO-AlphaEvolve achieves the lowest exploitability among all CSRO variants: PopExpl = 25.2 ± 20.3, PopReturn = 50.5 ± 1.9, AggScore = 25.4 ± 21.6 (Table 1). This is consistent with the PSRO objective — the oracle minimizes worst-case performance against the equilibrium mixture, which prioritizes robustness over exploiting weak opponents. The high standard deviation on PopExpl (±20.3) indicates substantial seed sensitivity; the best single seed achieved PopExpl = 3.3 (Table 7, "min" column). The mean PopReturn of 50.5 means AlphaEvolve's equilibrium strategy wins approximately 50.5 more rounds than it loses per 1,000-round match against the average bot — a modest but positive edge.

CSRO-LinearRefinement (code, Top 5) achieves the highest aggregate score among all tested CSRO variants: PopReturn = 159.8 ± 7.7, PopExpl = 37.7 ± 10.6, AggScore = 122.1 ± 9.8 (Table 1). This is competitive with the 27B Gemma 3 LLM Agent (PopReturn = 193.2, PopExpl = 67.2, AggScore = 126.0). The best single LinearRefinement run (no filter, code input) achieved PopReturn = 238.3 (Table 7, "max" column), which the paper notes "would have placed it in between 2nd and 3rd place in the competition" (Section 4.3.1) — a remarkable result given that this strategy was generated automatically from an LLM with no access to the competition bots during training.

CSRO-ZeroShot (description) achieves moderate performance: PopReturn = 130.2 ± 15.4, PopExpl = 66.7 ± 25.9, AggScore = 63.5 ± 11.4 (Table 1). This significantly underperforms LinearRefinement, confirming the value of the intra-iteration feedback loop.

PSRO-IMPALA performs catastrophically: PopReturn = −108.9 ± 17.6, PopExpl = 423.2 ± 28.0, AggScore = −532.1 ± 41.5 (Table 1). This means the neural oracle not only fails to find a winning strategy — it loses by a wide margin against the evaluation population on average and is massively exploitable. The paper does not provide a detailed diagnosis, but likely causes include: (a) the 1,000-round horizon with partial observability creates an extremely challenging credit assignment problem for RL; (b) the IMPALA training budget (even after hyperparameter sweeping) was insufficient to discover effective recurrent strategies; (c) the repeated game structure requires opponent modeling that is difficult to learn from scratch with generic reward signals.

LLM Agent baselines (Table 6) show a clear scaling trend with model size: PopReturn increases from 167.0 (270M) to 193.2 (27B); PopExpl improves (decreases) from 85.2 to 67.2; AggScore improves from 81.8 to 126.0. The 27B model's AggScore of 126.0 is the strongest among all methods tested in this paper. However, note the critical efficiency distinction: this agent requires 1,000 LLM calls per game, while CSRO requires zero LLM calls at inference time.

Contextual Regret Minimization (ContRM) and Tabular Q-learning results are reported from Lanctot et al. (2023): ContRM achieves AggScore = 148.5 (PopReturn = 164.8, PopExpl = 16.3), and QL with recall 10 achieves AggScore = −9.1 (PopReturn = −0.5, PopExpl = 8.6). The QL result demonstrates the paper's point about balanced evaluation: QL has very low exploitability but near-zero PopReturn, yielding a negative AggScore worse than playing uniformly at random (randbot).

Repeated Rock-Paper-Scissors: Detailed Oracle Variant Comparison

Table 7 in the Supplementary Materials provides a comprehensive sweep over all combinations of input format (code vs. description vs. no input), filtering strategy (Top 5 vs. Min support vs. none), and oracle mechanism (ZeroShot vs. LinearRefinement vs. AlphaEvolve), ranked by PopExpl. This table reveals several non-obvious patterns:

The benefit of code input depends on iterative refinement. In the ZeroShot setting, description input (AggScore 63.5) dramatically outperforms code input (AggScore −54.3). But in the LinearRefinement setting, code input with Top 5 filtering achieves the best overall AggScore (122.1). The paper hypothesizes that "generating a best response from a large set of source code in a single pass is a more complex task for which a textual summary provides a more tractable input" (Section 4.1), while the refinement loop allows the LLM to iteratively digest and respond to detailed opponent code.

Top 5 filtering consistently outperforms Min support. For LinearRefinement (code), Top 5 achieves AggScore 122.1 versus Min support at −125.8. For ZeroShot (desc.), Top 5 achieves 55.8 versus Min support at 46.9. The paper explains: "the meta-game equilibrium often concentrates its probability mass on the most recent, single-best counter-policy," so Min support filtering may provide only one opponent as context — "leading to overfitting, where the oracle generates a narrow policy that is highly effective against that one opponent but brittle against the wider, low-probability population" (Section 4.1). Top 5 ensures a minimum diversity of opponents in the prompt.

The no-opponent-input ablation is catastrophic. The ZeroShot variant with no opponent information achieves PopReturn = 135.3 ± 10.2 but PopExpl = 614.2 ± 60.8, yielding AggScore = −478.9 ± 70.2 (Table 7, last row). This demonstrates that opponent conditioning is "the most important component of the prompt for generating robust strategies" (Section 4.1). Without knowing what opponents do, the LLM generates strategies that beat weak opponents but are extremely exploitable themselves.

LinearRefinement with no filter (code) achieves the highest single-run PopReturn. One seed of LinearRefinement (code, no filter) achieved PopReturn = 238.3 (Table 7, "max" column) — the best single-run performance across all variants. However, the same variant shows high variance: PopExpl ranges from 15.7 to values exceeding 100 across seeds, and the mean PopExpl is 83.2 ± 27.4. This suggests that without filtering, the LLM sometimes overfits to specific opponents and produces brittle strategies.

AlphaEvolve shows the lowest variance in PopReturn (±1.9) but remains competitive in worst-case performance. Its design — distributed evolutionary search with multiple subpopulations — explicitly targets robustness through diversity maintenance, and the low variance in PopReturn suggests this mechanism works reliably across seeds. The high variance in PopExpl (±20.3) indicates that even AlphaEvolve occasionally produces a population missing a counter to some specific opponent in the evaluation set.

Repeated Leduc Hold'em Poker

Table 2 presents the primary results for repeated Leduc Hold'em poker, with a detailed per-opponent breakdown in Table 8. The evaluation population consists of three opponents: CFR+ (the single-hand Nash equilibrium strategy), AlwaysCall, and AlwaysFold.

CSRO-AlphaEvolve achieves the strongest overall performance: PopReturn = 49.3 ± 3.7, PopExpl = 4.4 ± 0.6, AggScore = 44.9 ± 4.1 (Table 2). This exceeds the CFR+ baseline (PopReturn = 39.8, PopExpl = 0.0, AggScore = 39.8) in aggregate score, meaning the CSRO-discovered equilibrium strategy achieves enough additional exploitation against the heuristic opponents (AlwaysCall and AlwaysFold) to more than compensate for its small exploitability against CFR+ (4.4 ± 0.6).

The correlation between oracle strength and equilibrium quality is monotonic. Across all three metrics, the ranking of CSRO variants follows the power of the best-response oracle: AlphaEvolve > LinearRefinement > ZeroShot (Table 2). AlphaEvolve achieves PopExpl 4.4, LinearRefinement 9.8, ZeroShot 19.6; AlphaEvolve achieves AggScore 44.9, LinearRefinement 34.0, ZeroShot 20.7. This directly validates the paper's claim that "both the average-case (PopReturn) and worst-case (PopExpl) performances of the CSRO variants directly correlates with the strength of the best-response oracle" (Section 4.2).

PSRO-IMPALA again performs poorly: PopReturn = 13.3 ± 6.9, PopExpl = 58.4 ± 3.3, AggScore = −45.0 ± 10.1 (Table 2). The high PopExpl indicates that the IMPALA-trained policy is heavily exploited by CFR+ (confirmed in Table 8: return against CFR+ is −58.4 ± 3.3). This reinforces the RRPS finding that training a recurrent neural oracle from scratch for repeated imperfect-information games is extremely challenging.

Per-opponent analysis reveals strategic specialization (Table 8). CSRO-AlphaEvolve achieves a return of 110.3 ± 9.7 against AlwaysCall — dramatically higher than CFR+ (62.1 ± 0.8) and PSRO-IMPALA (57.7 ± 3.3). The qualitative analysis explains this: AlphaEvolve's policy learns that AlwaysCall's folding probability is near zero, converts its expected value calculation to a pure value-betting strategy, and only raises with strong hands. This is a super-Nash exploitation — CFR+ plays the equilibrium strategy and wins 62.1 per 100 hands against AlwaysCall, but the CSRO policy specifically adapts to the opponent's predictability and nearly doubles that return.

Against AlwaysFold, LinearRefinement achieves the highest return (57.3 ± 8.8), slightly exceeding CFR+ (57.4 ± 0.2) and substantially exceeding AlphaEvolve (42.0 ± 3.2). The qualitative analysis explains that the policy learns AlwaysFold's folding probability is near 100%, converting the EV of raising to equal the current pot size — making bluffs always profitable regardless of hand strength.

Against CFR+, AlphaEvolve achieves the least-negative return (−4.4 ± 0.6), significantly better than LinearRefinement (−9.8 ± 3.0) and ZeroShot (−19.6 ± 2.1), and dramatically better than PSRO-IMPALA (−58.4 ± 3.3). Since CFR+ is by definition unexploitable in the single-hand game, no strategy can achieve positive expected value against it in expectation over hands — the best one can do is approach zero. AlphaEvolve's −4.4 indicates it is close to Nash equilibrium play against CFR+, leaking only a small amount due to its exploitative adaptations against the other opponents.

The different oracles discover non-dominated strategies. The paper explicitly notes that "various CSRO oracles find different non-dominated strategies" (Section 4.2): LinearRefinement exploits AlwaysFold better than AlphaEvolve (57.3 vs. 42.0), but AlphaEvolve minimizes exploitability against CFR+ better (4.4 vs. 9.8). This means the choice of oracle mechanism shapes the strategic character of the discovered policy — evolutionary search favors robustness, targeted feedback refinement favors exploitation — and the PSRO meta-solver's equilibrium mixture then balances these strategies. This is a concrete demonstration of the paper's implicit claim that the refinement mechanism is not just a performance optimizer but a strategic shaper.

Ablation Studies and Robustness Checks

Input format (code vs. description): The choice of providing opponent source code versus natural language summaries substantially impacts performance, and the effect interacts with the oracle variant. In ZeroShot, description input achieves AggScore 63.5 ± 11.4 while code input achieves −54.3 ± 118.7 (Table 7). In LinearRefinement, code input with Top 5 filtering achieves the best AggScore (122.1 ± 9.8), while description input achieves 67.7 ± 21.4 with no filter and 140.9 ± 22.2 with Top 5 filtering (but at cost of higher PopExpl: 185.2 vs. 37.7). This confirms that the optimal input representation depends on whether the oracle can iteratively process the information.

Filtering strategy (Top 5 vs. Min support vs. none): For LinearRefinement (code), Top 5 achieves AggScore 122.1, no filter achieves 84.5, and Min support achieves −125.8 (Table 7). The catastrophic failure of Min support filtering — despite being theoretically justified (only opponents with equilibrium support matter) — suggests that including too few opponents (often just one) leads to overfitting. For ZeroShot (desc.), the pattern is less extreme: Top 5 achieves 55.8, no filter achieves 63.5, and Min support achieves 46.9. The paper's hypothesis about equilibrium concentration explaining the Min support failure is plausible but not directly verified (no analysis of the actual equilibrium distributions is provided).

No opponent input (critical ablation): Removing all opponent information from the prompt produces AggScore of −478.9 ± 70.2 (Table 7). This is worse than a uniform random strategy by a wide margin. The finding is methodologically important because it demonstrates that the LLM is not simply retrieving a memorized strong RRPS strategy from pretraining — if it were, removing opponent descriptions would still produce a competent generic strategy. Instead, the LLM produces a strategy that works well against weak opponents (PopReturn 135.3) but is catastrophically exploitable (PopExpl 614.2), indicating that opponent conditioning is necessary for the LLM to reason about robustness.

Oracle mechanism (ZeroShot vs. LinearRefinement vs. AlphaEvolve): This is the central structural ablation. The three mechanisms form a clear performance hierarchy for exploitability minimization: AlphaEvolve (PopExpl 25.2) < LinearRefinement (37.7 with best config) < ZeroShot (66.7 with best config). For aggregate score maximization, the ordering is different: LinearRefinement (122.1) > AlphaEvolve (25.4) > ZeroShot (63.5). This non-monotonic relationship demonstrates that stronger optimization does not uniformly improve all metrics — it shapes which properties of the equilibrium are prioritized.

LLM Agent model scale (Table 6): For the Gemma 3 LLM Agent baseline, performance scales with model size across all three metrics. PopReturn: 167.0 (270M) → 162.7 (1B) → 187.2 (4B) → 193.2 (27B). PopExpl: 85.2 → 141.3 → 87.8 → 67.2. AggScore: 81.8 → 21.4 → 99.4 → 126.0. The non-monotonic jump at 1B (PopExpl worsens to 141.3) suggests that mid-scale models may have sufficient pattern-matching capability to attempt opponent modeling but insufficient reasoning capability to do so robustly, making them more exploitable than both smaller and larger models. This is a common U-shaped scaling phenomenon in LLM capabilities but is not discussed by the authors.

Per-bot performance of LLM Agent (Table 6, detailed rows): The 27B Gemma 3 model shows highly exploitative performance against simple pattern-based bots: it achieves near-perfect returns against deterministic bots (copybot: 969.8, rockbot: 995.0, rotatebot: 997.5, antiflatbot: 991.2). However, it loses significantly against the sophisticated competition winners (iocainebot: −35.6, greenberg: −29.3), as well as against some mid-tier bots (predbot: −67.2, mod1bot: −50.8, halbot: −55.9). This pattern — crushing simple opponents while losing to sophisticated ones — is characteristic of an agent with strong pattern-recognition but limited higher-order strategic reasoning, consistent with the Chinchilla-based findings in Lanctot et al. (2023).

Leduc per-opponent breakdown (Table 8): The analysis of per-opponent returns for Leduc Hold'em reveals strategic differences between oracle variants that aggregate metrics obscure:

Against AlwaysCall: AlphaEvolve (110.3) >> ZeroShot (99.7) > LinearRefinement (83.8) > CFR+ (62.1) > PSRO-IMPALA (57.7). The CSRO policies dramatically outperform both the Nash equilibrium and the neural oracle at exploiting a calling station.
Against AlwaysFold: LinearRefinement (57.3) ≈ CFR+ (57.4) > AlphaEvolve (42.0) ≈ PSRO-IMPALA (40.7) > ZeroShot (41.0). LinearRefinement matches the Nash equilibrium's exploitation of AlwaysFold, while AlphaEvolve — despite its overall superiority — is substantially worse at this specific exploitation task. This is the non-dominated strategy phenomenon in concrete terms.
Against CFR+: All CSRO variants outperform PSRO-IMPALA (−58.4) by a large margin, with AlphaEvolve (−4.4) the closest to the Nash equilibrium baseline (0.0). This confirms that the CSRO oracle can discover near-equilibrium strategies in the repeated game despite never being explicitly trained against CFR+ — the PSRO meta-game against other CSRO policies implicitly guides the population toward low-exploitability strategies.

Critical Assessment

Claim: CSRO converges to low-exploitability equilibria competitive with mature baselines. This claim has strong supporting evidence in Leduc Hold'em (AlphaEvolve PopExpl 4.4 vs. CFR+ 0.0) but mixed evidence in RRPS. In RRPS, the best CSRO exploitability (AlphaEvolve, PopExpl 25.2) is substantially higher than the lowest reported baselines (ContRM 16.3, QL 8.6). However, CSRO's exploitability is dramatically better than PSRO-IMPALA (423.2), demonstrating that the LLM oracle is vastly more effective than deep RL for this domain. The claim's strength depends on the implicit baseline: CSRO achieves reasonable exploitability given zero environment interactions during training, but does not achieve state-of-the-art exploitability compared to methods with full environment access.

Claim: CSRO outperforms a 27B LLM Agent baseline in efficiency while matching performance. The paper states that LinearRefinement's AggScore (122.1) is "competitive with the strongest baseline LLM agent, a 27B parameter Gemma 3 model (126.0)" and emphasizes the efficiency advantage (11 LLM calls to generate a policy vs. 1,000 calls per game to query the agent). This claim is well-supported: the performance gap is small (126.0 vs. 122.1) and the efficiency gap is massive (11 calls vs. 1,000 per game). However, the paper does not report the total number of LLM calls used by CSRO across all 20 iterations — LinearRefinement uses up to 11 calls per iteration, so the upper bound is approximately 220 LLM calls to generate the final population. This is still far fewer than the per-game cost of the LLM Agent (1,000 calls per game × arbitrary number of games), but the fair comparison is total generation cost vs. per-game execution cost, which are incommensurate unless the number of games the policy will be used for is specified. The efficiency advantage is real but its magnitude depends on deployment assumptions.

Claim: CSRO policies are interpretable, as demonstrated by qualitative analysis of strategic components. The paper provides two detailed code listings (Supplementary Materials Listings 1 and 2) with extensive inline commentary explaining strategic logic. The qualitative analysis (Section 4.3) convincingly demonstrates that the code contains named, modular components implementing identifiable game-theoretic concepts (second-order theory of mind, expected value calculation with opponent modeling). However, this is a demonstration of possibility rather than a controlled evaluation of interpretability. The paper does not conduct user studies, does not compare interpretability against neural policy explanation methods (e.g., saliency maps, probing classifiers), and does not quantitatively measure how well human readers can predict policy behavior from code inspection. The claim that CSRO policies are "demonstrably more interpretable" than neural networks is intuitively plausible but empirically unvalidated.

Claim: The choice of refinement mechanism determines the strategic character of the discovered equilibrium. The evidence for this is strong but correlational: AlphaEvolve achieves lowest exploitability while LinearRefinement achieves highest aggregate score, and the per-opponent Leduc breakdown (Table 8) shows different oracles producing different trade-offs. However, the paper does not provide a causal mechanism — why does evolutionary search favor robustness while linear feedback favors exploitation? Is this a general property of the algorithms, or specific to these domains and hyperparameters? An experiment that varied the AlphaEvolve fitness function (e.g., maximizing PopReturn instead of utility against σ) would strengthen this claim but was not conducted.

Weakness: Single model family, no LLM scale sensitivity analysis. All CSRO experiments use Gemini 2.5 Pro. There is no ablation over model size or model family. The LLM Agent baseline (Table 6) shows that model scale matters substantially for RRPS performance (AggScore ranges from 21.4 to 126.0 across Gemma 3 sizes). It is unknown whether a smaller LLM could serve as a CSRO oracle, or whether a different model family would produce qualitatively different strategies. This limits the generality of the findings — the results may be specific to Gemini 2.5 Pro's particular code generation and strategic reasoning capabilities.

Weakness: The 43-bot RRPS population is a fixed evaluation set with potential memorization concerns. The paper acknowledges that "the LLM's pre-training data likely contains knowledge of game strategies for a classic game like Rock-Paper-Scissors" (Section 6). Since the 43 competition bots are publicly available and well-documented, it is possible that the LLM has encountered descriptions or even implementations of some of these bots during pretraining. The paper argues that "the oracle must synthesize a novel best response to a specific, dynamically generated mixture of programmatic opponents provided in-context" which goes beyond retrieval, but the strength of this argument depends on how much the LLM can infer about specific bots from the equilibrium mixture description alone. An experiment with a held-out set of novel bots (e.g., freshly designed strategies not in any public repository) would more rigorously test generalization vs. memorization.

Weakness: Missing comparisons to stronger or more recent RL baselines. The PSRO-IMPALA baseline performs catastrophically, which makes CSRO look strong by comparison. But IMPALA is not state-of-the-art for deep RL in 2025 — modern algorithms like PPO with recurrence, MuZero, or model-based RL might achieve substantially better performance with the same environment interaction budget. Similarly, the paper compares against CFR+ for Leduc but not against standard PSRO with a well-tuned RL oracle — the IMPALA oracle is the only neural baseline provided. A stronger RL baseline would provide a more meaningful performance comparison, even if it doesn't address the interpretability gap.

Weakness: No sensitivity analysis over K (number of PSRO iterations) or M (refinement budget). All experiments use K = 20 and M = 10. It is unknown whether performance converges at 20 iterations, or whether additional iterations would further improve the equilibrium (or degrade it, if later oracles produce weaker strategies). Similarly, the choice of M = 10 for LinearRefinement is unjustified — would M = 5 produce similar results at half the cost? Would M = 20 produce meaningfully better strategies? The paper's claims about computational efficiency are weakened without this analysis.

Weakness: No ablation over the meta-solver. The paper uses a symmetric equilibrium solver (line 4 of Algorithm 1) but does not specify which one or ablate over alternatives (e.g., uniform distribution, α-rank, replicator dynamics). Since the meta-solver determines which opponents receive weight in the equilibrium mixture — and thus which opponents the oracle is optimizing against — the choice of meta-solver likely affects the strategic character of the discovered policies. The paper's own finding that Min support filtering fails because "the meta-game equilibrium often concentrates its probability mass on the most recent, single-best counter-policy" suggests the meta-solver may be producing degenerate mixtures that harm diversity. Exploring different meta-solvers (e.g., ones that encourage support over more policies via entropy regularization) could address this.

Weakness: Leduc Hold'em evaluation population is very small (three opponents). The Leduc results in Table 2 and Table 8 are computed against only three evaluation opponents. This is a much weaker generalization test than the 43-bot RRPS population. The finding that different CSRO oracles produce different per-opponent trade-offs (e.g., AlphaEvolve vs. LinearRefinement on AlwaysFold) is interesting, but with only three opponents, it is unknown whether these trade-offs generalize to a broader set of Leduc strategies. A larger population of heuristic Leduc bots (analogous to the RRPS competition bots) would provide a more rigorous evaluation.

Missing experiment: Combination of AlphaEvolve and LinearRefinement. The paper presents these as alternative oracle mechanisms, but a natural hybrid — using AlphaEvolve's evolutionary search to explore diverse strategies and LinearRefinement's feedback loop to locally optimize promising candidates — could potentially combine the exploitability-minimizing and return-maximizing properties. The paper does not explore this.

Missing experiment: Direct comparison of policy code length, complexity, and human-evaluated interpretability. The paper claims code policies are interpretable but does not quantify this. A simple experiment: present the code policies and neural policies to human raters and ask them to (a) explain the policy's strategy, (b) predict its behavior in a specific scenario, (c) identify potential weaknesses. This would provide empirical support for the interpretability claim and could reveal whether the self-documenting code is actually more comprehensible than behavior-based explanations of neural policies.

What the experiments do not test: The paper does not test whether CSRO scales to larger games. The two environments (RRPS and Leduc) are both small, analytically tractable games. The paper acknowledges this limitation: "the scalability of CSRO to games with vast, high-dimensional observation spaces (e.g., Stratego or StarCraft) remains an open question" (Section 6). For such games, the observation space cannot be trivially represented in a Python function signature (e.g., representing a StarCraft screenshot as a dict would be absurd), and the strategic complexity may exceed what an LLM can encode in a single code file of tractable length. The paper does not even test CSRO on the full (non-repeated) Leduc Hold'em game tree against a diverse set of poker strategies, which would be a natural intermediate scaling step.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Claims

CSRO's prompt construction mechanism—the function that selects relevant opponent strategies and communicates them to the LLM—requires access to the current meta-game equilibrium σ and the ability to extract opponent source code or generate natural language summaries. On its face, this seems straightforward: the meta-game matrix U and the population P are already available within the PSRO loop. But there is a hidden cost that the paper does not measure or account for: summarizing opponent strategies in natural language requires additional LLM calls, and the performance of downstream oracles depends critically on the quality of these summaries.

The paper states that opponent policies "can be directly included in the prompt or alternatively summarized by another LLM call to provide a high-level description of the strategic behavior the opponent code implements" (Section 2.4.1). This "another LLM call" is not free—it consumes API budget, adds latency, and introduces an additional source of error (poor summaries could mislead the oracle about opponent strategies). The paper does not report how many summarization calls were used, what prompt was used for summarization, or whether failed summaries (e.g., missing key strategic details) contributed to any of the poor-performing runs.

The consequence is that the reported efficiency advantage over LLM Agents—"LinearRefinement generates a complete, reusable policy where the number of LLM calls only grows linearly with the number of iterations (e.g., K=20 in our experiments)" (Section 6)—understates the total LLM cost when description-based input is used. Each iteration may require summarization calls proportional to the number of active opponents in the equilibrium support, plus the refinement calls (up to 11 for LinearRefinement, many for AlphaEvolve). The paper counts oracle refinement calls but not context-preparation calls. For a deployment where opponent populations grow large (the very setting where description-based input becomes necessary due to context limits), the summarization overhead could dominate the total cost.

The evidence in the paper is mixed regarding how much this matters. The best LinearRefinement configuration uses code input with Top 5 filtering (AggScore 122.1, Table 7), which avoids summarization calls entirely—only equilibrium filtering is needed, and that is a cheap operation on the meta-game matrix. However, the paper explicitly argues that code input becomes infeasible for larger populations (context window limits) and that description-based input will be necessary for scalability. The performance of description-based LinearRefinement is substantially weaker: AggScore 67.7 ± 21.4 with no filter, 140.9 ± 22.2 with Top 5 filtering but at cost of high PopExpl (185.2). The paper does not report how many summarization calls were used to achieve these numbers, making it impossible for a practitioner to estimate the true cost of running CSRO with description input.

The mitigation status is: not addressed. The paper does not account for summarization costs in any budget calculation, does not ablate over summarization quality, and does not discuss whether lightweight alternatives (e.g., extracting only docstrings without a full LLM pass, or caching summaries across iterations) could reduce this overhead. The limitation is partially acknowledged indirectly: the paper notes that "managing prompt complexity when facing a large mixture of policies" is the motivation for summarization (Section 2.2), but this is framed as a solution, not as a cost that itself needs optimization.

The Method Has Not Been Demonstrated to Scale Beyond Toy Domains

All experiments in the paper are conducted on two analytically simple, low-dimensional games: Repeated Rock-Paper-Scissors (three actions, effectively a single information state per round with history) and Repeated Leduc Hold'em (six-card deck, two betting rounds, a handful of information sets). Both games are small enough that their full game trees can be solved exactly with standard game-theoretic algorithms (CFR+ converges in 10,000 iterations for Leduc). The paper does not test CSRO on any game that approaches the complexity of the domains that motivated PSRO's development—Stratego, StarCraft, or even mid-scale poker variants like heads-up no-limit Texas Hold'em.

The paper acknowledges this limitation directly: "the scalability of CSRO to games with vast, high-dimensional observation spaces (e.g., Stratego or StarCraft) remains an open question. Representing complex state and opponent strategies within the context length of current LLMs remains a substantial engineering challenge" (Section 6). This is honest but undersells the gap between the test environments and the target domains. In RRPS, the observation is a dictionary with two string keys (my_action, opponent_action). In Leduc, the observation is a structured JSON dict with a few dozen fields. In StarCraft, the observation is a high-resolution image, a minimap, and a set of unit status vectors—none of which can be naturally represented in the prompt format CSRO uses, and none of which an LLM pretrained on text and code can directly "reason about" in the same way it reasons about poker hand rankings.

The consequence is not just that the method might perform worse on larger games—it is that the entire prompt construction paradigm may not transfer. CSRO's prompt communicates opponent strategies by describing their logic: "this bot uses a 5th-order Markov model" or "this bot value-bets with strong hands." For a StarCraft policy represented as a neural network with millions of parameters, there is no such compact strategic description available—any summary would be either vacuous ("this bot tries to win by building units and attacking") or so long as to exceed context limits. The context abstraction mechanism (LLM-generated natural language summaries) relies on the policies being self-documenting code, which is a property CSRO itself creates. If the opponent population includes neural policies (as in any standard PSRO deployment), summarization requires interpreting network behavior from input-output examples—a much harder and less reliable process.

The paper provides no evidence on this scalability question. The ablation over opponent input format (code vs. description vs. no input, Table 7) shows that the prompt construction choices matter significantly even for the small RRPS population. As the number of policies grows and the strategic complexity increases, the risk of prompt degradation—too much information for the LLM to process effectively, or too little to capture strategic nuance—grows as well. The paper's claim that context abstraction "allows our method to scale to complex games where full source code would exceed context windows" (Section 1) is a design intention, not an empirical finding.

The mitigation status is: explicitly acknowledged but not addressed. The limitation is listed as an "open question" in the conclusion with no proposed experiments, intermediate benchmarks, or theoretical analysis of scaling behavior. A natural intermediate step—testing CSRO on a game of intermediate complexity, such as full (non-repeated) Leduc Hold'em against a diverse population of heuristic poker bots, or Kuhn poker with a larger opponent population—was not taken.

The PSRO-IMPALA Baseline Is Too Weak to Support Strong Comparative Claims

The paper's central empirical claim is that CSRO "achieves performance competitive with baselines" (Section 1, abstract) and that "code-generation oracles can compete with mature baselines in established domains" (Section 1). The primary baseline against which CSRO's game-theoretic soundness is evaluated is PSRO-IMPALA—a standard PSRO implementation where the best response oracle is a deep LSTM-based recurrent network trained with IMPALA. This baseline performs catastrophically: AggScore of −532.1 ± 41.5 in RRPS (Table 1), PopExpl of 423.2, and PopReturn of −108.9 ± 17.6. In Leduc Hold'em, PSRO-IMPALA achieves AggScore of −45.0 ± 10.1 with PopExpl of 58.4 (Table 2).

These numbers are not just worse than CSRO—they are worse than a uniform random strategy. A policy that played uniformly at random in RRPS would achieve approximately 0 PopReturn and PopExpl determined by the most exploitative bot in the population (likely a pattern-matching bot that would converge to near-perfect prediction against uniform random, yielding high PopExpl, but not −532). The baseline is so weak that it provides essentially no information about the relative difficulty of the task or the performance ceiling for neural oracles.

The paper acknowledges that training deep RL oracles is "sample-inefficient, requiring millions or billions of game simulations to converge" (Section 1), but does not analyze whether the IMPALA training budget provided was sufficient. The hyperparameter sweep (Table 3) covers six parameters across plausible ranges, and the best configuration was selected for each environment (Tables 4, 5). However, the paper does not report training curves, final training performance, or whether the policies showed any learning progress at all. It is possible that IMPALA with these hyperparameters simply failed to learn in the 1,000-round repeated game setting—a known challenge for recurrent RL with long horizons and sparse rewards—and that a different algorithm (e.g., PPO with recurrence, R2D2, or a model-based approach) or a different reward shaping would have produced a functional oracle.

The consequence is that the paper's headline comparison—"all CSRO variants substantially outperform the PSRO-IMPALA baseline across all three metrics" (Section 4.1)—is true but uninformative. Outperforming a failed baseline does not demonstrate competitiveness with the state of the art in deep multi-agent RL. A practitioner choosing between CSRO and a deep RL approach needs to know how CSRO compares to a functional neural oracle, not to a training run that did not converge. The more meaningful comparisons are against the LLM Agent baselines (which use a different paradigm—online LLM querying, not PSRO) and against the heuristic bots (ContRM, QL) from prior work, but these comparisons are limited: ContRM's AggScore of 148.5 exceeds all CSRO variants, and the 27B Gemma 3 LLM Agent's AggScore of 126.0 slightly exceeds the best CSRO variant (122.1).

The mitigation status is: not addressed. The paper does not attempt to improve the IMPALA baseline, does not test alternative RL algorithms, does not report whether the IMPALA policies showed any learning, and does not qualify the strength of its comparative claims based on the baseline's failure. A fairer comparison—giving the neural oracle an equivalent "knowledge prior" by pretraining on related tasks, or using a more sample-efficient RL algorithm—was not attempted.

Interpretability Is Demonstrated Qualitatively but Not Validated Empirically

The paper's primary motivation for replacing deep RL oracles with LLM code generation is interpretability: CSRO policies are "inherently interpretable, represented by commented source code" (Section 1), and this makes them suitable "for deploying such agents in high-stakes, real-world applications where explainability is crucial" (Section 1). The qualitative analysis (Section 4.3) provides two detailed code listings with inline commentary identifying strategic components—second-order theory of mind, expected value calculation, opponent modeling—and argues that these demonstrate "a level of strategic reasoning and interpretability that is fundamentally absent in opaque, black-box policies."

But the paper provides no empirical evidence that these code policies are actually interpretable to human readers. Interpretability is not a binary property; it is a relationship between an artifact and a human interpreter, and it depends on factors the paper does not measure: the reader's expertise, the clarity of the generated code, the accuracy of the generated comments, the complexity of the strategy, and whether the documented strategy matches the actual behavior.

Consider what a skeptical practitioner would want to know: Can a human reader, given the code policy, correctly predict how it will behave in specific scenarios? Can they identify strategic weaknesses or failure modes by reading the code? Can they modify the policy to change a specific behavior (e.g., make it more aggressive or more conservative) without breaking other components? Does the presence of comments and docstrings actually improve comprehension over reading uncommented code or over reading behavioral descriptions of a neural policy? None of these questions are addressed. The qualitative analysis shows that the generated code contains comments describing strategic logic, but it does not test whether those comments are accurate (do they describe what the code actually does?), whether they are complete (do they cover the most important strategic decisions?), or whether they aid human understanding beyond what could be inferred from the code structure alone.

There is an additional concern specific to LLM-generated documentation: LLMs are known to produce plausible-sounding but factually incorrect explanations—"hallucinations" in the commentary that do not match the code's actual behavior. The paper provides no verification that the docstrings and comments in the generated policies are faithful to the implemented logic. A policy that says "this agent uses a 5th-order Markov model" but actually implements a 3rd-order model due to an off-by-one error is worse than an uncommented policy, because it actively misleads the reader.

The consequence: the paper's central claim—that CSRO addresses the "critical limitation of policy opacity" (Section 1)—is asserted rather than demonstrated. The policies are presented in an interpretable format (Python source code with comments), but whether this format translates into actual human understanding is unmeasured. A practitioner evaluating CSRO for a high-stakes deployment where explainability is a hard requirement (e.g., financial trading, medical decision support, military simulation) would need evidence that human operators can effectively audit, verify, and trust the generated strategies. The paper provides no such evidence.

The mitigation status is: not addressed at all. The paper treats "commented code" as synonymous with "interpretable" and provides no user studies, no behavioral prediction tests, no comparison against neural policy explanation methods, and no verification of comment-code consistency. This is the most significant gap between the paper's motivating claims and its empirical support.

The Relationship Between Pretraining Data and Discovered Strategies Is Uncharacterized

The paper's approach relies fundamentally on the LLM's pretrained knowledge of game strategies, programming patterns, and strategic reasoning. The authors acknowledge this directly: "the LLM's pre-training data likely contains knowledge of game strategies for a classic game like Rock-Paper-Scissors" (Section 6). But they argue that "the oracle must synthesize a novel best response to a specific, dynamically generated mixture of programmatic opponents provided in-context. The success of this process demonstrates a sophisticated capability for in-context strategic reasoning and code generation, not merely pattern retrieval."

This distinction—between retrieval and reasoning—is the crux of the paper's contribution. If CSRO is essentially retrieving memorized strategies from pretraining and adapting them superficially, then its performance on RRPS and Leduc does not demonstrate a general capability for game-theoretic reasoning, and the method would fail on truly novel games or opponent types not represented in the training data. If CSRO is genuinely synthesizing strategies through in-context reasoning, then the method's value proposition is much stronger—it could apply to new domains without retraining.

The paper provides no experiment to distinguish these hypotheses. The 43 RRPS competition bots are publicly available and well-documented; their strategies (Markov models, frequency analysis, pattern matching) are classic approaches discussed extensively in online resources, textbooks, and code repositories. It is entirely plausible that Gemini 2.5 Pro's training data includes descriptions, analyses, or even implementations of these specific bots or their strategic archetypes. The paper's finding that CSRO policies perform well against the bot population is consistent with both retrieval (the LLM recalls counter-strategies it has seen before) and reasoning (the LLM deduces counter-strategies from the bot descriptions in the prompt).

The Leduc Hold'em results partially—but only partially—address this concern. Leduc is a synthetic research game with no real-world competitive tradition, making direct memorization of specific strategies less likely. However, poker strategy concepts (value betting, bluffing, equity calculation, opponent modeling) are extensively discussed in both academic and popular literature, and the LLM has almost certainly encountered implementations of poker agents that use these concepts. The CSRO Leduc policy's use of expected value calculations with dynamic opponent modeling (Section 4.3.2) could be synthesized from general poker knowledge rather than being a genuinely novel strategic discovery.

The consequence of this uncharacterized dependency is that the paper cannot predict on which games CSRO will succeed. For RRPS and Leduc—games with strategies that are well-represented in the LLM's pretraining distribution—performance is strong. For a genuinely novel game with unfamiliar mechanics (e.g., a custom-designed multi-agent simulation for a specific industrial application), there is no evidence that CSRO would produce competent strategies. The LLM would lack the pretrained strategic concepts to draw upon, and the in-context reasoning capability might not be sufficient to derive effective strategies from the game rules alone.

The mitigation status is: acknowledged but not tested. The paper mentions the pretraining data concern in the discussion (Section 6) and argues against it—"the success of this process demonstrates a sophisticated capability for in-context strategic reasoning"—but provides no experiment to validate this interpretation. An experiment that would address this: test CSRO on a synthetic game with novel mechanics that is designed to be absent from pretraining data (e.g., a mathematically-defined game with arbitrary rules generated specifically for the experiment), and compare performance against the same method on games likely present in pretraining data. The paper does not conduct such an experiment. Additionally, the paper does not test with a model known to have less exposure to game strategy content (e.g., a model trained primarily on non-game, non-code corpora), which would help isolate the contribution of pretrained strategic knowledge versus in-context reasoning.

The Refinement Budget $M=10$ and Iteration Count $K=20$ Are Unexplained Hyperparameters with Unknown Sensitivity

All CSRO experiments use $K=20$ iterations of the outer PSRO loop and $M=10$ for the LinearRefinement inner loop budget. These values are stated without justification, sensitivity analysis, or convergence diagnostics. The paper provides no evidence that 20 iterations is sufficient for the meta-game to stabilize, that additional iterations would not yield further improvements (or degradation), or that 10 refinement steps strikes an appropriate balance between policy quality and computational cost.

This matters because both hyperparameters directly control the cost-quality tradeoff that the paper claims as a key advantage. The computational efficiency argument—"the number of LLM calls only grows linearly with the number of iterations (e.g., K=20 in our experiments)" (Section 6)—is parameterized by $K$ . If $K=40$ were needed for convergence, the cost advantage relative to other methods shifts. Similarly, the refinement budget $M$ determines the upper bound on LLM calls per iteration for LinearRefinement ( $M+1=11$ ). If $M=5$ achieves similar performance, the method is cheaper than reported; if $M=20$ is needed for reliable policy quality, it is more expensive.

The consequence is that a practitioner cannot determine the appropriate $K$ and $M$ for a new domain without re-running the full sweep—and the paper provides no guidance on how to choose these values. More critically, the paper's central claims about performance ("competitive with baselines") and efficiency ("reusable policy where the number of LLM calls only grows linearly") are both contingent on these hyperparameters. If convergence requires $K=50$ , the efficiency advantage is halved; if $M=3$ produces equivalent policies, the efficiency advantage is understated.

The paper provides some indirect evidence about $K$ dependence through the iteration-level visualization in Figure 1, which shows the evolution of payoffs and equilibrium support across 20 iterations for an example CSRO run. The figure suggests that the meta-game is still evolving at iteration 20 (new policies continue to be added and the equilibrium mixture shifts), but the paper does not report whether aggregate metrics (PopReturn, PopExpl, AggScore) stabilized. For $M$ , there is no evidence at all—no ablation testing whether performance changes with smaller or larger refinement budgets, and no reporting of how many refinement steps were actually used on average (i.e., how often the loop terminated early because $u \geq 0$ versus exhausting the budget).

The evidence gap is most acute for AlphaEvolve, where the "refinement budget" concept does not apply in the same way (AlphaEvolve uses a distributed evolutionary search with its own internal hyperparameters—population size, number of generations, mutation rate—that are not specified in the paper at all). The paper states that it follows "the method described in (Novikov et al., 2025; Romera-Paredes et al., 2024)" (Section 3.4) but does not report the specific AlphaEvolve configuration used, making the AlphaEvolve results effectively unreproducible without access to the same distributed infrastructure and hyperparameter settings.

The mitigation status is: not addressed. There is no sensitivity analysis for $K$ , no ablation over $M$ , no reporting of convergence diagnostics (e.g., how much the equilibrium mixture changes between iterations, whether the payoff matrix entries stabilize), and no reporting of the actual number of refinement steps used per iteration. The paper implicitly treats $K=20$ and $M=10$ as universal defaults, but provides no evidence that these values are appropriate even for the domains tested, let alone for new domains.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not incrementally improve an existing solver, propose a new meta-game algorithm, or demonstrate that LLMs can play games—all of those were known. What it does is redefine the ontological status of a policy in multi-agent equilibrium computation. A policy is no longer necessarily a tensor of weights; it can be a Python class with named variables, modular subroutines, and human-readable docstrings. This shift may seem cosmetic—after all, both representations compute a mapping from observations to actions—but it changes the epistemic relationship between the designer and the artifact. A neural policy must be tested to be understood; a code policy can be read. The paper makes this concrete: the best RRPS policy contains a method called _predict_meta_imitation with a docstring explaining its second-order theory-of-mind logic. You can audit that logic statically, without running a single game. No saliency map or probing classifier on a neural network provides equivalent access.

This is not yet a paradigm shift—the method is validated on two toy games with observation spaces representable as simple dictionaries, and the scalability to Stratego or StarCraft is explicitly an "open question" (Section 6). But it is a foundational reframing that redirects attention: rather than asking "how can we make neural policies more interpretable through post-hoc explanation?", CSRO asks "what if the policy were its own explanation, because the optimization process that produced it was instructed to make it so?" This shifts interpretability from a downstream analysis problem to a generation-time objective—the LLM is prompted to produce "self-explaining code with detailed comments and a docstring describing the intended strategy" (Section 2.2). The quality of explanation becomes a design choice in the prompt, not a research problem in explainable AI.

The work also resolves a tension between two competing narratives in prior literature. On one side, standard PSRO and its variants (Lanctot et al., 2017; Mcaleer et al., 2020; Vinyals et al., 2019) demonstrated that iterative best-response computation could find approximate equilibria in large games, but produced opaque policies. On the other side, LLM-based game-playing approaches (Lanctot et al., 2023; Kempinski et al., 2025) demonstrated that pretrained language models possess strategic reasoning capability, but consumed LLM calls at inference time on every turn—1,000 queries per game for RRPS—making them computationally impractical for repeated deployment. CSRO synthesizes these narratives: it uses the LLM's strategic reasoning during policy design (one or a few calls per PSRO iteration) to produce a standalone artifact that runs as ordinary code during policy execution (zero LLM calls per game). This is a categorical efficiency improvement: LinearRefinement uses at most 11 LLM calls per iteration, totaling approximately 220 calls for the full 20-iteration run, versus 1,000 calls per game for the LLM Agent baseline.

A diagnostic insight from the paper that should influence future research design is that the choice of intra-iteration refinement mechanism shapes the strategic character of the discovered equilibrium, not just its quality. AlphaEvolve's distributed evolutionary search with diversity maintenance produces the lowest exploitability (PopExpl 25.2 in RRPS, 4.4 in Leduc; Tables 1–2) because its fitness function—expected utility against the equilibrium mixture σ—naturally prioritizes robustness against strong opponents that dominate the mixture. LinearRefinement's conditional feedback loop produces the highest aggregate score (AggScore 122.1; Table 1) because it terminates when utility is non-negative—producing "good enough" policies that leave room for exploiting weak opponents in the broader evaluation population. This is not a hyperparameter tuning artifact; it is a structural consequence of how the optimization is framed. Researchers building on CSRO should not treat refinement mechanisms as interchangeable black-box optimizers—they should select the mechanism based on whether the deployment context values worst-case robustness (AlphaEvolve), average-case generalization (LinearRefinement), or rapid exploration (ZeroShot).

The paper also makes certain research directions less attractive by demonstrating a ceiling effect that was previously only suspected. The PSRO-IMPALA baseline—a recurrent neural oracle trained with IMPALA on millions of game simulations—achieves an AggScore of −532.1 in RRPS (Table 1) and −45.0 in Leduc (Table 2). This is not just worse than CSRO; it is worse than a uniform random strategy. While IMPALA is not state-of-the-art in 2025, the magnitude of the failure suggests that training recurrent policies from scratch for partially-observable repeated games with long horizons is a fundamentally harder problem than the literature acknowledges. The CSRO results provide an existence proof that strong strategies can be discovered for these games—the challenge for deep RL is not the game's strategic depth but the sample complexity of credit assignment over 1,000-round horizons. This suggests that research effort should shift toward methods that incorporate strong priors (through pretraining, demonstrations, or structured exploration) rather than toward incremental improvements in generic model-free RL for these settings.

Finally, the paper establishes context abstraction as a first-class design consideration for any LLM-based multi-agent system where the number of interacting components grows over time. The problem is structural: at iteration k, there are k policies; including all of their source code in the prompt is impossible beyond small k. CSRO's solution—equilibrium-support filtering (only include opponents with non-negligible probability in σ) combined with optional LLM-generated natural language summaries—is both theoretically motivated (filtering follows from the definition of the best-response objective) and practically validated (the ablation in Table 7 shows that including no opponent information produces AggScore −478.9, while Top 5 filtering produces 122.1). Any future system that uses an LLM to reason about growing populations of agents will face this bottleneck, and CSRO provides a concrete template for addressing it.

Follow-Up Research This Work Enables

A difficulty-prediction model for policy synthesis: can we predict which games and opponent mixtures CSRO will succeed on without running the full PSRO loop? The paper demonstrates strong performance on RRPS and Leduc—games whose strategic concepts (Markov models, pattern matching, value betting, bluffing) are well-represented in the LLM's pretraining distribution—but provides no evidence about performance on genuinely novel games. A high-value follow-up would be a systematic study: curate a benchmark of 10–20 games spanning different degrees of "pretraining familiarity" (from tic-tac-toe, which is ubiquitous, to a custom-designed synthetic game with novel mechanics and no online presence), run CSRO on each, and correlate the LLM's zero-shot strategic reasoning quality with downstream equilibrium quality. The independent variable would be a pre-registered measure of the game's presence in pretraining data (e.g., the LLM's perplexity on the game's Wikipedia article, or a knowledge-probing test of game-specific concepts). The dependent variable would be the final equilibrium's PopExpl and AggScore against a standardized external population. A strong finding—positive or negative—would determine whether CSRO is a general method for equilibrium computation or a clever way to leverage memorized strategic knowledge from pretraining.

Combining CSRO with an RL fine-tuning oracle: does RL-based refinement of code policies produce better strategies than either approach alone? The paper establishes that LLM-generated code policies can be competitive (AggScore 122.1 for LinearRefinement vs. 126.0 for a 27B Gemma 3 agent; Table 1) and that deep RL from scratch fails catastrophically (PSRO-IMPALA AggScore −532.1). The natural hybrid is to use CSRO to generate an initial code policy—which already encodes strategic concepts like Markov prediction or equity calculation—and then use RL to fine-tune its numerical parameters (e.g., the decay rate 0.985 in the best RRPS policy, or the opponent-model smoothing parameter α = 1.0 in the Leduc policy) through environment interaction. This would combine the LLM's strategic priors (which RL lacks, causing the IMPALA failure) with RL's ability to optimize continuous parameters from experience (which the LLM can only approximate through textual reasoning). The experiment would compare: (a) CSRO alone, (b) RL from scratch, (c) RL fine-tuning of a CSRO-generated code policy, and (d) CSRO fine-tuned by additional LLM refinement beyond M = 10 iterations. The hypothesis is that (c) achieves the best exploitability while (a) achieves the best interpretability; the trade-off curve would guide practitioners on when to accept reduced transparency for improved robustness.

Cheap difficulty estimation via prompt-only signals: can we determine whether a game is "within the LLM's strategic competence" without running environment evaluations? The paper's context abstraction mechanism requires evaluating policies against the environment to determine which opponents matter (via the equilibrium mixture σ). This is the correct game-theoretic answer but requires expensive environment interactions. An alternative worth exploring: before running any game simulations, prompt the LLM with the game rules and opponent descriptions and ask it to (a) estimate its confidence in generating a winning strategy, (b) predict which opponent characteristics will be hardest to exploit, and (c) propose a strategy skeleton (without full code). Correlate these pre-execution estimates with the eventual equilibrium quality across the benchmark of games proposed above. If the LLM's self-assessed confidence predicts downstream performance, this provides a cheap "difficulty estimator" that could gate whether to run the full CSRO pipeline or fall back to alternative methods. This is methodologically parallel to the difficulty estimation in the example paper (Section 3.2 of the compute-optimal scaling work) but applied to the meta-cognitive problem of whether the LLM-oracle itself is competent for a given domain.

Stress-test: does CSRO degrade when opponent strategies are deliberately designed to exploit LLM-generated code patterns? The paper evaluates CSRO against a fixed, external population of hand-coded bots that were not designed with knowledge of LLM-generated strategies. An adversarial stress-test would involve a human expert (or another LLM) reading the CSRO-generated policies, identifying recurring patterns or weaknesses in the generated code (e.g., the ensemble-of-experts structure with Markov models of orders 1–8 appears in the best RRPS policy; Listing 1), and designing counter-strategies specifically to exploit those patterns. This tests a specific vulnerability of the code-generation paradigm: because policies are human-readable and follow predictable coding patterns, an adversary who studies them can develop targeted exploits. Compare CSRO's robustness under this adversarial evaluation to the robustness of neural policies (which are harder to study but may also exhibit exploitable regularities). A negative result—CSRO policies being more exploitable than neural policies under adversarial inspection—would qualify the paper's interpretability-as-safety narrative and suggest that interpretability is a double-edged sword in adversarial settings.

Meta-solver ablation: which equilibrium concept best supports LLM-generated policy populations? The paper uses an unspecified meta-solver to compute the symmetric equilibrium mixture σ from the empirical payoff matrix U (Algorithm 1, line 4). The paper's own analysis reveals that "the meta-game equilibrium often concentrates its probability mass on the most recent, single-best counter-policy" (Section 4.1), which causes Min support filtering to fail by providing only one opponent as context. This suggests the meta-solver may be producing degenerate mixtures that harm the oracle's ability to generate diverse, generalizable strategies. A systematic ablation would replace the default solver with: (a) uniform distribution over the population (ignoring payoffs, maximizing diversity), (b) softmax over payoffs with a temperature parameter (controlling the exploration-exploitation trade-off), (c) maximum-entropy Nash equilibrium (adding an entropy bonus to encourage support over more policies), and (d) α-rank (Omidshafiei et al., 2019). The dependent variables would be final equilibrium exploitability, population diversity (measured by behavioral distance between policies), and the oracle's ability to generate policies that generalize to the external evaluation population. The hypothesis is that entropy-regularized meta-solvers produce more diverse opponent descriptions in the prompt, which leads to more generalizable generated policies, at the cost of slower convergence in the PSRO outer loop.

Human-interpretability study: do CSRO code policies actually enable better strategy understanding than behavioral descriptions of neural policies? The paper's central claim—that CSRO policies are "inherently interpretable" (Section 1)—rests entirely on qualitative demonstration. A rigorous evaluation would recruit human participants (ideally with varying levels of programming and game-theory expertise), present them with either a CSRO-generated code policy or a behavioral description of a neural policy (derived from extensive testing, e.g., "this policy raises 73% of the time with a King preflop, folds to raises 89% of the time with a Jack postflop"), and measure: (a) accuracy in predicting the policy's next action given a specific game state, (b) ability to identify a deliberate weakness inserted into the policy, (c) time to complete these tasks, and (d) subjective confidence. The hypothesis is that code policies improve accuracy and reduce time for programmers but may not benefit non-programmers; if the advantage only holds for expert audiences, the interpretability claim should be qualified accordingly. A second condition should test whether the generated comments are faithful—do the comments accurately describe what the code does, or do they sometimes contain plausible-sounding but incorrect strategic claims (hallucinations in the self-documentation)? This is critical because misleading documentation is worse than no documentation—it actively harms trust. A follow-up finding that, say, 15% of generated docstrings contain inaccuracies would be a significant qualification to the interpretability advantage.

Practical Applications and Downstream Use Cases

Automated generation of training opponents for reinforcement learning curricula. A persistent challenge in training robust RL agents is designing a diverse curriculum of opponent strategies that force the learner to develop generalizable skills. Standard approaches use human-designed bots, past versions of the agent (self-play), or procedurally generated opponents—all of which require manual design or extensive computation. CSRO provides a turnkey alternative: given a base environment (implemented in OpenSpiel), run the CSRO pipeline for K = 20 iterations to automatically generate a population of diverse, interpretable strategies. The resulting code policies can serve as a training curriculum where the RL agent must learn to beat an increasingly sophisticated set of opponents. The specific benefit over manual design is coverage: the CSRO population in RRPS discovered strategies spanning Markov models (orders 1–8), reactive predictors, periodic pattern matchers, and second-order theory-of-mind reasoning (Listing 1)—a breadth that would require significant human effort to design from scratch. The specific benefit over self-play is interpretability: if the RL agent fails against a particular opponent, the opponent's strategy can be read and understood, enabling targeted curriculum adjustments. The cost is approximately 220 LLM API calls (LinearRefinement with K = 20, M = 10), which is negligible compared to the computational cost of training the RL agent itself.

Regulatory compliance verification for multi-agent financial or bidding systems. In domains where algorithms participate in repeated strategic interactions under regulatory oversight—algorithmic trading, automated bidding in ad auctions, electricity market participation—regulators require evidence that agents are not colluding, manipulating markets, or engaging in prohibited strategic behaviors. Neural network policies provide no such evidence; they can only be tested empirically, which cannot prove the absence of strategic manipulation in all possible market states. CSRO-generated code policies are statically auditable: a regulator can inspect the generated Python code, verify that the act method's logic complies with market rules, and confirm—through code review rather than exhaustive testing—that no prohibited strategic reasoning is implemented. The Leduc policy's expected value calculation (Section 4.3.2, Listing 2) demonstrates the relevant property: a regulator reading the _calculate_action_ev method can directly verify that the agent is computing action values from pot odds and estimated opponent behavior, not from collusive signaling or information not legitimately available. The exploitability numbers provide a complementary guarantee: AlphaEvolve's PopExpl of 4.4 against CFR+ in Leduc (Table 2) means the policy cannot be exploited for more than 4.4 chips per 100-hand match by any opponent in the evaluation set, providing a quantitative bound on worst-case behavior that complements the qualitative code audit. The current limitation is the small scale of tested domains; extending this use case to realistic financial environments would require demonstrating CSRO on games with order-book dynamics, asymmetric information, and multi-agent interactions beyond two-player zero-sum.