Neural Population Learning beyond Symmetric Zero-sum Games

🎯 Pitch

A single neural network can now converge to coordinated equilibria in multiplayer, mixed-motive games—learning to share control of a cheetah's legs or coordinate in capture-the-flag—by distilling best-responses into a shared conditional policy, overturning the assumption that equilibrium solvers need per-player, per-iteration populations.

1. Executive Summary

This paper introduces NeuPL-JPSRO, a neural population learning algorithm that extends equilibrium-convergent methods to n-player general-sum extensive-form games by conditionally representing all players' strategies within a shared neural network and iteratively distilling new best-response policies while regularizing existing ones. The approach is validated on a suite of six OpenSpiel strategy games—including goofspiel, poker variants, sheriff, and trade_comm—where exact CCE gaps and values can be computed analytically, and then scaled to a cooperative MuJoCo cheetah-run domain (where players control separate legs of a shared embodiment) and a 4-player partial-observability capture-the-flag environment. The core mechanisms are a continual learning procedure that ensures co-player stationarity during best-response optimization (fixing reference policies and distilling new strategies each iteration) and a transferable representation that reuses encoder and memory networks across players—enabling a single conditional policy to best-respond to arbitrary co-player mixed-strategies sampled from the metagame CCE distribution. The paper demonstrates empirical convergence to a Coarse Correlated Equilibrium at a rate comparable to exact JPSRO, establishes that independent RL exploiters cannot find profitable deviations beyond certain iterations in capture-the-flag, and shows that the shared representation enables online adaptation to diverse co-players in non-zero-sum settings, establishing that equilibrium-convergent population learning can handle complex visuomotor domains requiring spatiotemporal reasoning—but only up to the computational bottleneck of representing exponentially many payoffs as the number of strategies grows.

2. Context and Motivation

The Core Problem: Scaling Equilibrium Solvers Beyond Symmetric Zero-Sum Games

The fundamental question this paper tackles is: how do we computationally find stable strategic solutions in n-player general-sum games that require complex skills like locomotion, perception, and memory? This matters because most real-world interactions—traffic coordination, economic markets, team sports, international diplomacy—involve multiple heterogeneous actors with mixed motives, yet the game-theoretic algorithms with strong theoretical guarantees have largely stalled at two-player zero-sum domains.

The gap between theory and practice has widened along two dimensions. First, algorithmic: principled equilibrium-finding methods exist for general-sum games (JPSRO converges to a Coarse Correlated Equilibrium in theory), but they have never been demonstrated at scale because representing and training independent deep RL policies for each player at each iteration is computationally prohibitive. Second, representational: the skills required for competent play—basic locomotion, visual encoding, recurrent memory—must be learned from scratch by every policy in the population, creating a wall of computational cost that compounds with each new best-response iteration.

This gap is significant for several reasons the paper draws out:

Cooperative and mixed-motive settings remain unsolved. While competitive symmetric zero-sum games have seen superhuman performance (chess, Go, poker, StarCraft), progress in domains requiring coordination between asymmetric players—division of labor in team tasks, negotiation under partial information, adaptive collaboration—has been far more limited. The paper explicitly connects this to the failure of Nash Equilibrium to model coordinated action choices: "NE describe an impoverished view of general-sum interactions as it forbids correlated action choices between players... consider a road junction, an NE can only suggest uncorrelated action choices for each driver when much improved outcomes could have been achieved by coordinating drivers with a trusted third-party (e.g. a traffic light)."
Computational intractability of exact solutions. Finding exact Nash Equilibria beyond two-player zero-sum games is PPAD-complete (Daskalakis et al., 2009)—not merely difficult in practice, but provably intractable in the worst case. This is not a matter of engineering effort; it is a complexity-theoretic barrier. The paper's pivot to Coarse Correlated Equilibria is motivated by their computational tractability: CCEs can be formulated as linear programs even in n-player general-sum settings, making them a practical solution concept where NE is unreachable.
The skill transfer bottleneck. The paper highlights a specific failure mode of existing population learning methods: "the costs of representing, training and evaluating a population of independent RL agents for each player quickly become intractable, especially in games that demand complex skills." If each new best-response policy must learn locomotion, visual encoding, and opponent modeling from tabula rasa, the cost of each iteration increases linearly with the number of strategies while the exploration problem worsens as co-players become more skilled—creating a vicious cycle where later iterations are both more expensive and harder to solve.

Where Existing Approaches Fall Short

The paper positions its contributions against a specific lineage of population learning methods, identifying precise shortcomings in each that motivate the design of NeuPL-JPSRO.

JPSRO converges in theory but cannot scale. Marris et al. (2021) introduced Joint Policy-Space Response Oracles, proving convergence to a normal-form CCE in n-player general-sum extensive-form games. The proof is constructive: starting from an initial restricted metagame, each iteration computes a best-response policy for each player against the current marginal CCE distribution, evaluates expected payoffs for all joint strategy combinations, solves for a new CCE using linear programming, and repeats. When the BR operator cannot find a policy with improvement above ϵ, the process terminates with an ϵ-CCE guarantee.

However, the computational reality of JPSRO is brutal. At iteration t, the payoff tensor G_t has t^n entries—each requiring simulation-based estimation of expected returns across potentially thousands of episodes. The BR operator must be invoked n times per iteration, each call requiring training a policy from scratch (or from a previous checkpoint) against a moving target of co-player strategies. For any game requiring deep RL—continuous control, visual observations, partial observability—the cost of even a single BR iteration dwarfs what is reportable in academic papers, let alone practical deployment. The paper notes that evidence of JPSRO convergence "has been limited to a few research games that can be solved analytically"—precisely the games where the computational bottlenecks don't apply because BR and EP operators are analytical.

NeuPL introduced shared representation but is restricted to symmetric zero-sum. Liu et al. (2022a, 2022b) proposed Neural Population Learning, which demonstrated that a single conditional network can represent an entire population of strategies for PSRO in symmetric zero-sum games by conditioning on opponent priors. This approach offered several critical advantages: skill transfer across strategies (the encoder and memory networks are shared), reduced training cost to be comparable with self-play, and online adaptation to arbitrary opponent mixtures. The empirical results in rock-paper-scissors and running-with-scissors were strong.

But the paper identifies two critical limitations that prevent NeuPL from solving the general-sum problem:

Symmetry and zero-sum restriction. NeuPL is architecturally and theoretically tied to symmetric zero-sum games. The strategy representation assumes player roles are interchangeable, the convergence arguments rely on properties of NE in zero-sum settings, and the conditioning mechanism encodes only a single opponent (the mixture is over strategies of the symmetrically defined co-player). Moving to asymmetric roles and general-sum payoffs requires a fundamentally more flexible representation—one that conditionally represents strategies for each player identity, encodes co-player joint-strategy distributions across heterogeneous players, and converges to CCE rather than NE.
Convergence requires unverified assumptions. The paper provides a detailed critique of NeuPL's convergence arguments in Section 3.1, identifying a specific and subtle failure mode. NeuPL's concurrent optimization across iterations implicitly assumes the full-support condition (Definition 3.1): that all states are reachable with positive probability under co-player joint policies at each iteration. Without this condition—which held in NeuPL's test domains because rock-paper-scissors is a normal-form game and running-with-scissors had stochastic, entropy-maximizing, partially-observable co-player policies—best-response policies may be non-stationary in states they don't visit during training. The paper provides a concrete counterexample (Figure 1, the anti-coordination game) showing that maximum-entropy RL does not uniquely identify a best-response policy when some states are unreachable: player 2's policy in the unreached states is arbitrary, and subsequent best-responses to player 2 may change arbitrarily, breaking the convergence argument.

This is a subtle but critical insight. The paper argues that "the problem is particularly salient in games where best-response strategies are often deterministic (e.g. goofspiel, also known as the game of pure-strategies)." In deterministic games with partial observability or branching structures, best-response policies may only visit a subset of states, leaving the rest unconstrained. When those unvisited states later become part of the action space under different co-player strategies, the earlier policies' behaviors there are undefined—they are moving targets for subsequent iterations.

Independent RL as a BR operator is computationally prohibitive. The paper notes that even if one ignored the theoretical limitations and attempted to scale JPSRO naively by using independent deep RL for each best-response iteration, the practical cost would be immense. Learning from scratch against increasingly skilled opponents creates a difficult exploration problem: "doing so at every iteration would be impractical (as we show in Section 4.3)." The capture-the-flag experiments directly demonstrate this: randomly initialized RL policies cannot even find profitable deviations against co-player mixtures after a few iterations (Figure 5, right, Blue line flatlines), while the same policies initialized with pre-trained encoder and memory networks succeed (Orange line). The paper's point is that without transfer learning, the BR operator cannot even be implemented in practice for complex domains—not as a matter of theory, but as a matter of basic exploration.

No prior method combines convergence guarantees with scalable neural representation in general-sum games. The paper positions itself at the intersection of two research trajectories that have remained separate. One trajectory—PSRO, JPSRO, and their variants—provides convergence guarantees but uses independent policy representations that don't scale. The other trajectory—population-based training, self-play, league training—scales to complex domains but lacks convergence guarantees to game-theoretic equilibria. The paper explicitly cites works like Vinyals et al. (2019) and Jaderberg et al. (2019) as demonstrating "what could be achieved by multiagent RL at scale, [but] their convergence characteristics remain to be understood." The gap the paper aims to fill is to achieve both simultaneously: "bringing game-theoretic algorithms such as (J)PSRO to real-world games in full generality and at scale."

Why (Coarse) Correlated Equilibria Instead of Nash Equilibrium

The paper's choice of CCE over NE is not arbitrary—it reflects a fundamental philosophical and practical shift that the authors make explicit.

The philosophical argument: NE forbids coordination, real interactions require it. The paper argues that the NE solution concept is fundamentally impoverished for general-sum settings because it requires mutual independence of strategies: σ(a) = ∏_p σ(a_p). In a traffic intersection, NE can only prescribe independent probabilistic strategies (each driver goes/stops with some probability), which yields worse outcomes than coordinated strategies (one driver goes while others wait). The paper characterizes CCE as describing equilibrium behavior when a "correlation device that rational, self-interested players would find beneficial to follow" mediates action choices—traffic lights are the canonical example. Every NE is a CCE (the factorization constraint can always be imposed), but CCEs that are not NEs represent coordination opportunities that NE would miss.

The computational argument: CCE is tractable where NE is not. The paper notes that CCE can be expressed as a linear program even in n-player general-sum games, whereas NE requires solving PPAD-complete problems. The CCE gap—the sum over players of their maximum unilateral deviation incentive, δ(σ) = Σ_p max_{a'_p} E_{a∼σ}[G_p(a'_p, a_{¬p}) - G_p(a)]—can be computed efficiently given the payoff tensor and a candidate joint distribution. The iterative best-response framework of JPSRO converges because each iteration adds strategies that reduce this gap, and the restricted metagame CCE gap upper-bounds the full-game CCE gap. This convergence logic does not depend on zero-sum structure or symmetry.

The practical argument: CCE enables principled coordination. The paper's cooperative MuJoCo experiment (Section 4.2) directly demonstrates why CCE matters: two players controlling different legs of a shared cheetah body must coordinate to achieve locomotion. CCE allows the meta-strategy solver to assign correlated action recommendations (both players use specific coordinated joint strategies) even though the underlying game is decentralized—players observe each other's behavior but don't communicate explicitly. The emergent behaviors show exactly this: at iteration 3, player 1 (rear leg) learns to "carry" player 2 (front leg) when the partner is unskilled; by iteration 16, both players converge to coordinated running. These are CCE strategies—correlated behavioral profiles that rational players would not unilaterally deviate from—not NE strategies that would require independent probabilistic mixtures.

How This Paper Positions Itself

The paper's positioning can be understood through three claims about what it contributes beyond prior work:

1. The first scalable, equilibrium-convergent algorithm for general-sum extensive-form games with complex perception and control. This is the headline contribution. The paper claims that NeuPL-JPSRO is the first method to simultaneously provide (a) provable convergence to CCE in n-player general-sum games, (b) practical scalability to domains requiring deep RL for complex visuomotor skills, and (c) shared representation enabling skill transfer across strategies and players. The convergence proof (Theorem 3.2) inherits from JPSRO—when distillation and regularization are exact, the process is equivalent and converges—while the architecture and training procedure are new.

2. A resolution to the co-player stationarity problem in shared-representation population learning. The paper's critique of NeuPL's convergence argument is more than a technical footnote; it motivates the central design choice of NeuPL-JPSRO—iterative rather than concurrent optimization. By fixing reference policies at each iteration, distilling the new BR policy into the population, and regularizing existing strategies to maintain their reference behaviors, the algorithm enforces stationarity without requiring the full-support condition. The paper explicitly states: "The key idea behind NeuPL-JPSRO is to take a continual learning approach that removes the need for the full-support condition entirely."

3. Empirical demonstration that exact convergence verification is possible, and that the method works where prior methods would fail. The paper validates its claims through a careful experimental design: first verifying convergence against analytically solvable CCE gaps and values in six OpenSpiel games (where the input to the evaluation is only the set of trained policies, removing any approximation artifacts), then scaling to domains where exact verification is impossible but exploiter-based convergence evidence can be provided. The capture-the-flag results (Figure 5, right) directly compare the CCE value to independent RL exploiters, showing that beyond iteration 6, "independent RL training can no longer outperform"—a practical certification of approximate equilibrium despite the game's complexity.

The paper also connects to a broader research narrative about the training-inference tradeoff in game-solving. While not stated as explicitly as in the training-inference scaling law literature, the paper's argument that "artificial agents [should] reuse fundamental skills such as locomotion, perception and memory across strategies" echoes the idea that investing in shared representations is a form of amortized inference—paying the cost of learning transferable skills once and reusing them across the population rather than paying it independently for each strategy. This is the representational analog of compute-optimal allocation: spend fewer "training generations" per strategy by sharing parameters, enabling deeper strategic exploration within a fixed compute budget.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

NeuPL-JPSRO is a training algorithm that builds a population of strategies for all players in an n-player game by representing every strategy—across all players—inside a single shared neural network, where each strategy is identified by a learned embedding vector that conditions the network's behavior. The problem it solves is that existing equilibrium-convergent algorithms for general-sum games (JPSRO) require training independent deep RL agents for each new strategy, which is computationally prohibitive for complex domains requiring transferable skills like locomotion and visual perception; NeuPL-JPSRO's solution "shape" is an iterative continual-learning procedure that freezes reference copies of previously learned strategies, trains one new best-response policy per player per iteration by reusing shared encoder and memory networks, then distills that new policy into the shared population while regularizing existing strategies to remain stationary.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Shared Neural Network ( $\Pi_\theta(\cdot|s, \nu)$ ) — a conditional policy network with parameters $\theta$ that takes a state $s$ and a strategy embedding vector $\nu$ as input, outputting an action distribution. The same parameters process observations through shared encoder and memory networks before a FiLM-conditioned policy head modulates behavior based on the embedding vector.
Strategy Embedding Vectors ( $V = \{V_1, \ldots, V_n\}$ where $V_p = \{\nu_p^0, \nu_p^1, \ldots\}$ ) — each strategy for each player is associated with a learned vector $\nu_p^t \in \mathbb{R}^d$ that parameterizes the player's $t$ -th strategy. These vectors are randomly initialized and optimized jointly with the network parameters $\theta$ during best-response training and distillation.
Reference Policy Snapshot ( $\Pi_{\hat{\theta}}(\cdot|s, \hat{\nu}_p^\tau)$ ) — at the start of each iteration $t$ , the current network parameters $\hat{\theta}$ and embedding vectors $\hat{V}$ are frozen as a stationary reference. These reference policies serve as fixed co-players during best-response optimization, ensuring the targets do not move while the new best-response is being learned.
Conditional Best-Response Policy Head ( $\Pi_\phi(\cdot|s, \sigma_{¬p}^{t-1})$ ) — an auxiliary policy network with parameters $\phi$ that shares the encoder and memory networks with $\Pi_\theta$ but receives a different conditioning input: a weighted representation of the co-player marginal CCE distribution $\sigma_{¬p}^{t-1}$ . This policy learns to best-respond to the current equilibrium mixture of co-player strategies.
Metagame Solver (MSS) — an off-the-shelf linear programming solver that takes the payoff tensor $G^t$ (estimated by a payoff estimator network $\psi_w$ ) and computes a CCE joint distribution $\sigma^t$ over all strategies discovered so far for all players. This distribution determines which co-player mixtures the next iteration's best-response policies should target, and at convergence identifies the equilibrium strategy of the full game.

Information flows as follows: the game initializes with a single uniform strategy per player → at iteration $t$ , reference policies are frozen → for each player $p$ , the conditional BR policy $\Pi_\phi$ is trained against the stationary co-player reference policies sampled according to $\sigma_{¬p}^{t-1}$ → the trained BR policy is distilled into $\Pi_\theta(\cdot|s, \nu_p^t)$ via KL-divergence minimization → existing strategies are regularized toward their reference behaviors → the payoff tensor is estimated using the payoff estimator network on all joint strategy combinations → the CCE solver computes a new joint distribution $\sigma^t$ → the process repeats until no player can find a profitable deviation above threshold $\epsilon$ , at which point $(\Pi_{\hat{\theta}}^{\hat{V}}, \sigma^{t-1})$ is an $\epsilon$ -CCE of the full game.

3.3 Roadmap for the Deep Dive

First, the convergence problem with shared representation — why NeuPL's concurrent optimization fails without the full-support condition, illustrated through the anti-coordination game counterexample (Figure 1), and how the paper formally characterizes the failure mode.
Second, the continual learning solution — the reference policy snapshot mechanism, the distillation operator for adding new strategies, the regularization operator for preserving existing strategies, and why this restores the equivalence to JPSRO (Theorem 3.2).
Third, the formal extension of JPSRO's convergence proof to stochastic best-response policies — why entropy-maximizing RL produces policies from a finite set, enabling termination guarantees even with approximate BR operators (Lemma A.2).
Fourth, the transferable best-response learning architecture — how the conditional BR policy head reuses shared encoder and memory networks, and how co-player mixed-strategies are encoded as weighted sums of strategy embeddings using the top-k representation (Equation 2).
Fifth, the BR optimization procedure — how training episodes are generated with stochastic best-responder selection, the probability schedule $Pr_{br}(\tau, t)$ , and the dual objectives of RL optimization and behavioral distillation.
Sixth, the payoff estimation network — how $\psi_w$ predicts expected returns for all joint strategy combinations without simulation, eliminating the exponential evaluation bottleneck of exact JPSRO.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-and-algorithms paper whose core idea is that equilibrium-convergent population learning can be made computationally practical for complex general-sum games by (1) iteratively freezing reference policies to ensure co-player stationarity during best-response optimization, (2) conditionally representing all strategies across all players in a single shared neural network to enable skill transfer, and (3) approximating the payoff tensor with a learned estimator to avoid simulation-based evaluation of exponentially many joint-strategy combinations.

The NeuPL Convergence Problem: Why Concurrent Optimization Fails

The paper's first technical contribution is diagnostic rather than constructive: it identifies precisely why NeuPL's approach of concurrently optimizing all strategies in a shared network does not guarantee convergence to equilibrium in general games, even though it worked empirically in the specific domains tested by Liu et al. (2022b).

The stationarity requirement. The convergence arguments of population learning algorithms (PSRO, JPSRO, NeuPL) all rely on the same logical structure: at each iteration $\tau$ , a best-response policy $\pi_p^\tau$ is computed against a stationary co-player mixture $\sigma_{¬p}^{\tau-1}$ ; once computed, $\pi_p^\tau$ becomes a stationary co-player target for all subsequent iterations $t > \tau$ . This means that the set of available strategies for each player grows monotonically (new strategies are added, old strategies never change), and the sequence of restricted metagame CCEs converges to a CCE of the full game because each iteration reduces the maximum deviation incentive.

Why NeuPL breaks this. In NeuPL, all strategies across all iterations are represented in a single conditional network $\Pi_\theta(\cdot|s, \nu)$ and optimized simultaneously. The paper identifies the specific mechanism of failure: "a sample-based maximum-entropy BR operator would assign equal probability to 'M' and 'R' in state $S_2$ , converging to the maximum-entropy BR which is stationary in all states reachable under the deterministic player 1 policy. Nevertheless, player 2 may be forced to act in additional (dashed) states in subsequent iterations with a different player 1 strategy — indeed, with function approximation and shared representation, player 2 would behave unpredictably in these states yet any behaviour in these states would still constitute a valid best-response to player 1's original strategy."

The crux of the problem is that in extensive-form games with state branching, a best-response policy only visits a subset of the game tree—those states reachable when actually playing against the specific co-player mixture it was trained against. In the anti-coordination game (Figure 1), player 1 in state $S_1$ publicly chooses L, M, or R; player 2 in state $S_2$ is rewarded for picking a direction different from player 1's choice. If player 1 plays a deterministic "always L" strategy, player 2 only ever visits the branch where player 1 chose L, and the maximum-entropy BR will randomize over M and R in that branch. The dashed branches in Figure 1 (where player 1 chose M or R) are never visited. In NeuPL's concurrent optimization, player 2's behavior in those unvisited states is unconstrained—any action distribution is a valid best-response because the expected return is unaffected by behavior in states that never occur. When a later iteration introduces a new player 1 strategy that does reach those states, player 2's behavior there is arbitrary, and the best-response to player 2's policy may change arbitrarily. This breaks the stationarity chain: strategies from earlier iterations are not truly stationary, so subsequent best-responses are optimizing against moving targets.

The full-support condition as a band-aid. The paper formalizes the condition under which NeuPL would work: Definition 3.1 states that "all states can be reached with positive probability at each iteration under the co-player joint-policy." Under this condition, every state is visited during best-response training, so the max-entropy BR operator uniquely determines behavior in all states (the maximum-entropy policy among all policies that achieve the same expected return). The paper notes that "the two domains considered in [24] likely satisfy this full-support condition: rock-paper-scissors is an NF game, and in running-with-scissors co-player policies are stochastic, under partial observability and entropy-maximising, resulting in full-support reach probabilities for all states."

The critical insight is that this condition is fragile and not guaranteed in general games. The paper explicitly calls out games where "best-response strategies are often deterministic (e.g. goofspiel, also known as the game of pure-strategies)" as particularly problematic. In goofspiel, players' optimal strategies involve deterministic card-playing sequences with no inherent stochasticity, and depending on the specific game state, many branches of the game tree will never be reached under a given BR policy. The paper evaluates NeuPL-JPSRO in goofspiel precisely to demonstrate robustness in this challenging regime (Section 4.1, Figure 3, top-left panel shows successful convergence in goofspiel_2p_5c over 60+ iterations with up to 64 strategies per player).

The Continual Learning Solution: Reference Policies, Distillation, and Regularization

To eliminate the dependence on the full-support condition, NeuPL-JPSRO introduces a continual learning approach that enforces co-player stationarity by construction. The key idea is drawn from the Progress and Compress framework (Schwarz et al., 2018), which the paper cites: "a conditional model continuously compresses existing skills while incorporating new ones with a shared skill latent space."

Algorithms 2 (full pseudocode). The paper provides a complete algorithmic specification (Algorithm 2), which we walk through in operational detail:

Step 1: Initialization (Lines 1–3). The strategy embedding vectors are initialized with a single vector per player: $V_1, \ldots, V_n = \{\nu_1^0\}, \ldots, \{\nu_n^0\}$ . These are associated with initial policies $\Pi_\theta(\cdot|s, \nu_p^0)$ , which in practice are uniform random policies (except in trade_comm where a uniform policy on the action ignoring the message makes no progress—the paper uses a different initialization there). The payoff tensor $G^0$ is evaluated for all single-strategy combinations (just one joint strategy, all players using their initial policy), and the CCE meta-strategy solver computes $\sigma^0$ (trivially, the distribution that puts all mass on that single joint strategy).

Step 2: Iteration loop (Lines 4–17). At each iteration $t$ :

Step 2a: Freeze reference (Line 5). The current network parameters and embeddings are snapshotted: $\hat{\theta} \leftarrow \theta$ , $\hat{V} \leftarrow V$ . These reference copies are held fixed for the entire iteration, never receiving gradient updates. This is the critical mechanism that enforces co-player stationarity: while the new BR policies are being optimized (Line 7), all co-player policies sampled from $\Pi_{\hat{\theta}}(\cdot|s, \hat{\nu}_p^\tau)$ remain exactly as they were at the start of the iteration.

Step 2b: Best-response learning with transfer (Lines 6–7). For each player $p$ , the system trains a best-response policy $\pi_p^t$ against the marginal CCE distribution $\sigma_{¬p}^{t-1}$ from the previous iteration, using co-player policies drawn from the frozen reference network $\Pi_{\hat{\theta}}$ . The BR operator returns both the policy $\pi_p^t$ and the estimated deviation incentive $\delta_p^t$ , which quantifies how much better this new policy performs compared to what player $p$ 's current equilibrium strategy achieves. The training of $\pi_p^t$ uses the conditional BR policy head $\Pi_\phi$ with access to shared encoder and memory parameters—we detail this in the "Transferable Best-Response Learning" sub-section below.

Step 2c: Distillation (Line 8). The newly trained BR policy is distilled into the shared neural population by minimizing the KL divergence between the population network's output under the new embedding vector $\nu_p^t$ and the BR policy's action distribution, for all states $s$ :

$\forall s, \quad \Pi_\theta(\cdot|s, \nu_p^t) \leftarrow \pi_p^t(\cdot|s)$

where " $\leftarrow$ " in the pseudocode means the KL-divergence minimization objective is applied. In the tabular case this can be an exact copy; with function approximation, the optimization drives the population network to match the BR policy's behavior under the new embedding.

Step 2d: Regularization of existing strategies (Line 9). Simultaneously, all existing strategies are regularized toward their frozen reference behaviors:

$\forall s, \quad \Pi_\theta(\cdot|s, \nu_p) \leftarrow \Pi_{\hat{\theta}}(\cdot|s, \hat{\nu}_p)$

for all previously discovered strategy embeddings $\nu_p \in V_p$ . This ensures that while the new strategy is being added (which requires updating the shared parameters $\theta$ ), the behaviors of all previously learned strategies are preserved. Without this step, the parameter updates from distillation would catastrophically overwrite existing strategies—exactly the problem that NeuPL's concurrent optimization could not prevent.

Step 2e: Update strategy set (Line 10). The new embedding $\nu_p^t$ is added to player $p$ 's strategy set: $V_p \leftarrow V_p \cup \{\nu_p^t\}$ .

Step 2f: Convergence check (Lines 12–14). If the maximum deviation incentive across all players $\max_p \delta_p^t < \epsilon$ , the algorithm terminates and returns the previous reference policies $\Pi_{\hat{\theta}}^{\hat{V}}$ and CCE distribution $\sigma^{t-1}$ as the $\epsilon$ -CCE. The paper sets $\epsilon = 1 \times 10^{-3}$ .

Step 2g: Payoff evaluation and re-solving (Lines 15–16). If not converged, the payoff tensor $G^t$ is evaluated for all strategy combinations in the expanded restricted game (using the payoff estimator network $\psi_w$ to avoid simulation), and the CCE solver computes a new joint distribution $\sigma^t$ .

Theorem 3.2 (CCE Convergence). The paper states the convergence guarantee concisely: "When using a CCE meta-strategy solver in NeuPL-JPSRO, and when distill and regularise operators are exact, the sequence of mixed-strategy converges to a normal-form CCE under the meta-strategy distribution." The proof is a trivial reduction: in this exact case, NeuPL-JPSRO is equivalent to JPSRO, which is known to converge as proved in Marris et al. (2021). The equivalence holds because (a) the reference policy snapshot ensures co-player strategies are stationary during each BR iteration, (b) exact distillation means the new strategy $\nu_p^t$ faithfully copies $\pi_p^t$ , and (c) exact regularization means existing strategies never change, making the sequence of restricted metagames identical to what JPSRO would produce.

The practical gap: approximate distillation and regularization. In practice, both distillation and regularization are implemented by minimizing KL divergence, which is approximate due to limited optimization steps, finite network capacity, and the inherent tension between the two objectives (distilling the new strategy vs. regularizing old ones). The paper does not provide theoretical guarantees for the approximate case, but empirically demonstrates (Section 4.1, Figures 7–8) that the learned strategies remain sufficiently stationary: the CCE gaps and values remain stable across iterations, and the population reliably represents up to 64 strategies per player without catastrophic forgetting.

Extending JPSRO to Stochastic Best-Response Policies

The original JPSRO convergence proof (Marris et al., 2021) assumes that the BR operator returns deterministic policies. This is a natural assumption for exact analytical solvers, but deep RL algorithms—particularly policy gradient methods like the MPO algorithm used in this work—produce stochastic policies. The paper provides a formal argument for why this does not break convergence, which is important because stochastic policies create a potential infinite regress: if the BR operator can return any of infinitely many stochastic policies that achieve the same expected return, the population might never exhaust the space of possible additions.

The finite unique stochastic policy argument (Appendix A). The key insight is Definition A.1: a Unique Stochastic Policy Mapping is a function that maps a set of deterministic policies to a single stochastic policy. The maximum-entropy mapping is the canonical example: among all policies that achieve a given expected return (all of which are valid best-responses), the one that maximizes entropy is uniquely determined. Lemma A.2 then states: "If a player has $|\Pi_p|$ deterministic policies, they have at most $2^{|\Pi_p|} - 1$ unique stochastic policies."

Why this matters for convergence. If the BR operator uses a unique stochastic policy mapping (such as maximum-entropy RL), then the set of possible BR outputs is finite because there are only finitely many subsets of deterministic policies to map from. In the worst case, all $2^{|\Pi_p|} - 1$ such stochastic policies would need to be enumerated, which is exponential but finite—the iterative process must eventually terminate because the restricted game action space is finite. The paper therefore uses maximum-entropy RL (MPO with entropy maximization as an auxiliary loss) for all best-response learning, satisfying this condition in practice.

The double role of entropy maximization. Beyond the theoretical convergence argument, entropy maximization serves practical purposes: it promotes exploration during best-response training (the BR policy should discover states and actions that might be useful against the co-player mixture), and it provides a principled tie-breaking mechanism when multiple actions are equally good (e.g., in the anti-coordination game's unvisited branches, maximum entropy ensures the policy randomizes uniformly rather than picking arbitrary deterministic actions that might create pathological co-player targets for later iterations).

Transferable Best-Response Learning Architecture

The paper's approach to scaling the BR operator is to reuse the shared encoder and memory networks already learned by the population, rather than training each BR policy from scratch. Figure 2 (referenced in Section 3.2.1) provides the architectural diagram, which we now describe in detail.

The shared backbone. The base network $\Pi_\theta$ can be decomposed into three parts:

Observation encoder — processes raw observations (pixels in capture-the-flag, joint positions/velocities in cheetah-run, feature vectors in OpenSpiel games) into a compact representation. For Melting Pot domains, this is a 2-layer CNN with output channels (16, 32), kernel shapes ((8, 8), (4, 4)), strides (8, 1), followed by a 2-layer MLP of size (512, 512). For all domains, the encoded features are further processed by a 3-layer MLP of size (512, 256, 128).
Memory network — processes the encoded observation over time. In Melting Pot (capture-the-flag), this is a single-layer LSTM with 256 hidden units, necessary because players must infer others' strategies from partial observation history. In OpenSpiel and dm_control domains (where state is fully observed), the memory network is feed-forward (identity).
Policy head — takes the memory output (state representation $s$ ) and a conditioning vector (either a strategy embedding $\nu$ or a co-player mixture representation), and outputs an action distribution through FiLM (Feature-wise Linear Modulation) conditioning (Perez et al., 2018). The FiLM architecture applies learned affine transformations to the network activations, modulated by the conditioning input, enabling a single set of parameters to represent diverse strategies.

The conditional BR policy head $\Pi_\phi$ . Instead of receiving a strategy embedding $\nu$ , this policy head receives a representation of the co-player mixed-strategy $\sigma_{¬p}$ . The paper designs a specific encoding function $g(V, \sigma_{¬p})$ (Equation 2) to transform the marginal CCE distribution into a vector that can condition the policy:

$g(V, \sigma_{¬p}) = \sum_{a_{¬p}} \sigma_{¬p}(a_{¬p}) f(\ldots, \nu_{a_{p-1}}^{p-1}, \nu_{a_{p+1}}^{p+1}, \ldots)$

where the sum is over all co-player joint strategy assignments $a_{¬p}$ , each weighted by its probability under the marginal CCE $\sigma_{¬p}$ , and $f$ is an aggregation function that combines the strategy embeddings of all co-players into a single vector representation.

The top-k approximation. The number of possible co-player joint strategy assignments $a_{¬p}$ grows exponentially: at iteration $t$ with $n$ players, there are $t^{n-1}$ possible assignments. The paper uses $K = 96$ and considers only the $K$ joint assignments with highest probability under $\sigma_{¬p}$ . Table 1 in Appendix B.1 provides empirical justification: across all six OpenSpiel games, the number of joint actions with non-trivial support (probability > $10^{-3}$ ) ranges from $24.0 \pm 16.7$ (leduc_poker_2p) to $234.4 \pm 158.5$ (trade_comm_2p_3i). The CCE solver (Max-Gini $\epsilon$ -CCE with $\epsilon = 0.01$ ) produces sparse distributions, so the top-96 approximation is "almost always lossless." The representation has constant memory footprint regardless of the actual number of co-player strategies, offering a tunable trade-off between accuracy and computational cost.

The aggregation function $f$ and symmetry exploitation. If all players have asymmetric roles, $f$ simply concatenates the strategy embeddings in player order. If some players have symmetric roles (e.g., both players in goofspiel), the paper sets $V_i = V_j$ and uses an order-invariant pooling operator over their embeddings, such that swapping the two symmetric players' strategy assignments produces the same representation. This is implemented in goofspiel (where players have symmetric roles) and capture-the-flag (where teammates have symmetric roles within each team).

The practical consequence: Bayes-optimal adaptation. Because $\Pi_\phi(\cdot|s, \sigma_{¬p})$ is trained to best-respond to arbitrary co-player mixtures (by sampling $\sigma_{¬p} \sim \text{Pr}(\Delta_{¬p})$ from a distribution over possible co-player distributions), the resulting policy can adapt online to different partners. When deployed, the player maintains a belief over which strategies its co-players are using (for example, a uniform prior $\bar{\sigma}_{¬p}$ at the start of an episode), and the policy $\Pi_\phi(\cdot|s, \bar{\sigma}_{¬p})$ trades off exploration and exploitation Bayes-optimally—probing the co-player's behavior to update its belief and maximizing expected return under uncertainty. This capability is demonstrated in the cheetah-run adaptation videos (footnote 4) where the rear leg player adapts to either cooperative or uncooperative front leg partners online.

Best-Response Optimization Procedure

The paper describes a specific training episode generation scheme and optimization procedure that balances three competing objectives: (1) improving the new BR policy against stationary co-players, (2) distilling the BR policy into the population network, and (3) regularizing existing strategies to remain stationary.

Episode generation (Appendix B.2). At iteration $t$ of best-response learning, training episodes are constructed as follows:

Sample a past iteration: A CCE distribution $\sigma^\tau$ is sampled from $\tau < t$ (with probability $Pr_{br}(\tau, t)$ defined below). This determines which metagame equilibrium serves as the co-player mixture for this episode.
Select the best-responder: With probability $Pr_{br}(\tau, t)$ , one player $p$ is randomly selected to be the best-responding player. This player uses the conditional BR policy head $\Pi_\phi(\cdot|s, \sigma_{¬p}^\tau)$ with the sampled co-player mixture.
Assign co-player strategies: For the selected co-player mixture $\sigma^\tau$ , a joint strategy assignment $a = (a_1, \ldots, a_n) \sim \sigma^\tau$ is sampled. All non-best-responding players $k \neq p$ execute their reference policies $\Pi_{\hat{\theta}}(\cdot|s, \hat{\nu}_k^{a_k})$ .
Collect trajectory: All players interact in the environment for a fixed trajectory length (16 steps in OpenSpiel and dm_control, 32 in Melting Pot), stored in a replay buffer of up to $2 \times 10^5$ (OpenSpiel), $1 \times 10^6$ (dm_control), or $1 \times 10^6$ (Melting Pot) trajectories.

Dual training objectives. From each trajectory, two loss terms are computed:

RL objective for the BR player: Player $p$ uses the trajectory to optimize its RL objective—in this case, the MPO (Maximum a Posteriori Policy Optimization) algorithm with Retrace off-policy correction and entropy maximization as an auxiliary loss. The MPO hyperparameters are: M-step epsilon $0.01$ , E-step epsilon $5 \times 10^{-3}$ , target network updates every 200 gradient steps.
Distillation objective: The BR policy's behavior is distilled into the population network by minimizing $D_{KL}(\Pi_\theta(\cdot|s, \nu_p^{\tau+1}) \|\| \Pi_\phi(\cdot|s, \sigma_{¬p}^\tau))$ —the KL divergence from the BR policy to the population network's output under the new embedding $\nu_p^{\tau+1}$ . This assigns the newly learned strategy a place in the shared representation.
Regularization objective: For all non-best-responding players $k$ , their policies are regularized by minimizing $D_{KL}(\Pi_\theta(\cdot|s, \nu_k^{a_k}) \|\| \Pi_{\hat{\theta}}(\cdot|s, \hat{\nu}_k^{a_k}))$ —the KL divergence from the reference policy to the current population network's output. This preserves the behavior of existing strategies against parameter drift.

The probability schedule $Pr_{br}(\tau, t)$ . This schedule controls how often the current iteration's CCE vs. past CCEs is used for co-player sampling, and how often a player acts as the best-responder:

$Pr_{br}(\tau, t) = \begin{cases} 1 & \text{if } t = 1 \\ \min(0.5, \max(0.2, t/T)) & \text{if } t > 1 \text{ and } \tau = t-1 \\ 0 & \text{otherwise} \end{cases}$

where $T$ is a horizon parameter (not explicitly defined in the paper, but appears to be the total number of planned iterations). The schedule has the following properties: at $t = 1$ , the initial CCE $\sigma^0$ is used exclusively (probability 1). For $t > 1$ , with probability $\min(0.5, \max(0.2, t/T))$ , the immediately preceding CCE $\sigma^{t-1}$ is used as the co-player mixture, gradually increasing from 0.2 to 0.5 over the course of training. With the remaining probability, earlier CCEs $\sigma^\tau$ for $\tau < t-1$ are used. The paper does not specify the exact distribution over past iterations when $\tau \neq t-1$ , but the intent is to interleave training against recent equilibrium strategies (which represent the current frontier) with training against historical strategies (which ensures the BR policy generalizes across the population and doesn't overfit to the latest CCE).

Optimization configuration. The paper uses the ADAM optimizer (Kingma and Ba, 2015) with learning rate $2 \times 10^{-4}$ , gradient clipping by maximum global norm of 10.0, and batch sizes of 1024 (OpenSpiel), 2048 (dm_control), or 2048 (Melting Pot). The entropy maximization follows a decreasing linear schedule at the beginning of each best-response iteration to encourage initial exploration that gradually anneals toward exploitation.

Convergence detection within an iteration. To determine when to terminate a best-response iteration, the system compares the expected value of the BR policy $\Pi_\phi(\cdot|s, \sigma_{¬p}^{t-1})$ against the payoff estimator's predicted value for that strategy assignment $\psi_w(\nu_1^{a_1}, \ldots, \nu_p^t, \ldots, \nu_n^{a_n})$ under the same co-player mixture $\sigma_{¬p}^{t-1}$ . If the improvement is less than $\epsilon = 10^{-3}$ , the iteration terminates and the new strategy is added.

Payoff Estimation Network

A critical computational bottleneck in JPSRO is evaluating the payoff tensor $G^t$ , which has $t^n$ entries at iteration $t$ —each requiring simulation of many episodes to estimate expected returns for all players. For $n = 4$ players at $t = 8$ iterations, this is $8^4 = 4096$ joint strategy combinations. Exact JPSRO avoids this by using analytical game solvers; approximate JPSRO would need to simulate each combination, making it "quickly computationally prohibitive at scale."

The payoff estimator architecture. The paper introduces a neural network $\psi_w(\nu_1^{a_1}, \ldots, \nu_n^{a_n}) \in \mathbb{R}^n$ parameterized by $w$ that takes the strategy embedding vectors for a specific joint strategy assignment $a = (a_1, \ldots, a_n)$ and outputs a vector of predicted expected returns—one scalar per player. Critically, $\psi_w$ is not conditioned on states; it predicts the expected return marginalized over the state and action distribution induced when all players follow their assigned strategies. The paper notes that "the strategy assignment of all players should be treated as privileged information to the RL policies and value functions"—the metagame recommendation is private information under the definition of CCE, so the payoff estimator operates at the meta-level, not as part of the policies themselves.

Training the estimator. The payoff estimator is trained using the same value learning objective as the underlying RL algorithm (MPO with Retrace), meaning it regresses toward the actual returns observed during episodes. Because episodes are generated by sampling joint strategy assignments from historical CCE distributions $\sigma^\tau$ (as part of the BR training process), the payoff estimator naturally accumulates data across diverse strategy combinations. As $\psi_w$ sees the strategy embeddings as input and the episode returns as targets, it learns to predict expected payoffs without requiring dedicated simulation of all $t^n$ combinations.

Symmetry exploitation in the payoff estimator. Similar to the co-player representation function $f$ , if players have symmetric roles, $\psi_w$ can use order-invariant pooling over the embeddings of symmetric players. This reduces the number of distinct inputs the network must learn to map, improving sample efficiency.

Interaction with the CCE solver. At each iteration, the payoff tensor $G^t$ is populated by evaluating $\psi_w$ on all $t^n$ strategy combinations (or exploiting symmetry to reduce this number). The CCE meta-strategy solver (Max-Gini $\epsilon$ -CCE with $\epsilon = 0.01$ ) then computes the equilibrium distribution $\sigma^t$ from this estimated payoff tensor. The $\epsilon$ -slack in the solver is noted as "necessary to ensure the numerical stability of the solver" and does not represent a theoretical weakening of the convergence guarantee—the slack can be made arbitrarily small at the cost of computational precision.

Computational footprint. The paper notes that "CCE-solving is computationally tractable and can be solved using off-the-shelf LP solvers" but acknowledges it becomes a bottleneck "when it comes to games with many players and many strategies." The reference to Marris et al. (2022) on "Turbocharging Solution Concepts" suggests that hardware-accelerated equilibrium solvers could mitigate this bottleneck, though such methods are not implemented in this work. The payoff estimator network partially addresses the bottleneck by replacing simulation-based payoff estimation with a forward pass through a neural network, but the LP solver itself still scales with the size of the restricted game.

The Metagame CCE Solver and Equilibrium Selection

The paper uses the Max-Gini $\epsilon$ -CCE solver from the open-source JPSRO implementation in OpenSpiel. The solver finds a CCE of the restricted game (the payoff tensor $G^t$ ) that maximizes the Gini welfare function, which is a measure of inequality aversion—it prefers equilibria where payoffs are more evenly distributed across players.

Why Max-Gini? The paper notes that "equilibrium selection remains an open question: every CCE describes a rational, stable state of the system... but only certain equilibria are socially valuable or fair." In zero-sum games, all CCEs are payoff-equivalent (the sum is zero, so one player's gain is exactly another's loss), making the choice of solver irrelevant for values. In general-sum games like sheriff and trade_comm, different CCE solvers would converge to different equilibria with potentially very different social welfare properties. The paper observes that "Neither NeuPL-JPSRO nor JPSRO converged to CCE with specific properties. In particular, the values of the CCE do not converge to the values of the maximum-welfare CCE in non-zero-sum games (sheriff and trade_comm)." This is an important limitation: the algorithm guarantees convergence to some CCE, but offers no control over which one.

The $\epsilon = 0.01$ slack. The solver allows an $\epsilon$ -CCE where each player's deviation incentive is at most $\epsilon$ rather than exactly zero. This is both a practical necessity (LP solvers have finite numerical precision) and a conceptual match to the overall framework (the BR operator itself terminates at $\epsilon = 10^{-3}$ , and the payoff estimates are approximate).

Summary of Design Choices and Their Justifications

Iterative freezing vs. concurrent optimization: ensures co-player stationarity without requiring the full-support condition that held in NeuPL's test domains but fails in deterministic extensive-form games like goofspiel. The cost is that existing strategies must be explicitly regularized against forgetting (the regularization loss), whereas NeuPL's concurrent optimization implicitly maintained older strategies through the shared representation.
Distillation + regularization vs. independent networks: enables skill transfer (encoder and memory are shared) while maintaining behavioral separation (each strategy has its own embedding and is regularized to preserve its reference behavior). The alternative—independent RL agents per strategy per player—would fail at scale because each would need to learn basic skills from scratch and the exploration problem would worsen at later iterations.
Top-k co-player representation ( $K = 96$ ): trades off exact representation of the marginal CCE for constant memory footprint. Justified empirically by the sparsity of CCE distributions (Table 1), making the approximation "almost always lossless."
Payoff estimator network instead of simulation: replaces $t^n$ simulation-based payoff evaluations with neural network forward passes, making the metagame re-solving step practical. The trade-off is approximation error in the payoff tensor, which could affect the computed CCE.
Maximum-entropy RL for BR operators: provides the theoretical finite-policy-set guarantee (Lemma A.2) while promoting exploration during training. The decreasing entropy schedule balances exploration early in each iteration against exploitation as the BR policy converges.
Symmetry exploitation in both $f$ and $\psi_w$ : reduces the effective number of distinct strategy combinations that must be represented and evaluated, leveraging the fact that in symmetric games, swapping player identities should not change the encoding or payoff prediction.
FiLM conditioning: enables a single set of network parameters to represent diverse strategies by applying learned affine transformations modulated by the strategy embedding, rather than maintaining separate network weights per strategy. This is what makes the population representation neural rather than tabular.
$Pr_{br}(\tau, t)$ schedule interleaving recent and historical CCEs: prevents the BR policy from overfitting to the latest equilibrium by ensuring it also trains against past co-player mixtures. This resembles fictitious play's approach of best-responding to the average historical distribution, but adapted to the CCE meta-strategy framework where the target distribution changes each iteration.

4. Key Insights and Innovations

Innovation 1: A diagnostic critique of neural population learning convergence that reveals the full-support condition as a hidden fragility

The paper's first intellectual contribution is not a new method but a diagnostic: it identifies precisely why NeuPL's concurrent optimization approach to shared-representation population learning does not guarantee convergence in general extensive-form games, despite working empirically in the specific domains tested by prior work. This matters because prior work treated NeuPL as a convergence-preserving extension of PSRO to neural representations—the paper shows this understanding is incomplete.

What makes this diagnosis distinctive is that it pinpoints a hidden distributional assumption that held in NeuPL's test domains (rock-paper-scissors as a normal-form game; running-with-scissors with stochastic, partially-observable, entropy-maximizing policies) but fails silently in many important game classes. The assumption—all states reachable with positive probability under co-player joint policies at each iteration (Definition 3.1)—is not merely a theoretical nicety; it determines whether a best-response policy's behavior is uniquely defined in unvisited branches of the game tree. In goofspiel (the "game of pure strategies," analytically known to require deterministic best-responses), this condition is violated in exactly the way that breaks NeuPL's convergence logic.

The counterexample in Figure 1 is pedagogically elegant: a three-branch public-choice game where player 1's deterministic strategy makes only one branch reachable for player 2, leaving player 2's behavior in the other two branches unconstrained. Maximum-entropy RL ties those branches by assigning uniform action distributions, but any other assignment would also be a valid best-response. When a later iteration introduces a different player 1 strategy that visits those previously unreachable branches, player 2's behavior there is an arbitrary artifact of its earlier training, and the best-response to player 2 may change arbitrarily—breaking the stationarity that convergence proofs require.

This insight transforms the problem from "how do we make neural population learning work for general-sum games?" into "how do we enforce stationarity without assuming full support?" The paper's continual learning solution (iteration-level freezing, distillation, regularization) follows directly from answering this reframed question. Prior work (NeuPL, PSRO variants) had not identified the fragility; the paper provides the first explicit statement of the condition and its violation, making it a conceptual advance in understanding the interaction between shared representation and game-theoretic convergence, not merely an engineering fix.

The evidence that this diagnosis is correct comes from two directions. First, NeuPL-JPSRO's convergence in goofspiel (Figure 3, top-left panel) over 60+ iterations with up to 64 strategies per player demonstrates that the algorithm works precisely in the regime where the paper predicts NeuPL would fail. Second, the detailed per-strategy stability plots (Figures 7-8) show that individual policies' CCE gaps and values remain stable across iterations—exactly what stationarity requires and what concurrent optimization in NeuPL could not guarantee.

Innovation 2: Positioning CCE (not NE) as the practical equilibrium concept for multi-skill general-sum games, and demonstrating that convergence is achievable

The paper makes a conceptual reframing of the goal of multiagent learning in complex domains: rather than pursuing Nash Equilibrium (which is PPAD-complete and forbids coordination), it argues that Coarse Correlated Equilibrium is the appropriate target for general-sum games requiring transferable skills. This is not a novel theoretical claim—Moulin and Vial defined CCE in 1978, and JPSRO provided convergence theory in 2021—but the paper's contribution is making the case empirically at scale by showing that CCE convergence is achievable with neural network function approximation in domains where NE convergence would be computationally hopeless.

What distinguishes this from JPSRO's original work is the demonstration that the theoretical convergence translates to practical behavior in domains requiring deep RL. JPSRO showed convergence guarantees analytically in small research games; NeuPL-JPSRO shows that the sequence of CCE gaps monotonically declines toward zero in the same games while using approximate best-response operators and payoff estimation (Figure 3), then extends to domains where exact verification is impossible but exploiter-based evidence suggests approximate convergence (capture-the-flag, Figure 5 right). This closes the gap between "convergence in theory" and "convergence in practice with function approximation," which is non-trivial because the proof assumes exact BR and EP operators.

The paper also makes a subtle normative argument about CCE vs. NE that goes beyond computational tractability. The MuJoCo cheetah-run domain (Section 4.2) is a cooperative common-payoff game where two players control different legs of a shared body. Nash Equilibrium in this setting would require each player to independently randomize over leg-control strategies in a way that happens to produce coordinated locomotion—an awkward match between solution concept and domain semantics. CCE naturally captures the correlated behavioral profiles that emerge: the rear leg player learns to carry the front leg player when the partner is unskilled (iteration 3), and both converge to coordinated running by iteration 16. The correlation device is the metagame's recommendation to both players to use specific paired strategies—exactly the kind of coordination that CCE permits and NE forbids. The paper's point is not that CCE is theoretically superior (this was known), but that it is the right solution concept for the kinds of multi-agent skill coordination problems that deep RL makes accessible.

The evidence for this reframing's practical validity is the convergence results themselves: across six OpenSpiel games with varying player counts and payoff structures (zero-sum and general-sum), CCE gaps approach zero at rates comparable to exact JPSRO (Figure 3), and per-player CCE values recover game-theoretic properties like the last-mover advantage in poker (Appendix C.1). The transfer from small analytical games to large deep RL domains succeeds because the CCE framework is computationally tractable at every step—the LP solver scales polynomially, the BR operator benefits from transfer learning, and the payoff estimator avoids exponential simulation costs.

Innovation 3: Transfer learning as a convergence enabler, not just a compute-saver

The paper demonstrates that transfer learning across strategies and players is not merely a computational optimization—it fundamentally enables convergence in complex domains by solving the exploration problem that would otherwise prevent BR operators from functioning. This shifts how transfer learning is understood in population-based methods: from an efficiency-improving add-on to a capability-unlocking necessity.

The capture-the-flag experiments (Section 4.3, Figure 5 right) make this argument with unusual clarity. The paper trains independent RL exploiters against the CCE mixtures at each iteration, both with and without parameter transfer from the shared population. The results show that without transfer, independent RL cannot find profitable deviations against co-player mixtures after just a few iterations (Blue line in Figure 5 right). With transfer—initializing the exploiter's encoder and memory networks from the pre-trained shared backbone—the same RL algorithm successfully finds deviations (Orange line). The paper's interpretation: "strong opponents tend to create difficult exploration problems for randomly initialised RL policies," and "approximate JPSRO would have led to the incorrect conclusion that no further improvement can be made after 3 iterations, forfeiting strategically interesting joint-strategies of this game."

This finding matters because it overturns a common assumption in the population learning literature: that the BR operator is a black-box subroutine whose quality matters only for convergence speed, not convergence correctness. The paper shows that in complex domains, the BR operator can fail entirely without transfer learning—it produces policies that don't improve on the existing equilibrium, not because no improvement exists, but because exploration from scratch cannot discover it. Transfer learning converts this failure mode into a success by amortizing the cost of learning basic skills (visual encoding, recurrent memory, opponent modeling) across the population, leaving each BR iteration to focus on learning strategic responses to specific co-player mixtures.

This is a fundamental finding, not an incremental refinement, because it establishes a qualitative threshold: below a certain domain complexity, independent RL per iteration might be merely expensive; above it, independent RL is non-functional regardless of budget. The cheetah-run results reinforce this: both players learn coordinated locomotion in 16 iterations with a single conditional network (Figure 4), achieving returns comparable to state-of-the-art single-agent RL and outperforming self-play baselines with lower variance. The cost is comparable to training one self-play agent—a dramatic reduction from the hundreds of independent agents that JPSRO would require.

The paper also shows a secondary benefit of transfer learning that goes beyond enabling convergence: online adaptation. By training Π_φ(·|s, σ_{¬p}) to best-respond to arbitrary co-player mixtures (not just the current CCE), the policy learns a Bayes-optimal response function that can infer co-player strategies from interaction and adapt online. This is demonstrated in cheetah-run (footnote 4) where the rear leg player, given a uniform prior over partner strategies, probes the front leg player's behavior and either takes control of the body if the partner is uncooperative or collaborates effectively if the partner is competent. This capability emerges naturally from the transfer learning architecture—the policy's internal representation of co-player mixtures generalizes across priors—and would be unavailable in independent RL approaches where each policy is trained against a fixed co-player distribution.

Innovation 4: Empirical evidence that stationarity regularization can maintain behavioral fidelity across large neural populations, enabling deeper strategic cycles than previously demonstrated

The paper provides the first empirical demonstration that continual learning through distillation and regularization can maintain distinct strategy behaviors for up to 64 strategies per player in a shared neural network while preserving convergence to equilibrium. This is a systems-level finding about the practical limits of neural population representation: prior work (NeuPL) demonstrated populations of moderate size (~10-15 strategies) in simple domains; this paper scales to far larger populations (64 for each of 2-3 players in OpenSpiel games) in the more demanding setting of extensive-form games requiring the represented strategies to remain stationary as co-player targets across many iterations.

The evidence for this claim is in the per-strategy tracking plots (Figures 7-8 in Appendix C.1). Each panel shows a set of lines—one per strategy for a given player—tracking either CCE gap or CCE value over gradient steps. Brighter colors indicate later iterations. The critical observation is that lines remain approximately horizontal once training for a given strategy completes—the dashed portion (active BR training) converges to a stable solid region, and that stability is maintained even as hundreds of thousands of additional gradient steps update the shared network parameters for subsequent strategies. In goofspiel (Figure 7), strategies from iteration 0 through iteration 60+ coexist in the same network without catastrophic interference; in sherrif and trade_comm (Figure 8), the same stability is observed across diverse payoff structures.

Why does this matter beyond the specific algorithm? It establishes an empirical feasibility frontier for neural population learning: the combination of KL-divergence regularization toward frozen reference policies plus FiLM-based conditional architectures can encode strategic diversity at a scale that makes equilibrium convergence achievable for games whose strategy cycles are long enough to require dozens of distinct policies per player. This was not obvious a priori—the tension between the distillation objective (adding new strategies) and the regularization objective (preserving old ones) could have produced a representational bottleneck where adding strategy N+1 degrades strategies 1 through N enough to break the stationarity assumption that convergence depends on.

The paper's technical choices matter for this finding in specific ways. The FiLM conditioning architecture (which modulates activations through learned affine transformations) is not incidental—it means that different strategies share the same network structure but can route information differently through the layers, creating effective representational capacity that scales with the embedding dimension rather than the number of strategies. The explicit regularization to reference policies (rather than relying on the shared representation to implicitly maintain old behaviors, as NeuPL did) provides a direct gradient signal to prevent forgetting. And the Pr_{br}(τ, t) schedule—which gradually increases the probability of training against the most recent CCE from 0.2 to 0.5—ensures that the network continues to practice old strategies even as new ones are added, preventing the distribution shift that would cause forgetting.

This finding has implications beyond population learning for game theory. It suggests that neural networks with appropriate conditioning architectures and continual learning procedures can serve as general policy libraries—repositories of diverse skills that remain individually accessible while sharing representational infrastructure. The connection to the Progress and Compress framework (Schwarz et al., 2018) is explicit: like that work, NeuPL-JPSRO demonstrates that progressive knowledge accumulation through compression into a shared representation is feasible at non-trivial scale, with the additional requirement that the compressed skills must remain stationary (not just functional) to serve as fixed targets in a game-theoretic convergence process.

5. Experimental Analysis

Evaluation Methodology

Dataset. The experiments span three categories: (1) six OpenSpiel strategy games (goofspiel_2p_5c, kuhn_poker_2p, kuhn_poker_3p, leduc_poker_2p, sherriff_2p, trade_comm_2p_3i) as described in Appendix C.1 with specific configurations (Table 2), where exact CCE gaps and values can be computed analytically using known game solvers; (2) a cooperative MuJoCo cheetah-run domain (Tassa et al., 2020) modified into a two-player common-payoff game by assigning front and rear leg actuators to separate players with a shared forward-velocity reward; (3) a 4-player Melting Pot capture-the-flag (Leibo et al., 2021) modified to restrict visibility with brick walls, requiring spatiotemporal reasoning from first-person pixel observations with sparse +10/-10 flag-capture rewards. The OpenSpiel games use the standard test configurations from prior work (Marris et al., 2021; Liu et al., 2022b; Farina et al., 2019), with the specific game loader parameters enumerated in Table 2.
Base model(s). All experiments use a single shared conditional neural network Π_θ(·|s, ν) with the MPO (Maximum a Posteriori Policy Optimization) algorithm (Abdolmaleki et al., 2018) with Retrace off-policy correction (Munos et al., 2016). The architecture varies by domain: for OpenSpiel games, a feedforward MLP processes feature vectors; for MuJoCo, the same MLP processes joint positions/velocities; for Melting Pot, a 2-layer CNN (16, 32 output channels, (8,8) and (4,4) kernels, strides 8 and 1) followed by a 2-layer MLP (512, 512) and a single-layer LSTM memory network with 256 hidden units processes first-person pixel observations and game state features. All domains use a 3-layer MLP encoder (512, 256, 128) and FiLM conditioning (Perez et al., 2018) for strategy modulation. The choice of a single model family across all experiments demonstrates the generality of the approach, not sensitivity to architecture.
Metrics. Two primary metrics are used depending on domain: (1) CCE gap δ(σ) = Σ_p δ_p(σ), where δ_p(σ) = ⌊max_{a'_p ∈ A_p} E_{a∼σ}[G_p(a'_p, a_{¬p}) - G_p(a)]⌋_+—the sum across players of the maximum incentive to unilaterally deviate from the joint distribution σ, with δ(σ) = 0 if and only if σ is an exact CCE; (2) CCE value, the expected payoff to each player under the equilibrium distribution σ. For OpenSpiel games, both are computed exactly using analytical game solvers: "the only input to the evaluation procedure from NeuPL-JPSRO are the sets of trained policies Π_V_θ... The value of each policy, the optimal deviation actions as well as the CCE distributions are computed using analytical solvers." For the MuJoCo domain, expected return (forward velocity reward) is the metric, compared against self-play baselines. For capture-the-flag, expected returns of independent RL exploiters are compared against CCE values to certificate approximate convergence.
Baselines.
- Exact JPSRO (Marris et al., 2021): The algorithm from Algorithm 1, using analytical BR solvers and exact EP evaluation via game solver enumeration. Serves as the gold-standard convergence reference in OpenSpiel games. The paper modifies the default OpenSpiel BR solver from deterministic first-action selection among indifferent choices to a maximum-entropy solver, removing implicit coordination bias.
- Self-play (for cheetah-run): Independent RL agents optimizing through self-play (standard single-agent RL applied to the multiagent control problem).
- State-of-the-art single-agent RL (for cheetah-run): Shahriari et al. (2022), providing a reference for what well-optimized single-policy training achieves in the same environment.
- Independent RL exploiters (for capture-the-flag): Standard RL policies trained from scratch (without pre-trained encoder/memory) to best-respond to fixed CCE co-player mixtures, serving as an approximate BR operator to certify whether profitable deviations exist.
- Initialized exploiters (for capture-the-flag): Same RL policies but initialized with pre-trained encoder and memory network parameters from the shared NeuPL-JPSRO population, demonstrating the role of transfer learning in the BR operator.
Generation budget / compute accounting. The paper measures computational cost primarily in gradient steps and number of independently trained RL policies required, rather than FLOPs or wall-clock time. The key efficiency claim is that NeuPL-JPSRO trains a single conditional network shared across all players and strategies (with one auxiliary BR policy head per iteration), making total training cost "comparable to that of self-play" rather than scaling with n × t independent agents as in approximate JPSRO. For OpenSpiel/dm_control/Melting Pot, the paper uses 1× V100 GPU, 2× V100 GPU, and 1× 8-core TPU-v4 respectively for gradient updates, with 384 independent CPU processes for episode simulation and an additional accelerator for policy inference. Training episode batch sizes are 1024 (OpenSpiel), 2048 (dm_control), 2048 (Melting Pot) trajectories, with replay buffer capacities of 2×10⁵, 1×10⁶, and 1×10⁶ trajectories respectively. The maximum number of BR iterations (and thus strategies per player) varies by game: 10-64 iterations for OpenSpiel, 16 for cheetah-run, 8 for capture-the-flag.
Cross-validation / statistical protocol. OpenSpiel experiments are run over 5 independent seeds per game, with detailed per-seed convergence curves provided in Appendix C.1 (Figures 7-8). The paper reports both aggregate curves (Figure 3, averaged across seeds) and per-seed breakdowns showing strategy-level stability. For the capture-the-flag exploiter experiment, 6 independent BR policies are trained and the maximum return across them is reported (optimistic bound on deviation incentive), with average-case results in Appendix C.3 (Figure 10). The cheetah-run experiment reports 3 seeds with individual trajectories visualized (Figure 4, left panel). Unlike the compute-optimal test-time scaling literature, there is no cross-validation for strategy selection—the equilibrium computation is deterministic given the sequence of added policies, and convergence is verified by monitoring CCE gaps computed analytically from the trained policies.

Main Quantitative Results

Convergence in n-Player General-Sum OpenSpiel Games

Headline result: NeuPL-JPSRO demonstrates empirical convergence to a Coarse Correlated Equilibrium across all six OpenSpiel games, with CCE gaps declining to near zero at rates comparable to exact JPSRO using analytical BR solvers—despite using approximate deep RL best-response operators and a learned payoff estimator network. Figure 3 shows the sum of CCE gaps and CCE values across players for NeuPL-JPSRO (blue) vs. exact JPSRO (red), averaged over 5 seeds.

Breakdown by game:

goofspiel_2p_5c (zero-sum, symmetric): NeuPL-JPSRO converges over ~60 iterations, reaching a sum CCE gap near zero with an initial CCE gap of ~7-8 declining to <0.5. Exact JPSRO follows a nearly identical trajectory. The CCE value for each player is exactly 0 throughout (zero-sum property), and the algorithm discovers and represents up to 64 strategies per player—"far exceeding the size of the population reported in prior works [24]." The paper notes this demonstrates "the potential of NeuPL-JPSRO to converge in games with long strategy cycles."
kuhn_poker_2p (zero-sum, asymmetric): Both methods converge within ~40 iterations, with CCE gaps declining from ~0.7 to <0.1. Per-player CCE values (Figure 6, Appendix C.1) show the last-mover advantage: Player 0 (first mover) converges to approximately −0.06 expected value while Player 1 (second mover) converges to approximately +0.06, recovering the known analytical equilibrium.
kuhn_poker_3p (zero-sum, asymmetric): Convergence occurs within ~10 iterations, with CCE gaps declining from ~0.6 to <0.1. The three-player setting requires the algorithm to conditionally represent strategies for each of three asymmetric players and encode co-player joint-strategy distributions over the other two players—the weighted representation (Equation 2) successfully handles this.
leduc_poker_2p (zero-sum, asymmetric): Convergence within ~34 iterations, with CCE gaps declining from ~3.5 to <0.5. The paper notes that "NeuPL-JPSRO, an approximate method, has observed faster convergence to a CCE in some games than exact JPSRO in early iterations (e.g. Leduc Poker in the first 20 iterations)." It explains this counterintuitive result through the rock-paper-scissors example: an exact BR to a deterministic always-rock policy produces always-paper (highly exploitable), while an approximate maximum-entropy BR produces a stochastic mixture (e.g., 80% paper, 10% rock, 10% scissors) that is less exploitable and thus yields a lower CCE gap in the restricted metagame. "The empirical rate of convergence to an equilibrium depends on the dynamics of the game and the choice of initial policy as well."
sheriff_2p (general-sum, asymmetric): Convergence within ~60 iterations, CCE gaps declining from ~12 to <1. Per-player CCE values (Figure 6) show Player 0 converging to approximately −4 expected value and Player 1 to approximately +2, but the paper explicitly notes that "neither NeuPL-JPSRO nor JPSRO converged to CCE with specific properties. In particular, the values of the CCE do not converge to the values of the maximum-welfare CCE." The Max-Gini solver selects one equilibrium among many possible, and there is no guarantee of social optimality.
trade_comm_2p_3i (common-payoff, symmetric): Convergence within ~60 iterations, CCE gaps declining from ~0.3 to <0.03—the smallest scale among all games. The paper notes that "in trade_comm, learning a maximum-entropy BR to a uniform policy makes no progress: for a policy that ignores the message received, the BR would be agnostic to what message to send," requiring a different initialization than the standard uniform policy.

Strategy stationarity evidence (Figures 7-8, Appendix C.1): The per-strategy tracking plots show individual strategy CCE gaps and values across gradient steps. Each line corresponds to one policy in the neural population, with brighter colors indicating later iterations. The critical observation is that "the neural network is capable of conditionally representing a large number of strategies for each player and maintain their behaviours over time reliably in most cases." In goofspiel (Figure 7), strategies from iteration 0 through iteration 60+ remain stable—their CCE values are flat horizontal lines once training completes. The paper acknowledges occasional instability: "In a few cases, we have observed that policies may change over time. This is reminiscent of the phenomenon of catastrophic forgetting observed in the continual learning literature." However, this does not prevent overall convergence.

Convergence speed comparison with exact JPSRO: Figure 3 shows qualitatively similar convergence trajectories for both methods across all games, with NeuPL-JPSRO occasionally faster in early iterations (leduc_poker) due to stochasticity of approximate BR policies reducing exploitability, and occasionally marginally slower at late iterations where exact analytical BRs can find precise improvements that approximate RL may miss. The paper does not claim NeuPL-JPSRO is faster—it claims comparability in convergence while being scalable to domains where exact JPSRO is impossible.

Online Adaptation in Cooperative MuJoCo Cheetah-Run

Headline result: NeuPL-JPSRO discovers a sequence of 16 coordinated locomotion strategies using a single conditional network shared across two players controlling different legs of the same body, with expected returns improving from ~550 at iteration 1 to ~900 at iteration 16 (Figure 4, left panel)—comparable to single-agent state-of-the-art RL and outperforming self-play baselines with lower variance across seeds.

Comparison to baselines: The maximum return achieved by independent self-play trials (dashed line in Figure 4, left) is approximately 850-900, but with higher variance across seeds. NeuPL-JPSRO achieves comparable returns more reliably—the solid lines for three seeds converge to a tight band around 900. The paper states that "the final averaged return outperforms self-play baselines with less variances across seeds."

Emergent coordinated behaviors (Figure 4, middle and right): The middle panel visualizes the sequence of CCE joint distributions over 16 iterations as a grid, showing which strategy combinations are assigned high probability at each iteration. Early iterations (1-3) show concentrated distributions on specific joint strategies; later iterations spread mass across multiple coordinated profiles. The right panel shows qualitative behavioral visualization: at iteration 3, the rear leg player learns to raise the front leg player, "minimising the disruption caused by an unskilled co-player." By iteration 16, "both players cooperate competently" with coordinated running.

Online adaptation demonstration (videos, footnotes 3-4): By conditioning the BR policy Π_φ(·|s, σ̄_{¬p}) with a uniform co-player prior σ̄_{¬p}, the rear leg player "is capable of probing its front leg partner through interaction and adapt accordingly online. Following initial feedback, the rear leg player either took control of the full body if its partner appears uncooperative or worked effectively with a front leg partner that proved competent." This demonstrates the Bayes-optimal adaptation property generalized from competitive settings (Liu et al., 2022a) to cooperative asymmetric-role settings using CCE rather than NE.

Strategic Team-Play in 4-Player Capture-the-Flag

Headline result: NeuPL-JPSRO discovers a sequence of 8 strategies using a single conditional network in a 4-player partial-observability 2v2 capture-the-flag game, with empirical evidence of approximate CCE convergence: independent RL exploiters trained from scratch cannot find profitable deviations beyond iteration 6 (Figure 5, right), while exploiters initialized with pre-trained encoder and memory parameters succeed at earlier iterations—demonstrating that transfer learning enables the BR operator to function where naive independent RL would fail.

Evolution of strategic behaviors (visualized in footnote 5): The paper describes emergent coordinated strategies: "Throughout the first few iterations, players incrementally learned to implement coordinated strategies such as 'Home Base Defence', 'Opponent Base Camping' and finally 'Teammate Following'. Other skills such as timing the tagging cool-down mechanism have emerged, too." The first-person views for each player are annotated with the iteration of the strategy at play, showing the progression from simple to complex coordinated behaviors.

Exploiter-based convergence certification (Figure 5, right; Figure 10, Appendix C.3): The key empirical test of approximate convergence:

Blue line: Expected returns of independent RL exploiter policies trained from scratch against the CCE marginal mixture σ^t_{¬p} at each iteration t. At iterations 1-5, the exploiter achieves returns higher than the CCE value (red dashed line, which is exactly 0 for this zero-sum game), indicating profitable deviations exist and the metagame has not converged. At iteration 6, the blue line drops to or below the CCE value—"independent RL training can no longer outperform" the equilibrium strategy.
Orange line: Same exploiters but initialized with pre-trained encoder and memory network parameters from the shared population. These outperform the scratch-trained exploiters substantially at iterations 4-5: at iteration 4, the scratch exploiter achieves approximately −50 expected return while the initialized exploiter achieves approximately +50. At iteration 5, the gap narrows, and by iteration 6 both are near zero.
Red dashed line: The CCE value for each player (identically zero since the game is zero-sum).

The "optimistic best-of-six" reporting choice: The paper states "Solid lines show the optimistic best-of-six exploiter returns" and Figure 10 in Appendix C.3 shows average-case results where "independent RL struggled to outperform the CCE equilibrium strategies as soon as iteration 5." The optimistic reporting is justified because "our goal is to certify if any profitable deviation actions can be found"—the maximum across independent trials is the relevant statistic for determining whether an approximate CCE has been reached.

Transfer learning as a BR enabler (Figure 5, right; Figure 9, Appendix C.3): The gap between blue and orange lines is the paper's evidence that "a randomly initialised RL policy struggled to best-respond to strong opponents (Blue) when a policy partly initialised with pre-trained encoder and memory network parameters succeeded (Orange)." Without transfer, "approximate JPSRO would have led to the incorrect conclusion that no further improvement can be made after 3 iterations, forfeiting strategically interesting joint-strategies of this game." With transfer, the BR operator succeeds in finding improved responses through iteration 5, demonstrating that shared representation is not merely a cost-saver but a capability enabler.

Payoff matrix visualization (Figure 9, Appendix C.3): A heatmap showing expected returns for each policy of a representative player when facing co-player policies, with orange (purple) indicating higher (lower) value. "We observe that combinations of policies discovered at later iterations compete favorably against those of earlier iterations and receive higher payoffs"—the strategic progression is monotonic in the sense that newer strategies dominate older ones in direct competition.

Ablation Studies and Robustness Checks

Initial policy choice (trade_comm vs. all other OpenSpiel games): In five of six OpenSpiel games, the initial policy is uniform random in all states. In trade_comm_2p_3i, this fails: "learning a maximum-entropy BR to a uniform policy makes no progress: for a policy that ignores the message received, the BR would be agnostic to what message to send." The paper uses a different initialization for this game (not specified in detail), demonstrating that the algorithm's convergence depends on an initial policy that provides a meaningful starting point for the BR operator. This is not a failure of the method per se, but identifies a requirement: the initial restricted game must have non-trivial strategic structure for the iterative improvement process to begin.

Maximum-entropy vs. deterministic BR solver in exact JPSRO: The paper modifies the default analytical BR solver in OpenSpiel from "first action deterministically among indifferent ones" to a maximum-entropy solver. The justification: the deterministic solver "removes this implicit coordination bias" that could artificially favor certain equilibria through tie-breaking conventions. This is not directly ablated in a controlled experiment (exact JPSRO is always shown with max-entropy), but the paper notes that "Our results for the JPSRO baseline differ from what have been reported in [26]" due to this choice, implying sensitivity to the BR operator's tie-breaking behavior.

Top-k co-player representation size (K = 96): The paper uses K = 96 for the co-player mixed-strategy representation across all experiments. Table 1 in Appendix B.1 provides empirical justification by showing the number of joint actions with non-trivial support under the full CCE distribution: for example, goofspiel_2p_5c has 73.6 ± 42.2 joint actions with probability >10^{-3}, and trade_comm_2p_3i has 234.4 ± 158.5. The paper claims that "our representation of marginal CCE distributions is almost always lossless thanks to the inherent sparsity of the CCE solver." However, the co-player marginal distribution has one fewer dimension than the full joint distribution (the best-responding player's action dimension is excluded), so the 1e-3 probability column is the relevant comparison—most games have < 100 joint actions at this threshold, making K=96 a tight but generally sufficient bound. The paper does not ablate smaller values of K to test sensitivity.

Symmetry exploitation in representation and payoff estimation: For symmetric games (goofspiel, capture-the-flag within teams), the paper uses order-invariant pooling over strategy embeddings and symmetric payoff estimator architectures. The effect is not ablated directly, but the ability to scale to 64 strategies per symmetric player in goofspiel (effectively representing 64 strategies once rather than twice) demonstrates the practical benefit of this design choice. Without symmetry exploitation, the effective number of joint strategy combinations would grow as t^n rather than leveraging the reduced combinatorial space.

Payoff estimator network vs. exact EP evaluation: In OpenSpiel games, the CCE gaps and values are computed analytically from the set of trained policies—the payoff estimator is not used for the evaluation itself, only for the metagame solving step within the algorithm. This means the convergence results (Figure 3) are independent of payoff estimator accuracy. The payoff estimator's impact would appear as a discrepancy between the CCE solved using estimated payoffs vs. the CCE that would be solved using true payoffs; the paper does not report this comparison explicitly. The fact that NeuPL-JPSRO converges at rates comparable to exact JPSRO (which uses true payoffs) suggests the payoff estimator is sufficiently accurate, but this inference is confounded by other sources of approximation (BR operator stochasticity, distillation/regularization imperfection).

Pr_{br}(τ, t) schedule (no explicit ablation): The paper fixes the probability schedule for sampling co-player mixtures (Section 3.2, Appendix B.2) without ablating alternatives. The schedule uses min(0.5, max(0.2, t/T)) for the most recent CCE, gradually increasing from 0.2 to 0.5. The paper does not investigate sensitivity to this choice—it may be that different schedules are optimal for different game dynamics (fast-cycling games might benefit from more historical mixing, monotone games from more recent focus), but this is not explored.

Convergence threshold ε settings (no unified ablation): The paper uses ε = 10^{-3} for BR termination and ε = 0.01 for the CCE solver, but does not ablate these choices. In practice, the BR iteration termination is typically driven by gradient-step budgets rather than the ε threshold, and the CCE solver slack may interact with the Max-Gini solver's equilibrium selection behavior. The paper notes the CCE solver slack is "necessary to ensure numerical stability" but does not explore whether tighter slack would change which equilibrium is found.

Distillation and regularization as approximate operators (no controlled ablation): Theorem 3.2 assumes exact distillation and regularization; in practice, these are implemented via KL-divergence minimization over finite gradient steps. The paper does not ablate the number of distillation/regularization gradient steps, the relative weighting of the two losses, or the architecture's capacity (embedding dimension, network depth/width) against population size. The occasional forgetting observed in the per-strategy plots (Figures 7-8) indicates this is not a solved problem—some strategies drift over time—but the paper does not quantify the frequency or severity of this drift or its impact on convergence rate.

Number of exploiters per iteration in capture-the-flag (6 independent trials): The paper trains 6 independent RL exploiters per iteration and reports the maximum return. This is a valid choice for an optimistic bound on deviation incentive, but the distribution of exploiter returns is not reported (mean, variance, quartiles). Figure 10 in Appendix C.3 only shows the mean. The gap between max and mean is substantial—at iteration 4, the max is ~+50 while the mean in Figure 10 appears ~+20 for initialized exploiters. This means the convergence certification (no profitable deviation beyond iteration 6 under the max-of-6) is weaker than if a larger number of exploiters had been trained, or if the mean had also converged. The paper does not discuss how many exploiters would be needed for a statistically rigorous certification.

Effect of K on representation fidelity (not ablated): While Table 1 provides empirical support for K=96, the paper does not systematically vary K and measure the impact on BR policy quality or convergence rate. The claim that the representation is "almost always lossless" depends on the sparsity of CCE distributions from the Max-Gini solver, which itself depends on the ε=0.01 slack—smaller ε might yield denser distributions that overflow the top-96 approximation.

Critical Assessment

Claim 1: NeuPL-JPSRO converges to a CCE in n-player general-sum games.

What was actually demonstrated: The paper demonstrates that the CCE gap of the restricted metagame declines toward zero across six OpenSpiel games when the trained policies are evaluated using exact analytical game solvers (Figure 3). This is genuinely nontrivial: the policies are produced by approximate deep RL (MPO with Retrace), distilled and regularized via approximate KL-divergence minimization, and the metagame is solved using a payoff estimator network—yet the resulting restricted-game CCEs have CCE gaps approaching or crossing the ε=10^{-3} convergence threshold.

What was not demonstrated: The paper does not demonstrate that the full-game CCE gap (the maximum deviation incentive against the full space of policies, not just those in the restricted metagame) converges to zero. This is the standard gap between restricted-game and full-game equilibria in population learning: the restricted-game CCE gap is an upper bound on the full-game CCE gap, but only if the BR operator finds policies that strictly improve the expected payoff—which in turn requires the BR operator to search over the full policy space. The paper's BR operator searches over the space of policies representable by the conditional network with MPO optimization, which is a subspace of all possible policies. The convergence guarantee (Theorem 3.2) inherits from JPSRO and assumes the BR operator is exact over the full policy space. In practice, the BR operator is approximate and limited by network capacity and RL sample complexity—there may exist a policy outside the representational capacity of Π_φ that would find profitable deviations even after the algorithm terminates.

This is not a fatal weakness—it is inherent to all approximate population learning methods—but the paper's presentation could be clearer about the distinction between restricted-game convergence (which is demonstrated) and full-game convergence (which is only implied under assumptions about BR operator coverage). The exploiter experiment in capture-the-flag (Figure 5 right) attempts to address this by independently training BR policies from scratch and showing they cannot find profitable deviations beyond iteration 6, but this is a pragmatic certification ("no profitably deviating policy was found by this search procedure") rather than a proof.

The CCE gap evaluation is exact but narrow: The analytical evaluation in Section 4.1 computes CCE gaps using exact game solvers, which is a strength—it removes approximation artifacts from the metric itself. However, this evaluation only considers pure-strategy deviations (the max_{a′_p ∈ A_p} operator in Equation 1 is over the finite set of strategies in the restricted metagame). The full CCE gap should consider deviations to any strategy in the full game, including mixtures and strategies not represented by any embedding. The paper's exact evaluation verifies that no strategy in the metagame provides a profitable deviation, but does not verify this for the full policy space.

Convergence to which equilibrium is uncontrolled: The paper explicitly acknowledges this limitation: "Every CCE describes a rational, stable state of the system in that no one has an incentive to deviate from their equilibrium strategy but only certain equilibria are socially valuable or fair. NeuPL-JPSRO converges to one such equilibrium, but to which remains unclear." In sheriff and trade_comm (general-sum and common-payoff), the equilibrium selected by the Max-Gini solver is not the maximum-welfare equilibrium. This is a fundamental limitation of the CCE solution concept combined with equilibrium selection ambiguity—not a failure of the algorithm per se, but a constraint on what guarantees can be provided. A user deploying this system in a practical general-sum setting cannot know whether the resulting equilibrium is cooperative, exploitative, or mediocre for all parties.

Empirical convergence rate is not theoretically characterized: The paper does not provide convergence rate bounds—the number of iterations required to reach a given CCE gap depends on game dynamics in ways that are not predictable from the algorithm's specification. In goofspiel, 60+ iterations are needed; in kuhn_poker_3p, 10 iterations suffice. This is consistent with prior population learning results (PSRO, JPSRO), but it means the computational cost is game-dependent and not known in advance.

Overall assessment: The evidence supports the claim that NeuPL-JPSRO converges to an approximate restricted-game CCE at rates comparable to exact JPSRO, in games where the BR operator and payoff estimator are sufficiently accurate. This is a genuine achievement—prior neural population learning methods lacked this convergence evidence, and prior convergent methods (JPSRO) could not scale to neural representations. However, the "convergence to CCE" claim should be understood as restricted-game convergence with empirically validated BR operator effectiveness, not as a proof of full-game CCE convergence with high-probability guarantees.

Claim 2: NeuPL-JPSRO scales to complex domains requiring transferable visuomotor skills.

What was actually demonstrated: The paper demonstrates that NeuPL-JPSRO can handle (a) continuous control in a physically simulated MuJoCo environment with two asymmetric players controlling a shared body (16 strategies, coordinated locomotion emerging), and (b) partial-observability pixel-based 4-player team games with sparse rewards and memory requirements (8 strategies, coordinated capture-the-flag behaviors emerging). These domains are substantially more complex than the normal-form and small extensive-form games that prior convergent methods have handled.

What was not demonstrated: Two important scaling limits are unexplored:

Number of players: The largest game is 4-player (capture-the-flag as 2v2). The payoff tensor at iteration t has t⁴ entries; at t=8, this is 4096 entries, which is tractable. Scaling to n=6 or n=10 players would produce payoff tensors that the LP solver may not handle, even with the payoff estimator network. The paper acknowledges this: "it remains challenging to scale up to many players each with many strategies due to the size of the payoff tensor." The co-player representation also scales exponentially—the top-96 approximation becomes increasingly lossy as the number of joint strategy assignments grows.
Strategy cycle length: The maximum population size demonstrated is 64 strategies per player (goofspiel). While this exceeds prior neural population learning results, it is unclear whether the regularization can maintain stationarity for hundreds of strategies—the observed occasional forgetting (Figures 7-8) may compound at larger scales. The paper does not characterize how forget rate scales with population size.
Domain diversity: Both complex domains involve coordinated motor control (legged locomotion, navigation/tagging). Domains requiring different types of transferable skills—language understanding, tool use, long-horizon planning, social reasoning—are not tested. The claim of "generality" is supported by two domains sharing fundamental characteristics (physical embodiment, continuous action spaces, team coordination).

The cheetah-run demonstration is a cooperative common-payoff game being solved by a CCE algorithm: This is an interesting design choice—in a common-payoff game, the equilibrium concept is less critical because all players share the same objective. The emergence of coordination is still nontrivial (the algorithm must discover strategies where both players' actions are mutually consistent), but the convergence pressure from CCE is weaker than in zero-sum or general-sum settings. The paper does not compare NeuPL-JPSRO to a simpler cooperative multiagent RL algorithm (e.g., joint-action learning, centralized training) on this domain—it is possible that a non-game-theoretic cooperative method would achieve higher returns with less computation.

The exploitation of transfer learning is validated but its limits are unexplored: The capture-the-flag exploiter experiment (Figure 5 right) convincingly shows that initialized exploiters outperform scratch-trained ones. However, the paper does not explore when transfer learning stops helping—are there domains where the shared encoder representation is actually detrimental because it biases the BR policy toward population-typical behaviors rather than genuinely novel strategic responses? In games where the optimal BR to a new co-player mixture requires fundamentally different perception or memory strategies (e.g., switching from reactive to planning-based play), the shared representation might constrain rather than enable.

Overall assessment: The evidence supports the claim that NeuPL-JPSRO can be deployed to domains requiring deep RL for visuomotor control—this is a qualitative advance over prior convergent methods that were limited to analytically solvable or shallow-observation games. The cheetah-run and capture-the-flag experiments convincingly demonstrate skill transfer across strategies and emergence of coordinated behaviors. However, the "generality" claim is supported by two domains—both involving motor coordination—and the scalability limits (player count, population size, domain type) are not characterized. The paper's contribution here is establishing feasibility, not characterizing the envelope.

Claim 3: Transfer learning enables the BR operator to function in complex domains where independent RL would fail.

What was actually demonstrated: The capture-the-flag exploiter experiment (Figure 5 right) provides direct evidence: at iterations 4-5, scratch-trained exploiters achieve approximately −50 expected return (worse than the CCE value of 0), while initialized exploiters achieve approximately +50 to +25 (finding profitable deviations). The scratch-trained exploiters effectively fail to implement the BR operator—they cannot find policies that improve on the equilibrium—while the initialized exploiters succeed. The paper's interpretation that this is an exploration problem ("strong opponents tend to create difficult exploration problems for randomly initialised RL policies") is plausible and consistent with the observed gap narrowing at later iterations when co-player mixtures are more mixed (less exploitable by simple strategies).

What was not demonstrated: Several alternative explanations and confounds are not ruled out:

Training budget adequacy: The scratch-trained exploiters are trained for 200,000 gradient steps (Figure 5 right, x-axis). It is possible that with more training, scratch-trained exploiters would eventually find the profitable deviations that initialized exploiters find in fewer steps. The paper does not show that the scratch-trained curves have asymptoted—they may still be improving slowly. The claim "transfer learning enables the BR operator to function" would be strengthened by showing that increasing the training budget for scratch-trained exploiters does not close the gap.
Architecture equivalence: The scratch-trained exploiters use "standard RL algorithms" (not specified in architectural detail). If the scratch-trained exploiter has a smaller or simpler architecture than the initialized one (which inherits the CNN+LSTM+MLP structure of the shared population), the performance gap may partly reflect representational capacity rather than transfer learning per se. The paper should clarify whether both exploiters have identical architectures, differing only in parameter initialization.
The CCE value as reference: The paper compares exploiter returns to the CCE value (red dashed line). If the CCE value itself has been affected by the approximate BR operator in earlier iterations (which used transfer learning), the "convergence" being certified is relative to an equilibrium that may be different from what exact JPSRO would have found. This is not a confound for the transfer learning claim itself, but it complicates the interpretation of convergence.

Overall assessment: The evidence strongly supports the claim that in this specific domain, pre-trained encoder and memory parameters substantially accelerate BR learning—the gap between blue and orange lines is large and consistent across iterations 3-5. The stronger claim—that transfer learning enables the BR operator (i.e., it would not function at all without it)—is plausible but not rigorously demonstrated, because the scratch-trained exploiters might succeed with more training budget or architectural tuning. The paper's phrasing "approximate JPSRO would have led to the incorrect conclusion that no further improvement can be made after 3 iterations" assumes that the scratch-trained exploiter's failure is permanent rather than a matter of training budget. This is a reasonable practical claim (200,000 gradient steps is already substantial) but not a proven one.

Claim 4: NeuPL-JPSRO's iteratively-frozen reference policies solve the stationarity problem that prevents NeuPL from converging in general games.

What was actually demonstrated: The algorithm converges in goofspiel_2p_5c (Figure 3, top-left)—a deterministic extensive-form game with 5 cards that the paper explicitly identifies as a domain where "best-response strategies are often deterministic" and where the full-support condition would be violated. The per-strategy stability plots (Figures 7-8) show that individual strategies maintain approximately constant behaviors after their training iteration completes. The counterexample in Figure 1 provides the theoretical argument for why NeuPL without freezing would fail.

What was not demonstrated: The paper does not run a controlled comparison showing NeuPL (concurrent optimization) failing to converge in goofspiel while NeuPL-JPSRO (iterative freezing) succeeds. The claim that the freezing is necessary depends on a theoretical argument (the full-support condition is violated in goofspiel) plus the demonstration that NeuPL-JPSRO converges—but without the negative result (NeuPL failing), we cannot be certain that the freezing is the cause of convergence rather than a sufficient-but-not-necessary design choice. It is possible that NeuPL's concurrent optimization would also converge in goofspiel (perhaps for different reasons, or because the full-support condition approximately holds despite determinism), and the freezing is helpful but not critical.

The paper also does not ablate the freezing mechanism itself—for instance, comparing full freezing (reference policies fixed for the entire iteration) to partial freezing (allowing slow drift of reference policies with a strong regularization coefficient). Such an ablation would reveal whether the hard stationarity constraint is necessary or whether soft constraints suffice.

Overall assessment: The theoretical argument for the necessity of co-player stationarity is sound (Section 3.1, Figure 1), and the empirical evidence is consistent with the claim (convergence in goofspiel, strategy stability in Figures 7-8). However, the lack of a direct ablation or failure demonstration for the alternative (NeuPL in goofspiel) means the claim rests on theoretical reasoning rather than controlled empirical comparison. This is a weakness, though perhaps an understandable one given that running a full NeuPL baseline across six OpenSpiel games would be computationally expensive.

Missing experiments that would strengthen the paper:

NeuPL baseline in OpenSpiel games (especially goofspiel): A direct comparison showing NeuPL failing to converge where NeuPL-JPSRO succeeds would convert the theoretical critique into empirical evidence.
Scalability sweep over K (top-k representation size): Showing how BR policy quality degrades as K decreases would characterize the tradeoff between representation fidelity and computational cost.
Population size vs. forgetting characterization: Quantifying how often strategies drift beyond a threshold (e.g., KL divergence from reference exceeding some value) as a function of population size, embedding dimension, and regularization strength.
BR operator approximation error characterization: Comparing the analytically optimal BR (available in OpenSpiel games) to the RL-trained BR to quantify how much approximation error in the BR operator slows or prevents convergence.
Payoff estimator accuracy ablation: In games where exact payoffs are known, comparing metagame CCEs computed from estimated vs. true payoffs would reveal whether payoff approximation error matters for convergence.
Cooperative multiagent RL baseline for cheetah-run: A non-game-theoretic cooperative training method (e.g., MADDPG, QMIX, centralized PPO) would contextualize whether NeuPL-JPSRO's equilibrium convergence provides benefits beyond what simpler cooperative algorithms achieve in common-payoff settings.
Larger player counts: A scaling experiment (e.g., 3v3 or 4v4 capture-the-flag, or n-player common-payoff games with n>4) to probe the LP solver bottleneck and characterize when the payoff tensor becomes intractable.
Sensitivity to Pr_{br}(τ, t) schedule: Ablating the mixing probability to see whether convergence rate depends on this hyperparameter.
Embedding dimension ablation: The strategy embedding dimension d is not specified in the paper, yet it governs the representational capacity per strategy. How does performance scale with d, and what is the minimum d needed to represent N distinct strategies without interference?

6. Limitations and Trade-offs

The Difficulty Estimation Prelude Is Not Accounted for in the Headline Efficiency Gains

The assumption or constraint. The entire compute-optimal allocation policy depends on estimating each prompt's difficulty before allocating the test-time compute budget. The paper's method for doing this—generating 2048 samples per question and either evaluating ground-truth correctness (oracle) or averaging the PRM's final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this in Section 3.2: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost is difficulty estimation cost + strategy execution cost, and the former dominates: generating 2048 samples per question consumes more compute than the largest test-time budgets studied (256-512 generations). This means the 4× figure is an upper bound on achievable efficiency in a setting where difficulty estimation cost is externalized. A deployment that pays for difficulty estimation would see substantially smaller—or possibly negative—net gains, depending on how many questions share the same difficulty estimate (e.g., a batch of 10,000 questions from a static distribution could amortize the estimation cost, but a single question cannot).

What evidence exists in the paper. The paper does not measure or report the total cost including difficulty estimation. Table 1 (Section 3.2) describes the difficulty estimation protocol but does not include it in any compute budget calculation. The predicted-vs-oracle comparison (Figures 4 and 8) shows that the PRM-based estimator works nearly as well as the oracle, but neither variant accounts for the sampling cost of the estimator itself. The authors flag this as "a key avenue for future work" but provide no characterization of how the cost scales with desired accuracy or how it interacts with the optimal allocation.

Mitigation status. The paper does not attempt to mitigate this limitation. Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" without requiring 2048 samples, but no such model is developed or evaluated. The limitation renders the current method impractical for deployment without a cheap difficulty estimator that does not yet exist.

The CCE Convergence Guarantee Assumes Exact Operators That the Implementation Approximates

The assumption or constraint. Theorem 3.2 states: "When using a CCE meta-strategy solver in NeuPL-JPSRO, and when distill and regularise operators are exact, the sequence of mixed-strategy converges to a normal-form CCE under the meta-strategy distribution." The proof is a trivial reduction to JPSRO's convergence proof under these exactness assumptions. However, the implementation uses: (a) approximate deep RL (MPO with Retrace) for the BR operator, (b) KL-divergence minimization over finite gradient steps for distillation and regularization, and (c) a learned payoff estimator network ψ_w instead of exact expected payoff evaluation. None of these operators is exact.

The consequence. There is a gap between the theoretical guarantee (convergence assuming exact operators) and the practical algorithm (convergence with approximate operators). The approximation errors can interact in ways that prevent convergence or slow it substantially:

BR operator approximation error: The RL-trained policy π_p^t may not be a true best-response to σ_{¬p}^{t-1}—it may fail to find profitable deviations even when they exist, or it may converge to a policy that appears profitable under the approximate value function but is not in expectation. This could cause the algorithm to terminate prematurely, certifying an ϵ-CCE that is not actually an ϵ-CCE of the full game.
Distillation/regularization approximation error: The KL-divergence minimization may not perfectly copy the BR policy into Π_θ(·|s, ν_p^t) or perfectly preserve existing strategies at their reference behaviors. The paper acknowledges this in Appendix C.1: "In a few cases, we have observed that policies may change over time. This is reminiscent of the phenomenon of catastrophic forgetting observed in the continual learning literature." Drifting strategies mean co-player targets are not truly stationary, which undermines the iteration structure that the convergence proof depends on.
Payoff estimator approximation error: The CCE meta-strategy σ^t is solved using estimated rather than true payoffs. Errors in ψ_w propagate to errors in the equilibrium computation, meaning the co-player mixtures that subsequent BR operators target may not be the true restricted-game CCE mixtures.

What evidence exists in the paper. The empirical evidence in Figure 3 is consistent with convergence (CCE gaps decline toward zero) but does not prove that approximation errors are negligible. The per-strategy stability plots (Figures 7-8) show that most strategies remain stable, but occasional drift is visible. The paper does not quantify the approximation error of the BR operator (e.g., by comparing RL-trained BR policies to analytically optimal ones in OpenSpiel games where the latter are available), the distillation/regularization error (e.g., by measuring KL divergence from reference policies at convergence), or the payoff estimator error (e.g., by comparing estimated to true payoffs). The convergence threshold ϵ = 10^{-3} for BR termination and ϵ = 0.01 for the CCE solver are numerical convenience values, not error bounds derived from operator approximation quality.

Mitigation status. The paper does not provide error bounds, sensitivity analyses, or theoretical results for the approximate case. The choice of maximum-entropy BR operators (Appendix A) provides a finite-policy-set guarantee that supports termination in principle, but does not bound the approximation error relative to truly optimal BR policies. The occasional catastrophic forgetting is acknowledged but left to future work. A practitioner cannot know, from the evidence provided, how close the final solution is to a true CCE of the full game versus an artifact of approximation error accumulation.

Single Benchmark Family and Model Family Limit Generality Claims

The assumption or constraint. All experiments use PaLM 2-S* as the base model and the MATH benchmark as the sole evaluation domain. The authors state in Section 4: "We believe this model is representative of the capabilities of many contemporary LLMs," but this claim is not verified by testing other model families or benchmarks. The paper studies six OpenSpiel games, a single MuJoCo task (cheetah-run), and a single Melting Pot task (capture-the-flag with modified visibility). All complex domains involve physical motor control (legged locomotion, navigation, tagging).

The consequence. Several aspects of the findings could be model-specific or domain-specific:

PRM quality and over-optimization behavior depend on the base model's output distribution. A model with different calibration, different error patterns, or different pass@1 distribution across difficulty levels would produce different difficulty-dependent scaling curves. The thresholds at which beam search becomes harmful vs. helpful (Figure 3, right) are specific to PaLM 2-S*'s capabilities on MATH.
Revision model training depends on the base model's ability to learn from in-context incorrect examples. The edit-distance-based pairing procedure and the 38% correct-to-incorrect reversion rate (Section 6.1) may not generalize to other model families with different in-context learning characteristics.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic multi-step reasoning. The difficulty-dependent patterns—beam search hurting easy problems, revisions helping easy problems, no method helping the hardest problems—may not extend to other reasoning domains (code generation, logical reasoning, scientific QA, common-sense reasoning) or to tasks requiring factual recall rather than step-by-step deduction. The paper provides no evidence about which findings are domain-invariant and which are MATH-specific.
The complex-domain results (MuJoCo, Melting Pot) are all from physically simulated motor control tasks. Whether the shared-representation population learning approach transfers to domains requiring language understanding, strategic planning, tool use, or social reasoning is an open question with no evidence.

What evidence exists in the paper. The paper acknowledges the single-benchmark limitation implicitly by not testing alternative domains for the core scaling results. The OpenSpiel results cover six diverse strategy games (varying player counts, payoff structures, symmetry), which provides some cross-domain evidence for the convergence property—but these are small analytical games, not complex deep RL domains. The MuJoCo and Melting Pot results demonstrate feasibility in two physical control domains, but these share fundamental characteristics (continuous action spaces, team coordination, motor skills). The paper's claim of "generality" (Section 7) is supported by two complex domains from related families.

Mitigation status. The paper does not address this limitation directly. The choice of MATH as the primary benchmark is justified in Section 4 (test-time compute expected to help most when the model possesses necessary knowledge and the challenge is drawing complex inferences), but no cross-domain validation is performed. The paper does not discuss which findings are likely to generalize versus be domain-specific, or provide theoretical arguments for domain-invariant properties. The "generality" claim (Section 7) conflates "works on two related complex domains" with "works in generality."

The BR Operator's Effectiveness Determines Everything, and Its Failure Mode Is Unexplored

The assumption or constraint. The entire convergence argument depends on the BR operator (Line 7 of Algorithm 2) being able to find policies π_p^t that improve over the current equilibrium strategy when such improvements exist. The paper implements this using deep RL (MPO with Retrace) with access to shared encoder and memory parameters, but treats the BR operator as a black box whose internal failure modes are not systematically characterized.

The consequence. If the BR operator fails to find a profitable deviation when one exists, the algorithm terminates prematurely and certifies an ϵ-CCE that is actually exploitable—a false negative in convergence detection. This can happen for several reasons:

Exploration failure: The RL agent cannot discover states or action sequences needed to exploit the co-player mixture, particularly against skilled opponents that create narrow exploitation windows. The capture-the-flag exploiter experiment (Figure 5, right) directly demonstrates this: scratch-trained exploiters achieve returns below the CCE value at iterations 4-5, while initialized exploiters achieve substantial positive returns—the scratch-trained BR operator is effectively broken.
Representational failure: The conditional network architecture may not have sufficient capacity to represent the optimal BR policy for a particular co-player mixture. If the true BR requires qualitatively different perception or memory strategies than those learned by the population so far, the shared encoder may constrain rather than enable.
Optimization failure: The RL algorithm may converge to a local optimum that does not achieve the true best-response value, particularly when the reward landscape is shaped by adversarial co-player strategies.

The capture-the-flag experiment (Section 4.3) reveals this limitation starkly: at iterations 4-5, scratch-trained exploiters find returns of approximately −50 (worse than playing the equilibrium strategy), while initialized exploiters find returns of approximately +50. The paper interprets this as evidence that transfer learning is necessary—but the deeper implication is that the BR operator's reliability is itself a function of initialization and training budget, and there is no guarantee that even the initialized BR operator has found the true best response. The "optimistic best-of-six" reporting masks the fact that 5 of 6 exploiters may have failed; we don't know because the distribution is not reported.

What evidence exists in the paper. Figure 5 (right) and Figure 10 (Appendix C.3) show exploiter returns for capture-the-flag, but only at the aggregate level (max across 6 trials, or mean across 6 trials). The paper does not report the full distribution of exploiter returns, does not vary the training budget for exploiters, and does not compare the RL-trained BR to analytically optimal BRs in the OpenSpiel games (where exact BRs are computable). This means we have no direct measurement of BR operator approximation error—the gap between what the RL policy achieves and what the true best response would achieve.

Mitigation status. The paper partially addresses this through the exploiter experiment design: by independently training BR policies and comparing their returns to the CCE value, the system can detect when the BR operator has failed (exploiters outperform the equilibrium) and continue iterating. However, this only detects false positives (thinking we've converged when we haven't)—it cannot detect false negatives (the BR operator failing to find an improvement that exists). The paper does not provide theoretical guarantees on BR operator coverage or empirical evidence that the RL-trained BR policies are close to optimal in any domain. The convergence guarantee (Theorem 3.2) is conditional on the BR operator being exact, and the paper provides no bound on what happens when it is approximate. A practitioner cannot know, at termination, whether the algorithm stopped because the game has reached a true CCE or because the BR operator's capacity was exhausted.

The Payoff Tensor Scales Exponentially in the Number of Players, Limiting Scalability to n > 4

The assumption or constraint. At each iteration t, the CCE meta-strategy solver must process a payoff tensor G^t with t^n entries—one expected payoff vector per joint strategy combination across n players. The paper replaces simulation-based evaluation with a learned payoff estimator ψ_w, which makes evaluation cheaper (a forward pass per entry), but does not reduce the number of entries that the LP solver must process. The co-player mixed-strategy representation (Equation 2) uses a top-K approximation with K=96, but even with this, the number of joint strategy assignments in the full payoff tensor grows as t^n.

The consequence. The computational bottleneck shifts from simulation-based payoff estimation to LP solving over exponentially large tensors. The paper acknowledges this in Section 3.2.3: "we note that although CCE-solving is computationally tractable and can be solved using off-the-shelf LP solvers, we have found it to be a bottleneck when it comes to games with many players and many strategies." Concretely:

For n=4 players at t=8 iterations (the capture-the-flag setting), the tensor has 8^4 = 4096 entries—tractable. For n=6 players at t=8 iterations, the tensor has 8^6 = 262,144 entries—still possibly tractable but substantially more expensive. For n=10 players at t=10 iterations, the tensor has 10^10 entries—intractable.
The top-K=96 approximation for co-player representation does not reduce the payoff tensor size—it only reduces the input size to the BR policy head. The LP solver still operates over the full restricted game.
Hardware-accelerated equilibrium solvers (Marris et al., 2022, cited in the paper) could mitigate this, but are not implemented in this work. The paper uses off-the-shelf OpenSpiel LP solvers.

What evidence exists in the paper. Table 1 (Appendix B.1) shows that the number of joint actions with non-trivial support under the CCE distribution ranges from ~24 (leduc_poker_2p) to ~234 (trade_comm_2p_3i) for 2-3 player games. This suggests that for small player counts, the effective tensor size (nonzero entries) is smaller than the full t^n. However, this sparsity property may not hold as the number of players increases—more players mean more opportunities for correlated deviations, potentially leading to denser CCE distributions. The paper does not test any game with more than 4 players (capture-the-flag is 2v2), so the scalability limit is uncharacterized.

Mitigation status. The paper acknowledges the limitation explicitly: "it remains challenging to scale up to many players each with many strategies due to the size of the payoff tensor. A future direction would be to consider sample-based equilibria solutions, without needing to tabulate the entire payoff tensor upfront" (Section 6). The reference to Marris et al. (2022) suggests a possible mitigation (neural equilibrium solvers), but this is not implemented. The current method is effectively limited to n ≤ 4 players with t ≤ 64 strategies per player, and a practitioner seeking to solve larger games would need to develop or adopt sample-based CCE solvers not provided by this work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper alters the conversation around game-theoretic multiagent learning by demonstrating that equilibrium convergence and scalable neural representation are not in tension—they can coexist in n-player general-sum games. Before this work, the field operated under an implicit dichotomy: methods with convergence guarantees (JPSRO, no-regret learning) were limited to analytically solvable games or required training independent deep RL agents per strategy per player—a cost that exploded combinatorially and made complex-domain deployment infeasible. Methods that scaled to complex domains (population-based training, league training, self-play) lacked convergence guarantees and left the strategic stability of their solutions uncharacterized. NeuPL-JPSRO breaks this dichotomy by showing that a single conditional network shared across all players and all strategies can serve as the policy representation for a provably convergent population learning algorithm, reducing the training cost from scaling with n × t independent agents to being comparable with self-play while maintaining restricted-game CCE convergence at rates matching exact analytical methods.

This is less a paradigm shift than a demonstration of a previously closed door opening—the paper shows that the theoretical path laid out by JPSRO (Marris et al., 2021) can be traveled with neural network function approximation, and that the practical obstacles (co-player stationarity with shared parameters, catastrophic forgetting of strategies, exponential payoff evaluation costs) have workable solutions. The conceptual reframing is not in the equilibrium concept (CCE was already tractable) nor in the convergence proof (which inherits from JPSRO), but in what we can now expect from multiagent RL systems in general-sum settings: not just emergent complexity through competition and co-adaptation, but verifiable strategic stability against unilateral deviation.

The paper reconciles a specific contradiction that existed in the neural population learning literature. NeuPL (Liu et al., 2022a, 2022b) showed that shared-representation population learning could implement PSRO efficiently in symmetric zero-sum games, but lacked convergence evidence in games requiring deterministic best-responses—the very class where PSRO's theoretical foundations are strongest. The paper's diagnostic (Section 3.1) provides the explanation: NeuPL's concurrent optimization implicitly relied on the full-support condition (all states reachable at each iteration under co-player policies), which held in its test domains (rock-paper-scissors as a normal-form game, running-with-scissors with stochastic entropy-maximizing policies under partial observability) but fails in general extensive-form games with deterministic BRs. NeuPL-JPSRO's iterative freezing removes this dependency entirely, converting what appeared to be a fundamental limitation of shared representation into a solvable continual learning problem. This resolution matters because it maps out when neural population learning works and when it doesn't, providing a transition criterion (the full-support condition) that future work can reference rather than rediscover.

The paper also shifts research priorities within the multiagent RL community along several axes. First, it makes the CCE solution concept—rather than NE—the practical target for general-sum multiagent systems. The argument is both computational (CCE is tractable via linear programming where NE is PPAD-complete) and representational (CCE naturally captures the correlated behavioral profiles that emerge when agents share a neural population, as in the cheetah-run domain where the two players' leg-control strategies are coordinated through the metagame distribution). Second, it elevates transfer learning from a compute optimization to a convergence enabler: the capture-the-flag results (Figure 5, right) demonstrate that without shared encoder and memory initialization, the BR operator cannot find profitable deviations, leading to false convergence—a finding that implies approximate JPSRO with independent RL agents would systematically underestimate strategic depth in complex domains. Third, it redirects algorithmic development toward continual learning methods for population management (distillation, regularization, forgetting prevention) as first-class concerns in game-solving, rather than as implementation details.

Research directions that become more attractive after this work include: developing cheap difficulty estimators for compute-optimal allocation (the paper identifies this as the critical bottleneck for deployment), building robust verifiers resistant to over-optimization (the search-scaling ceiling), combining search with revisions in a unified framework (the two mechanisms show complementary difficulty-dependent strengths), and extending the compute-optimal paradigm to domains beyond math reasoning. Research directions that become less attractive include: developing increasingly complex search algorithms without corresponding improvements in verifier robustness (the paper shows lookahead search—the most sophisticated optimizer—paradoxically performs worst due to over-optimization), and pursuing exact Nash Equilibrium solvers for complex general-sum domains (the paper makes a compelling case that CCE is the pragmatically correct target given computational barriers and the need for correlated coordination).

Follow-Up Research This Work Enables

Characterizing the full-game CCE gap via systematically stronger BR operators. The paper demonstrates restricted-game CCE convergence using RL-trained BR policies, but the gap between restricted-game and full-game CCE depends on whether the BR operator can search over the full policy space. A direct follow-up would take the OpenSpiel games where exact BRs are computable analytically and measure, at each iteration, the difference between the RL-trained BR's expected payoff improvement and the true maximum improvement. This would produce a per-iteration BR operator approximation error curve that quantifies how much of the restricted-game CCE gap remains as hidden exploitability. The experiment would also reveal whether the approximation error accumulates (later iterations have larger gaps because the BR operator loses fidelity as co-players become more skilled) or stabilizes. For games like leduc_poker where the paper notes NeuPL-JPSRO converged faster than exact JPSRO in early iterations, this would distinguish whether the speedup comes from approximate BR policies being genuinely less exploitable (as the paper hypothesizes via the rock-paper-scissors example) or from the RL agent failing to find sharp best-responses that the analytical solver discovers.

Scaling the payoff tensor bottleneck with sample-based or neural equilibrium solvers. The paper identifies the LP solver over t^n payoff entries as "a bottleneck when it comes to games with many players and many strategies" (Section 3.2.3) and gestures toward Marris et al. (2022) on hardware-accelerated equilibrium solvers. A concrete next step would be to replace the off-the-shelf OpenSpiel LP solver with a neural equilibrium solver that estimates CCE distributions from sampled payoff queries, eliminating the need to materialize the full G^t tensor. The experiment would scale capture-the-flag to 3v3 (6 players) or 4v4 (8 players), measuring whether CCE gaps continue to decline for the same number of BR iterations and whether the neural equilibrium solver's approximation error introduces new convergence pathologies. The comparison would also characterize the tradeoff between solver approximation error and population size: does a coarser equilibrium solution (larger ϵ in the solver) reduce the number of BR iterations needed, or does it lead to cycling where the algorithm fails to converge because the solver oscillates between different equilibria?

Stress-testing the regularization against catastrophic forgetting at scale. The paper reports occasional strategy drift in the per-strategy stability plots (Figures 7-8) but does not characterize how forgetting scales with population size, embedding dimension, or the number of gradient steps between the creation of a strategy and its final use. A systematic study would initialize populations of varying maximum sizes (32, 64, 128, 256 strategies per player) in goofspiel or a synthetic cyclical game with controllable cycle length, and measure (a) the KL divergence from each strategy to its frozen reference at the end of training, (b) the correlation between KL divergence and iteration age (do older strategies drift more?), and (c) whether the drift actually matters for the CCE gap—is there a threshold beyond which co-player non-stationarity prevents convergence? This would establish practical limits on how large a population the current architecture and regularization scheme can support, and would guide future architectural improvements (larger embedding dimensions, stronger regularization coefficients, elastic weight consolidation, or episodic memory replay for old strategies).

Combining NeuPL-JPSRO with population-based training to bootstrap the initial policy set. The paper initializes the population with a single uniform policy per player (except in trade_comm where this fails). In complex domains, this means the first several BR iterations are spent learning basic skills (locomotion, visual encoding) rather than strategic responses—the paper notes that in cheetah-run, the front leg player only becomes competent after the rear leg player learns to carry it (iteration 3). A natural extension would use population-based training (PBT, as in Jaderberg et al., 2019) or quality-diversity methods to pre-populate the initial strategy set with a diverse set of competent-but-not-strategically-optimized policies before starting NeuPL-JPSRO iterations. The experiment would compare convergence speed (CCE gap vs. total gradient steps, including PBT pre-training) for cheetah-run and capture-the-flag with and without a pre-trained initial population. The hypothesis is that strategic BR iterations would be more efficient when starting from diverse competent policies rather than from uniform random, because the BR operator would focus on exploiting opponent weaknesses rather than discovering basic motor primitives.

Testing whether the full-support condition is necessary by running NeuPL (concurrent optimization) in goofspiel. The paper's theoretical argument that NeuPL fails without full support (Section 3.1) is compelling but lacks direct empirical validation. A controlled experiment would run NeuPL (without iterative freezing, without explicit regularization, with concurrent optimization of all strategies) on goofspiel_2p_5c—the game the paper explicitly identifies as having deterministic best-responses where the full-support condition is violated. The prediction is that CCE gaps would fail to converge (or would converge to a significantly higher value than NeuPL-JPSRO) because earlier strategies drift as their parameters are updated during later iterations, creating moving co-player targets. A stronger version would also test NeuPL in kuhn_poker (where the paper's theory does not predict failure, since poker has stochastic policies) to verify that concurrent optimization can work when the full-support condition approximately holds. This experiment would convert the paper's diagnostic contribution from theoretical critique to empirically validated boundary condition.

Extending to settings where the CCE is not the natural solution concept—common-payoff and team games. The paper uses CCE for common-payoff (trade_comm) and cooperative (cheetah-run) settings, but in such games the equilibrium concept is doing less work—any joint policy that maximizes the shared return is Pareto-optimal, and CCE convergence provides strategic stability against deviation but not optimality. A revealing experiment would compare NeuPL-JPSRO against cooperative multiagent RL algorithms (e.g., QMIX, MADDPG, centralized PPO) on the cheetah-run domain, measuring both final shared return and the cost (in gradient steps) to achieve it. If cooperative MARL achieves the same or higher return in fewer gradient steps, it would clarify that CCE convergence provides value primarily in competitive and mixed-motive settings, not in purely cooperative ones where simpler coordination mechanisms suffice. Conversely, if NeuPL-JPSRO outperforms cooperative MARL (perhaps because the CCE framework's iterative best-responding discovers coordination strategies that joint optimization misses), that would be a surprising and practically significant finding.

Practical Applications and Downstream Use Cases

Multi-agent coordination under partial observability with no communication. The capture-the-flag domain models a setting where agents must coordinate with teammates and compete against opponents using only visual observations—no explicit communication channels. NeuPL-JPSRO's convergence to a CCE means that the learned policies implement an implicit coordination convention mediated by the metagame distribution: at deployment, each agent can sample from the CCE joint distribution (using a shared random seed as the correlation device) to select a strategy, knowing that teammates will select complementary strategies and that no opponent has a profitable unilateral deviation. This is directly applicable to physical multi-robot systems (warehouse logistics, search-and-rescue, agricultural monitoring) where communication bandwidth is limited or unreliable but agents share a pre-deployment training phase. The paper's cheetah-run results quantify the benefit: two agents controlling separate actuators on a shared body achieve forward-velocity returns of ~900 in 16 BR iterations, outperforming independent self-play with lower variance across seeds. The key practical advantage is that the CCE provides a stability guarantee—no single robot can improve its individual performance by deviating from the coordinated joint policy—which matters in safety-critical settings where unpredictable agent behavior could cause physical damage.

Online adaptation to diverse co-players in mixed-motive interactions. The paper demonstrates that the conditional BR policy Π_φ(·|s, σ̄_{¬p}), when given a uniform prior over partner strategies, can infer partner behavior online and adapt accordingly—in cheetah-run, the rear leg player either takes control of the body if the front leg partner is uncooperative or collaborates effectively if the partner is competent (Section 4.2, footnote 4). This capability extends beyond the specific population of trained strategies: because the BR policy is trained to best-respond to arbitrary co-player mixtures (not just the current CCE), it generalizes to partners whose strategies interpolate between or lie outside the discrete set of trained embeddings. A practical deployment scenario is adaptive driver assistance systems where the autonomous vehicle must coordinate with human drivers exhibiting diverse, unknown styles—the trained policy conditions on a belief over driver types (updated from observed behavior) and selects actions that are Bayes-optimal under uncertainty. The key metric from the paper is that this adaptation emerges from the same training process that produces the equilibrium-convergent population, requiring no additional training phases or environment interactions.

Training data generation for self-improvement in multiagent systems. NeuPL-JPSRO's iterative BR process naturally produces a curriculum of increasingly sophisticated strategic interactions, stored in the population of strategy embeddings and the sequence of CCE distributions. This curriculum can be mined to generate training data for downstream models—for instance, a single unified policy that imitates all strategies in the population, or a world model that predicts co-player behavior given the metagame context. The paper's payoff estimator network ψ_w (which learns to predict expected returns for any joint strategy combination without simulation) is itself a form of extracted knowledge from the population. In practical terms, an organization deploying multiagent systems could run NeuPL-JPSRO once to generate a diverse strategy library and corresponding equilibrium characterization, then distill this into lightweight deployment policies that don't require the full continual learning machinery at inference time. The paper's demonstration of 64 strategies per player in goofspiel within a single conditional network (Figure 3) shows the population density achievable—substantially richer than what manual strategy design or single-policy training could produce.

Verification of strategic stability in competitive multiagent deployments. In competitive settings—automated trading, auction bidding, adversarial testing of AI systems—deploying a policy that is exploitable by an adaptive opponent can lead to systematic losses. NeuPL-JPSRO's convergence certification (CCE gap declining to near zero, exploited by independent RL exploiters in Figure 5 right) provides a quantitative stability guarantee: at convergence, no unilateral deviation by any player can improve its expected payoff beyond the ϵ threshold. For a trading firm deploying an automated market-making algorithm, this means they can bound the maximum loss an adversarial counterparty could extract by deviating from equilibrium behavior. The practical workflow would be: run NeuPL-JPSRO on a simulated market environment with realistic opponent models → verify convergence via independent exploiter training (as in Figure 5) → deploy the resulting CCE strategy with a statistical guarantee that no opponent strategy in the considered policy space can achieve more than ϵ improvement. The paper's capture-the-flag results (Figure 5 right) operationalize this: at iteration 6, even the best of 6 independently trained RL exploiters achieves approximately zero expected return against the equilibrium, certifying approximate CCE status. The computational cost of this verification is moderate (6 × 200,000 gradient steps for the exploiters, comparable to one BR iteration) and amortizes across the deployment lifetime of the policy.

When to Prefer This Method

The paper articulates a clear tradeoff between NeuPL-JPSRO and two alternatives—exact JPSRO (convergent but not scalable) and approximate JPSRO with independent RL agents (scalable in principle but practically failing due to exploration hardness). The decision rule is:

Prefer NeuPL-JPSRO when: the domain requires transferable skills (visual encoding, recurrent memory, motor primitives) that must be reused across strategies to make best-response learning tractable; the number of players is modest (≤4 in the paper's experiments, bounded by the LP solver's scalability); and a CCE solution (rather than NE or a specific welfare-maximizing equilibrium) is acceptable. The capture-the-flag domain is the canonical example: independent RL exploiters fail to find profitable deviations after a few iterations without pre-trained parameters (Figure 5, right, Blue line), while the shared representation enables strategic depth to be discovered (8 iterations, coordinated behaviors like Home Base Defence and Teammate Following).
Prefer exact JPSRO when: the game is analytically solvable (both BR and EP operators have closed-form implementations), the number of strategies needed for convergence is small, and the goal is to characterize the exact equilibrium properties (CCE values, gap to maximum-welfare CCE). The OpenSpiel games in Section 4.1 are this setting—NeuPL-JPSRO matches exact JPSRO in convergence rate but provides no additional benefit for small analytical games.
Prefer population-based training without convergence guarantees when: the primary goal is emergent behavioral diversity rather than strategic stability, the player count or strategy cycle length exceeds what the continual learning and LP solving can handle, or deployment does not require worst-case guarantees against adversarial exploitation. The paper does not implement this comparison, but the limitation is implicit in the payoff tensor bottleneck (Section 6).

The paper does not position NeuPL-JPSRO against cooperative multiagent RL methods (MADDPG, QMIX, centralized PPO) for common-payoff settings like cheetah-run—this is a gap in the tradeoff analysis. For purely cooperative tasks, a practitioner would need to weigh the CCE stability guarantee (which may be unnecessary when all agents share objectives) against the computational overhead of iterative BR solving, population management, and LP-based equilibrium computation.