PokéChamp: an Expert-level Minimax Language Agent

🎯 Pitch

PokéChamp reimagines minimax search by using an LLM to propose actions, model an opponent’s hidden strategy, and evaluate positions—without any task-specific training. With GPT‑4o, it wins 84% of matches against the strongest rule‑based bot and reaches the top 30% of human players, yet even a compact 8B‑parameter Llama model beats a prior GPT‑4o bot 64% of the time.

1. Executive Summary

This paper introduces PokéChamp, a minimax agent for competitive Pokémon battles that integrates Large Language Models into three modules of tree search — player action sampling (generating a small set of viable moves to prune the search tree), opponent modeling (predicting the adversary's likely actions under partial observability), and value function estimation (evaluating leaf nodes at a cutoff depth instead of simulating to terminal states) — requiring no additional LLM training. Evaluated in the Generation 9 OverUsed format on Pokémon Showdown, PokéChamp powered by GPT-4o achieves a 76% win rate against the best existing LLM-based bot and 84% against the strongest rule-based bot, with an open-source 8-billion-parameter Llama 3.1 variant still outperforming the prior state-of-the-art LLM agent (PokéLLMon with GPT-4o) at a 64% win rate — establishing that LLM-augmented minimax search yields expert-level play (projected Elo of 1300–1500, top 30%–10% of human players) without task-specific training, though gains are bounded on hard problems where the base model's pass@1 is near zero.

2. Context and Motivation

The Core Problem: Generalist Agents Lack Strategic Planning Capabilities

The central problem this paper tackles is the planning deficit in large language model agents for complex, partially observable competitive games. While LLMs have demonstrated remarkable generalist capabilities across a wide range of domains — answering questions, writing code, summarizing text — they systematically fail at the kind of structured, multi-step strategic reasoning that competitive games require. The paper points to a stark, specific example in Section 1.1: even state-of-the-art LLMs "are still unable to play the Nash strategy for Tic-Tac-Toe" — a simple 3×3 grid game that is mathematically solved and trivial for any tree-search algorithm. If an LLM cannot reliably plan in a game with a state space of roughly $3^9 \approx 19,\!683$ possible boards, it is fundamentally unequipped for games where the first-turn state space alone is estimated at $10^{354}$ (Section 1).

This gap matters for reasons that extend far beyond game-playing. The underlying capability that enables success in competitive games — reasoning through sequences of actions, anticipating an adversary's responses, making decisions under uncertainty, and evaluating intermediate positions — is the same capability required for real-world multi-agent scenarios: negotiation, cybersecurity, autonomous driving in traffic, and strategic business planning. If LLMs are to serve as generalist agents in these domains, they must overcome their planning limitations. The paper's investigation into Pokémon battles is thus not merely about building a better game bot; it is a case study in whether and how LLMs can be augmented with classical planning algorithms to produce competent strategic reasoning without task-specific training.

The significance of the problem is amplified by the evidence the paper marshals about the inadequacy of LLM-only approaches. The Nethack learning environment (Küttler et al., 2020), a roguelike game designed to test open-ended RL agents, has seen LLM-powered attempts (Jeurissen et al., 2024) that "still performs poorly compared to an extensive heuristic bot" (Section 1.1). This pattern — LLMs underperforming hand-coded heuristics — recurs across game domains and underscores a genuine technical gap: LLMs possess extensive declarative knowledge (facts, strategies, mechanics) but lack robust procedural planning capabilities (how to chain that knowledge into optimal multi-step decisions under adversarial pressure).

Why Pokémon Battles Are an Ideal — and Unusually Difficult — Testbed

The paper's choice of Pokémon battles as an experimental domain is not arbitrary. Section 2 lays out a detailed argument for why this environment poses specific challenges that stress-test the intersection of LLM capabilities and planning algorithms:

Scale that defies exhaustive search. The state space is astronomically large — $10^{354}$ possibilities for the first turn alone — stemming from over 1,000 Pokémon species, each with configurable moves, abilities, held items, natures, and a distribution of Effort Values (EVs) and Individual Values (IVs) that affect statistics. Battles can last from 6 to over 100 turns (Section 1). Any approach that relies on enumerating the state space — as classical minimax does in games like Chess or Go — is computationally intractable. The search tree must be aggressively pruned using domain knowledge, which is precisely the kind of knowledge LLMs might provide.

Partial observability as a first-class challenge. Unlike Chess or Go, where both players see the full board, competitive Pokémon is a partially observable Markov game (Section 3). A player knows their own team completely but only sees the opponent's currently active Pokémon. The opponent's remaining five Pokémon, their movesets, items, abilities, and stat distributions remain hidden until gradually revealed through gameplay. Section 2 notes that players face "asymmetric observation spaces," meaning strategic decisions must be made under genuine uncertainty. This is a harder problem class than fully observable games — Poker, not Chess — and it tests whether an agent can infer hidden information from observed behavior (opponent modeling), which is a core challenge in real-world multi-agent settings.

An evolving metagame requiring adaptation. The competitive landscape — which Pokémon are popular, which strategies are dominant, which counters are being developed — shifts over time as players discover new combinations. Section 2 refers to an "evolving metagame" that makes "adaptation essential." A successful agent cannot simply memorize a static strategy; it must be responsive to the current competitive context. This tests whether LLMs, with their frozen pre-training data (which has a cutoff date), can integrate real-time statistical information about the current metagame — a tension the paper explicitly investigates and finds problematic (Section 5.3, where the speed-optimized variant underperforms due to LLM priors outweighing real-time statistics).

Tight real-time constraints. Section 2 specifies that each player has 150 seconds total for a match, with 15 additional seconds per turn. Exceeding these limits results in an automatic loss. This precludes exhaustive search not only computationally but temporally — the agent must decide within seconds, simulating the latency constraints of real-world deployment. Any planning approach must be not just accurate but fast, creating a natural pressure to use LLMs for aggressive pruning rather than exhaustive enumeration.

Generation-specific mechanics add combinatorial complexity. Modern Pokémon games introduce mechanics like Terastallization (Gen 9), which allows a Pokémon to change its type once per battle to any of the 18 Pokémon types. This alone multiplies the action space dramatically: on any turn where Terastallization is available, the agent must consider not only which move to use but whether to change type, and to which type. The paper tests this explicitly through dedicated puzzles (Section 5.2) and shows that PokéChamp can leverage these mechanics while baseline agents struggle.

The combination of these factors — massive state space, partial observability, evolving metagame, real-time constraints, and mechanically complex actions — makes Pokémon battles a more demanding test of integrated LLM planning than most prior game benchmarks. It is not simply a matter of "can the LLM recall the right move"; it is "can the LLM, in seconds, prune an astronomical search space to a handful of plausible strategies, model a partially hidden opponent, and evaluate intermediate positions accurately enough to pick a good action."

Prior Approaches and Their Specific Shortcomings

The paper categorizes prior work into two broad approaches, each with well-documented failures that motivate the need for a hybrid LLM-planning architecture.

Reinforcement Learning from Scratch (Tabula Rasa)

The dominant paradigm for achieving superhuman performance in competitive games has been deep reinforcement learning with massive-scale self-play. This approach has produced landmark results: AlphaGo and AlphaZero (Silver et al., 2016, 2017) for Go and Chess, Libratus and Pluribus (Brown & Sandholm, 2018, 2019) for Poker, AlphaStar (Vinyals et al., 2019) for StarCraft II, and OpenAI Five (Berner et al., 2019) for Dota 2. Section 1 summarizes this lineage.

However, the paper identifies a critical limitation: these approaches "typically require substantial task-specific training and engineering." AlphaZero, for instance, played millions of games against itself to learn. This is not merely expensive — it is inflexible. When the game changes (a new Pokémon generation, a balance patch, a shift in the metagame), a tabula rasa agent must be retrained from scratch, or at least extensively fine-tuned. The computational and engineering cost of adapting to new game states is prohibitive for many real-world applications where the environment is not static.

Within the Pokémon domain specifically, the paper notes that prior RL efforts (Huang & Lee, 2019) demonstrated "competitive performance against heuristic bots" and that Whidden (2023) achieved "human-level effort" through "iterative learning and reward optimization." But the key phrase in the paper's framing is: "Unlike prior work, PokéChamp does not train exhaustively Pokémon. In fact, our method does not explicitly train at all, yet performs at an expert level" (Section 1.1). The goal is not just performance but zero-shot or training-free performance — leveraging the LLM's pre-existing knowledge of Pokémon strategy rather than learning it through millions of games. This is a fundamentally different aspiration: a generalist agent that can be deployed immediately in a new game environment based on its pre-training knowledge, rather than a specialist that requires environment-specific training.

LLM-Only Approaches (Prompting Without Planning)

The more recent alternative is to use LLMs directly as game agents, relying on their pre-training knowledge and chain-of-thought prompting to make decisions without any explicit search. This approach has been explored across multiple game domains (Section 1.1): Pokémon (PokéLLMon, Hu et al., 2024b), StarCraft (Ma et al., 2023), social deduction games like Avalon (Shi et al., 2023; Stepputtis et al., 2023), and open-world environments like Minecraft (Voyager, Wang et al., 2023). The paper also notes efforts to integrate LLMs with specific reasoning frameworks — Tree of Thoughts (Yao et al., 2024) for exploring multiple reasoning paths, ReAct (Yao et al., 2022) for interleaving reasoning and action, and Reasoning via Planning (Hao et al., 2023) for using LLMs as world models.

Yet the evidence these papers produce is, as Section 1 puts it, that LLM agents "still fundamentally fail at basic planning algorithms." The Tic-Tac-Toe failure is the paper's headline example — if chain-of-thought prompting cannot reliably solve a solved game, trusting it for Pokémon battles (with a $10^{354}$ state space) is unjustified. The Nethack results further demonstrate that LLMs, even with sophisticated prompting, underperform hard-coded heuristics in complex environments. The paper does not argue that LLMs are useless — quite the opposite, it argues they possess valuable knowledge — but rather that their raw planning capability is insufficient and must be augmented with structured search.

PokéLLMon (Hu et al., 2024b), the prior state-of-the-art LLM-based Pokémon agent, represents the closest comparison point for PokéChamp. It uses self-consistency prompting — generating multiple candidate actions and selecting the most common one. The paper's results show PokéLLMon achieves only a 40% win rate against the Abyssal heuristic bot in Gen 9 OU, compared to PokéChamp's 84% (Table 2). This gap quantifies precisely what is lost by relying on prompting alone without tree search: PokéLLMon samples actions from the LLM distribution and takes a consensus, while PokéChamp uses the LLM to guide a structured minimax search that explicitly considers opponent responses and future states.

The Specific Gap: No Framework Combines LLM Knowledge with Classical Planning

The paper positions itself in the space between these two approaches. On one side, RL-based methods achieve superhuman performance but require massive environment-specific training and do not generalize. On the other side, LLM-only methods generalize but fail at planning. The gap is an architecture that:

Leverages the LLM's pre-existing strategic knowledge — its understanding of Pokémon types, matchups, common strategies, and game mechanics learned from internet-scale training data — without requiring additional fine-tuning or self-play.
Integrates this knowledge into a principled game-theoretic planning framework — minimax tree search — that provides the structured lookahead and adversarial reasoning that LLMs lack natively.
Addresses partial observability explicitly — modeling the opponent's hidden information rather than treating the game as fully observable or ignoring uncertainty.

No prior system, to the paper's knowledge, has combined these three elements for competitive Pokémon or for complex partially observable games more broadly. The paper's contribution is the demonstration that this combination is both feasible (requiring no training) and effective (achieving expert-level performance), thereby establishing a new paradigm for LLM-augmented game agents.

How This Paper Positions Itself Relative to Existing Work

The paper frames its contribution through several specific contrasts that define its intellectual niche:

Against RL from scratch: PokéChamp requires no task-specific training, instead relying on the LLM's pre-existing knowledge. Section 1 draws this contrast explicitly: "Unlike prior work [in RL-based game AI], PokéChamp does not train exhaustively Pokémon." This is not just an implementation detail — it is a philosophical claim about the viability of generalist agents that transfer knowledge across domains without environment-specific optimization.

Against LLM-only agents: PokéChamp integrates structured search, addressing the planning failures that plague prompting-based approaches. The paper is not arguing that LLMs are insufficient — it is arguing that LLMs plus search is the right combination, with the LLM serving as a domain-informed heuristic for pruning the search tree, modeling the opponent, and evaluating positions. The LLM's role is to make minimax tractable in a massive state space, not to replace minimax.

At the intersection of game theory and language models: Section 6 explicitly locates the work at the "intersection of imitation learning, best response estimation, and Nash equilibrium approximation." This is a precise theoretical framing: pure imitation learning (predicting what humans do from the battle dataset) would estimate the best response to the current metagame but would be exploitable by opponents who deviate from meta; computing the exact Nash equilibrium is intractable in a game of this complexity. PokéChamp's minimax formulation "finds a conservative action that approximates the true best response" — it is not claiming optimality, but rather robustness to adversarial play. The paper is transparent about the open theoretical question: "the exact relationship between our method and the optimal best response remains an open question for future investigation."

As infrastructure for future research: Beyond the agent itself, the paper positions its dataset (3 million battles, 500k high-Elo matches), its puzzle benchmarks (1v1 scenarios, Terastallization and Dynamax puzzles), and its game engine updates as enabling infrastructure for the broader research community. Section 1 concludes: "We hope this work fosters further research that leverage Pokémon battle as benchmark to integrate LLM technologies with game-theoretic algorithms addressing general multiagent problems." The ambition is not merely to build a strong bot but to establish Pokémon battles as a standard testbed for LLM-planning integration, analogous to how Chess and Go served as testbeds for classical game AI.

Practical constraints as design principles: The paper repeatedly emphasizes that its architectural choices are driven by real-world deployment constraints — the 150-second clock, the 15-second turn limit, the requirement for zero-shot operation. These are not afterthoughts; they are first-class design considerations that shape the search depth, the pruning aggressiveness, and the decision to use LLMs as value functions rather than expanding to terminal states. This contrasts with much prior game AI research that assumes effectively unbounded computation during inference.

3. Technical Approach

3.1 Reader Orientation

PokéChamp is a minimax tree search agent for Pokémon battles where the LLM serves as an intelligent heuristic to prune the search space, model the opponent, and evaluate intermediate positions — it replaces three traditionally hand-engineered or learned components of game-tree search with prompted LLM calls, requiring no training or fine-tuning. The system addresses the problem that classical minimax is computationally intractable in Pokémon battles (the state space is astronomically large, roughly $10^{354}$ for the first turn) while pure LLM prompting lacks structured lookahead. The "shape" of the solution is: use the LLM's pre-existing knowledge of Pokémon strategy to aggressively narrow the branching factor (action sampling and opponent modeling), then apply shallow minimax search (only a few plies deep) with an LLM-based evaluation function at the leaves, producing a single recommended action that is safe against adversarial opponent play.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, which interact in a pipeline on each turn of the battle:

Approximate World Model (Game Transition Simulator): Given the current observation (the player's full team info and the opponent's partially known team), this component estimates the hidden opponent state — specifically, the attack and defense stats of the opponent's Pokémon — using historical Showdown usage data. It then uses a local battle simulator and the standard damage formula to compute, for each available action, an admissible one-step heuristic: the minimum number of turns to knock out the opponent's current Pokémon, and the minimum number of turns for the opponent to knock out the player's Pokémon.
Player Action Sampling (LLM Prompt): The LLM receives a structured prompt containing the team strategy, the current observable state, recent battle history, the one-step lookahead heuristics, and the set of legal actions. It outputs a small set of candidate actions — typically a handful of the most strategically viable moves and switches — which become the edges of the minimax search tree at the root node. This aggressively prunes the search space from dozens of possible actions to a manageable few.
Opponent Modeling (LLM Prompt + Historical Data): For each candidate player action, the system must predict what the opponent will do in response. This is harder than action sampling because the opponent's full state is partially hidden. The LLM receives a prompt similar to the player's but framed from the opponent's perspective. The system also uses historical data from real-player games to estimate the opponent's likely stat distributions (EV spreads) given their observable behavior. The combination of LLM reasoning and statistical priors produces a small set of predicted opponent actions for each node.
Minimax Tree Search (Depth-K): The system builds a game tree rooted at the current state, with depth limited to K plies (typically 1–2 full turns, meaning 2–4 plies in the alternating move structure). At each node, edges correspond to the LLM-sampled actions for the player to move. The algorithm applies the standard minimax recursion: the player maximizes over their action choices at each player node, the opponent minimizes at each opponent node, and leaf nodes are evaluated by the LLM value function.
LLM Value Function: At leaf nodes (depth K), rather than simulating to the end of the game — which could be 50+ turns away — the system prompts the LLM to evaluate the game state. The LLM assigns a numerical score based on factors like move effectiveness, remaining Pokémon counts, type advantages, speed relationships, and the overall win probability. The minimax algorithm backs these scores up the tree to select the root action with the best worst-case outcome.

Information flows on each turn: the observable state $\rightarrow$ the approximate world model computes heuristics $\rightarrow$ the action-sampling prompt generates candidate player actions $\rightarrow$ for each candidate, the opponent-modeling prompt generates opponent responses $\rightarrow$ the simulation projects forward one step $\rightarrow$ this process repeats to depth K $\rightarrow$ at leaves, the value function prompt scores the resulting positions $\rightarrow$ minimax backs up the scores to select the most robust action $\rightarrow$ the action is executed on Pokémon Showdown.

3.3 Roadmap for the Deep Dive

I will explain these components in the following order, which builds from the foundation (what happens before search) upward to the full algorithm:

First, the mathematical formalization of Pokémon as a partially observable Markov game (Section 3). This establishes notation and precisely defines what the system is trying to solve — the minimax objective under partial observability. Without this, the algorithmic choices in later sections would lack a rigorous motivation.
Second, the approximate world model (Section 4.1), which is the pre-computation layer. Before the LLM is ever called, the system computes one-step lookahead heuristics using the battle engine and historical statistics. This is crucial because it provides the "ground truth" admissible information that keeps the LLM grounded in game mechanics, preventing it from hallucinating impossible outcomes.
Third, the player action sampling module (Section 4.1), which uses the LLM to prune the search space. I will detail the exact prompt structure, what information is included and why, and how the LLM's output is combined with tool-generated actions (the one-step lookahead and the Abyssal bot's top switch).
Fourth, the opponent modeling module (Section 4.1), which addresses partial observability. I will explain how hidden stats are estimated from historical data, how the LLM predicts opponent actions, and the crucial difference between this module and player action sampling.
Fifth, the LLM value function (Section 4.1), which evaluates leaf nodes. I will detail the scoring criteria, the prompt design, and why an LLM-based evaluation is chosen over traditional handcrafted heuristics or learned value networks.
Sixth, the full minimax tree search algorithm (Section 4), which assembles all three LLM modules. I will explain the depth-K cutoff, how the three LLM roles interact, and the critical distinction between the search that happens internally (the minimax tree) and what happens externally (executing the selected action and observing the real opponent's response).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems integration paper whose core idea is that three strategically placed LLM calls — for proposing actions, predicting opponent responses, and evaluating positions — can replace the hand-engineered or learned components of a minimax tree search, producing a competitive agent with zero task-specific training.

Mathematical Formalization: Pokémon as a Partially Observable Markov Game

The paper formalizes Pokémon battles as a partially observable Markov game (POMG) to provide a rigorous foundation for the minimax approach and to make precise why opponent modeling is necessary. The tuple defining the game is:

$(S, X, Y, A, B, H, P, r)$

where:

$S$ is the latent state space — the complete, ground-truth description of the game at any point. This includes both players' full teams (all six Pokémon with their exact moves, abilities, items, stats, current HP, status conditions, and any temporary modifiers), the game history, and any environmental factors (weather, entry hazards, terrain). No player ever observes $S$ directly; it is the "God's-eye view" of the battle.
$X$ and $Y$ are the observation spaces (also called infosets) for the max-player (the learner) and the min-player (the opponent), respectively. An observation $x \in X$ contains: the complete information about the learner's own team (all stats, moves, items, abilities, current HP, status), the history of all publicly observable events (moves used, switches, damage dealt, faints, etc.), and the currently visible information about the opponent's active Pokémon (species, current HP percentage, status conditions, stat changes, but NOT its full moveset, item, or exact EVs/IVs). The observation does NOT include the opponent's benched Pokémon species, their moves, items, abilities, or stats — this is the partial observability.
$A$ and $B$ are the action spaces for the max-player and min-player. An action is either: (a) selecting one of the active Pokémon's up-to-four moves, (b) switching to one of the up-to-five benched Pokémon (the one currently active cannot be selected), or (c) a generation-specific action like Terastallization (which changes the Pokémon's type) possibly combined with a move. The action space size varies by game state but can be up to approximately 10–15 legal actions per turn in typical situations, larger when Tera is available.
$H$ is the horizon length — the maximum number of turns before the game terminates (by a win condition, a timeout, or a draw). Battles can last from 6 to over 100 turns, so $H$ is variable and potentially large.
$P: S \times A \times B \to \Delta(S)$ is the transition function. Given a current latent state $s$ , the player's action $a$ , and the opponent's action $b$ , it produces a probability distribution over next states $s'$ . The distribution arises from multiple sources of randomness: move accuracy (a move might miss with probability, e.g., 0.1 for a 90%-accurate move), damage rolls (damage is multiplied by a uniform random factor in [0.85, 1.00]), critical hit probability (1/24 by default), and secondary effect probabilities. The transition is therefore stochastic, not deterministic.
$r: S \times A \times B \to [0, 1]$ is the reward function. The paper uses a sparse binary reward: 1 for a win, 0 for a loss. Intermediate states produce zero reward except at game termination. This is the standard episodic game formulation where the objective is to maximize the probability of winning.

The paper imposes two structural assumptions that are critical for the minimax algorithm to be well-defined:

Tree structure: For any state $s_h$ at step $h$ , there exists a unique history $(s_1, a_1, b_1, \ldots, s_{h-1}, a_{h-1}, b_{h-1})$ of past states and joint actions that leads to $s_h$ . This means the game cannot have information-state cycles that would create multiple paths to the same state — a property that holds in Pokémon because the state encodes the full history, making every state reachable by exactly one path. This is required because minimax with memoization (transposition tables) assumes tree structure; without it, the same state might be evaluated differently depending on how it was reached, complicating the backup procedure.

Perfect recall: For any infoset $x_h$ of the max-player at step $h$ , there exists a unique history of past observations and actions $(x_1, a_1, b_1, \ldots, x_{h-1}, a_{h-1}, b_{h-1})$ that leads to $x_h$ . This means the player remembers everything they have observed and everything they did — they have access to the full battle log. Perfect recall is a standard assumption in extensive-form game solving that ensures the agent's information grows monotonically and that mixed strategies over action sequences are well-defined. In the Pokémon context, it means the agent's observation at turn $h$ contains the complete history of all publicly observable events from turns 1 through $h-1$ , plus the agent's own private information (their team).

What this means for the agent. The formalization establishes that the learner (PokéChamp) is the max-player trying to maximize the probability of winning, and the opponent is modeled as the min-player trying to minimize it. The agent must choose actions based only on its observation $x_h$ , not on the latent state $s_h$ — it does not know the opponent's benched Pokémon, their moves, or their stats. The minimax search must therefore operate over infosets, not states, and must explicitly reason about what the opponent might do given what the opponent can observe.

The paper defines the value function for a policy pair $(\mu, \nu)$ (where $\mu: X \to \Delta(A)$ is the max-player's policy and $\nu: Y \to \Delta(B)$ is the min-player's) as the expected sum of rewards over the trajectory:

$V^{\mu,\nu} = \mathbb{E}_{\mu,\nu} \left[ \sum_{h=1}^{H} r(s_h, a_h, b_h) \right]$

where the expectation is taken over trajectories $(s_1, a_1, b_1, \ldots, s_H, a_H, b_H)$ sampled by following the two policies and the stochastic transition function.

What it computes: the expected total reward (which, because rewards are binary and sparse, is simply the probability of winning) when the max-player follows policy $\mu$ and the min-player follows policy $\nu$ . This couples the two policies — the max-player's value depends on what the opponent does.

Why this form: the expectation over trajectories is necessary because the transition is stochastic (moves can miss, damage has variance) and because both players may use mixed (randomized) strategies. The summation over $h$ is trivial for binary terminal rewards — it simplifies to the probability of reaching a winning terminal state — but the formulation is general for any reward structure.

The Nash equilibrium is defined as a pair $(\mu^*, \nu^*)$ where neither player can improve by unilaterally changing their strategy:

$\inf_\nu V^{\mu^*,\nu}(s) = V^{\mu^*,\nu^*}(s) = \sup_\mu V^{\mu,\nu^*}(s)$

Under the tree structure and perfect recall assumptions, the minimax theorem holds:

$\sup_\mu \inf_\nu V^{\mu,\nu}(s) = V^{\mu^*,\nu^*}(s) = \inf_\nu \sup_\mu V^{\mu,\nu}(s)$

What this means operationally: The Nash equilibrium achieves a value that is simultaneously the best the max-player can guarantee against a worst-case opponent ( $\sup_\mu \inf_\nu$ ) and the worst the min-player can force against a best-case max-player ( $\inf_\nu \sup_\mu$ ). When the opponent is truly adversarial (playing to minimize the max-player's win probability), the Nash equilibrium strategy is the optimal counter. The paper explicitly states: "Nash equilibrium is also the optimal policy against the adversarial opponent." This is the game-theoretic justification for using minimax: if the agent computes a strategy that is safe even against the worst possible opponent behavior, it will perform well against any real opponent, regardless of their skill level or playstyle.

The gap between theory and practice. Computing the exact Nash equilibrium for a POMG with state space $10^{354}$ is intractable. The paper does not claim to compute the Nash equilibrium. Instead, the minimax tree search with LLM-based pruning and evaluation is an approximation — it searches a shallow tree over a heavily pruned action space, evaluates leaves with a heuristic (the LLM value function), and assumes the opponent is adversarial (the min operator) rather than modeling their true policy. The paper acknowledges in Section 6: "the exact relationship between our method and the optimal best response remains an open question for future investigation." The formalization thus serves as a target specification — what an ideal agent would compute — and the remaining sections describe the practical approximations that make search feasible within the 15-second turn limit.

Approximate Game Transition (World Model)

Before the LLM is ever called, PokéChamp runs a pre-computation step that estimates the effects of each legal action using a simplified battle simulator. This component is crucial because it provides ground-truth admissible heuristics — computed from the actual game mechanics, not hallucinated by the LLM — that ground the subsequent LLM prompts in physical reality.

The central challenge is that the true game state $s_h$ is not fully observed. The learner sees their own team completely but only sees partial information about the opponent. Specifically, the attack stat $A$ and defense stat $D$ of the opponent's Pokémon in the damage formula are unknown — these depend on the opponent's Effort Values (EVs), Individual Values (IVs), and nature, which are hidden until revealed through damage calculations or other in-game signals.

Stat estimation from historical data. To estimate $A$ and $D$ , PokéChamp uses historical usage statistics from the Pokémon Showdown metagame. For each Pokémon species in each competitive format, the Showdown platform tracks aggregate data on common EV spreads, natures, items, and moves. For example, in Gen 9 OU, a Dragapult might most commonly run 252 EVs in Speed and 252 EVs in Special Attack with a Timid nature. PokéChamp queries this historical data and assigns the opponent's hidden stats to the most likely (modal) values given the observed species and format. This is a point estimate rather than a full distribution — the system assumes the opponent uses the statistically most common set, which is a reasonable approximation for casual-to-mid-level play (most players use standard sets) but can be exploited by opponents who deliberately use off-meta stat distributions.

Additionally, for stats that are partially revealed through gameplay — for example, if the opponent's Speed stat causes them to move first or second relative to a Pokémon with known Speed — the system can refine its estimate using the LLM. The LLM is prompted to infer the opponent's Speed investment based on turn-order observations: "opponent's Dragapult moved before player's Garchomp, so Dragapult must have at least X Speed." This inference is incorporated into the stat estimates.

The damage formula. Once stats are estimated, the system computes expected damage for each move using the standard Pokémon damage formula. The paper provides the core equation (Equation 2 in the main text, expanded in Equation 3 in the appendix):

$\text{Damage} = \left\lfloor \frac{1}{50} \left\lfloor \left\lfloor \left( \frac{2}{5} \cdot \text{Level} + 2 \right) \cdot \text{Power} \cdot \frac{A}{D} \right\rfloor + 2 \right\rfloor \cdot \text{Other Mechanics} \right\rfloor$

where:

Level is the Pokémon's level (typically 100 in competitive play, making the first term $42$ ).
Power is the base power of the selected move (e.g., 80 for "Shadow Ball," 120 for "Focus Blast" — though Focus Blast has only 70% accuracy, introducing variance).
$A$ is the attacker's relevant Attack or Special Attack stat (depending on whether the move is physical or special).
$D$ is the defender's relevant Defense or Special Defense stat (depending on the move category).
Other Mechanics encapsulates all multiplicative modifiers (Equation 3 in the appendix): Same-Type Attack Bonus (STAB, 1.5× if the move type matches the user's type), type effectiveness (0.25×, 0.5×, 1×, 2×, or 4× depending on type matchups), weather effects (1.5× for Water moves in Rain, 0.5× for Fire moves in Rain), critical hit multiplier (1.5×, with probability 1/24 by default), burn status (0.5× for physical moves if the attacker is burned), random damage variation (uniform in [0.85, 1.00]), and various ability- or item-specific modifiers.

The floor operations ( $\lfloor \cdot \rfloor$ ) indicate integer arithmetic — Pokémon damage is computed in integer steps, with several intermediate rounding stages, making the calculation non-linear and sensitive to small stat differences.

What this computes: for a given player action (move $a$ ) against the opponent's estimated stat distribution, the formula produces an expected damage value (or, more precisely, a range of possible damage values due to the random factor). By comparing this expected damage to the opponent Pokémon's current and maximum HP, the system computes the minimum and maximum number of turns to knock out (KO) the opponent's Pokémon — called the "turns to KO" heuristic.

Why this form matters for the agent: the one-step lookahead provides an admissible lower bound on the number of turns needed to achieve a KO, assuming no switches, no healing, no status changes, and average damage rolls. This is admissible because the model uses the most likely stat spreads and known game mechanics — it is a computable, conservative estimate that does not rely on LLM knowledge. The heuristic serves three purposes:

It grounds the LLM's strategic reasoning. The action-sampling prompt includes turns-to-KO estimates for each move, so the LLM does not need to compute damage values itself (which it would do unreliably). It can focus on higher-level strategy: "Move A KOs in 2 turns but Move B KOs in 3 turns; however, Move B has better type coverage against potential switch-ins."
It provides a fallback action. If the LLM fails to produce useful candidate actions (e.g., due to API errors or nonsensical output), the system can fall back to a purely heuristic action: select the move with the fewest turns to KO the opponent, or the "top switch choice from the Abyssal bot" (Section 4.1). The Abyssal bot is a rule-based heuristic bot used in official Pokémon games, and its switching logic — which considers type matchups, hazard damage, and speed tiers — serves as an additional non-LLM fallback.
It enables Terastallization reasoning. Terastallization changes the user's type(s) for the remainder of the battle, which affects both STAB calculations and type-effectiveness multipliers. The one-step lookahead computes four variants of the turns-to-KO heuristic: (a) neither player Terastallizes, (b) only the opponent Terastallizes, (c) only the player Terastallizes, (d) both Terastallize. This gives the LLM explicit information about how the type change affects the matchup, which would be very difficult for the LLM to compute reliably in its head.

The approximate transition model is simplified in two ways that reduce computational cost:

Expected value only. Rather than sampling many stochastic rollouts, the system uses expected damage (the average of the uniform [0.85, 1.00] random factor, i.e., 0.925). This ignores variance — a move with 120 base power and 70% accuracy has high expected damage but high variance (it might miss entirely), while a move with 90 base power and 100% accuracy has lower expected damage but zero variance. The expected-value simplification means the one-step lookahead does not distinguish between these cases; the LLM must account for accuracy and variance in its higher-level reasoning.
No opponent action uncertainty in the heuristic. The one-step lookahead assumes the opponent uses their "best" attacking move (the one that minimizes turns to KO the player's Pokémon). It does not model opponent switching, status moves, or setup moves (like Swords Dance, which boosts the opponent's Attack but deals no damage). This is a first-order approximation — the LLM's opponent modeling step handles more nuanced opponent behavior.

The output of the approximate world model is a structured text block (illustrated in Appendix B.6, Listing 1) that is injected directly into the action-sampling prompt. For each of the player's available moves and for each possible Tera configuration, it lists the moves with their turns-to-KO estimate, and symmetrically lists the opponent's moves with their turns-to-KO estimates against the player. This block is the machine-computed, ground-truth backbone of the prompt.

Player Action Sampling (LLM as Search-Space Pruner)

This module is the first LLM call in the minimax pipeline. Its purpose is to reduce the branching factor of the search tree from the full legal action space (typically 8–15 actions) to a small set of strategically plausible actions (typically 3–6). The key insight is that the LLM's pre-existing knowledge of Pokémon strategy — type matchups, common tactics, team synergy, threat assessment — can identify which actions are worth exploring in depth, replacing the need for hand-crafted heuristics or learned policy networks.

The prompt structure. The LLM receives a multi-part prompt containing the following information (Section 4.1, "Player action sampling"):

Team strategy: Before the battle begins or at the start of the turn, the LLM generates an overall strategic assessment based on both players' full (visible) teams. This includes: which Pokémon on each side are the biggest threats, what the general game plan should be (e.g., "break through the opponent's defensive core with special attackers," "preserve Gholdengo to block Defog"), and which matchups are favorable or unfavorable. This strategic framing provides context for the action decision — the LLM is not just picking a move in isolation but choosing an action that advances a coherent plan.
Observable state $x_h \in X$ : A structured description of the current battle state. This includes the player's active Pokémon (species, current HP, status conditions, stat changes, active item, ability), the opponent's active Pokémon (species, current HP percentage estimate, status conditions, stat changes, revealed item/ability), the player's benched Pokémon (species, current HP, status), the opponent's revealed benched Pokémon (if any have been seen via switching), and any field conditions (weather, entry hazards on either side, terrain, screens).
Battle history: The events of the last $N$ turns (the paper does not specify an exact $N$ , but it must be sufficient to capture the recent strategic context — typically the last 2–5 turns). This represents the perfect recall assumption from the mathematical formalization: the agent remembers everything it has observed. The history includes which moves were used, which Pokémon switched in or out, damage dealt, status inflicted, and any other publicly observable events. This is crucial for opponent modeling — if the opponent has switched Pokémon three times in the last four turns, that signals a particular playstyle (likely defensive/stall) that should influence action selection.
One-step lookahead heuristics: The output of the approximate world model described above — the turns-to-KO estimates for each legal move against the opponent, and the opponent's turns-to-KO estimates against the player. This is the "admissible heuristic information regarding the likely effect of actions under the current metagame" (Figure 2 caption). It is structured as a compact table or list, formatted for the LLM to parse easily.
Available actions: The set of legal actions $A_h(x_h)$ at the current infoset. This includes all usable moves (moves that are not disabled, have remaining PP, and are not prevented by status like Sleep or Taunt) and all legal switches (Pokémon that are not fainted and not currently active). The LLM is constrained to only output actions from this set.

What the LLM produces. The LLM outputs a set of candidate actions $\{a_h^{(i)}\}_{i=1}^{m} \subset A_h(x_h)$ , where $m$ is typically 3–6 actions. These are the actions that will form the edges of the minimax search tree at the root node. The LLM is prompted to select actions that are diverse in strategic intent — for example, including one offensive move, one defensive switch, and one utility move — rather than multiple moves of the same type, because the subsequent tree search will explore the consequences of each distinct strategy.

Augmentation with tool-generated actions. In addition to the LLM's output, the system always includes two non-LLM actions in the candidate set (Section 4.1):

Top move from one-step lookahead: The move with the fewest turns to KO the opponent's current Pokémon, according to the approximate world model. This is a purely greedy offensive heuristic — it assumes no switching and no setup moves — but it ensures that the agent always considers the straightforward "hit the opponent hard" option.
Top switch choice from the Abyssal bot: The Abyssal bot's recommended switch, which uses rule-based logic considering type matchups, predicted opponent moves, hazard damage, and speed tiers. This ensures the agent considers at least one plausible switch even if the LLM overlooked it, and provides a defensive/positional alternative to the offensive one-step lookahead move.

By combining LLM-generated actions with tool-generated actions, the system hedges against LLM failures: if the LLM hallucinates nonsensical actions, the tool-generated actions remain as a safety net. If the LLM generates high-quality strategic actions, they augment the greedy heuristics with nuanced plans.

Why this approach over alternatives. The paper contrasts this with three alternatives:

Full action space search: Exploring all 10–15 legal actions at each node, even at shallow depth, would produce a branching factor that is computationally prohibitive. A depth-4 tree (2 full turns) with branching factor 12 at each node has $12^4 = 20,\!736$ leaf nodes, each of which requires at least a value-function evaluation and possibly further simulation. The 15-second turn limit precludes evaluating this many nodes, especially with LLM API latency.
Random or rule-based pruning: Using a fixed heuristic to prune actions (e.g., "always consider the three strongest moves by base power") would miss contextual strategic considerations. A move with 40 base power that provides crucial utility (e.g., "Stealth Rock," which sets entry hazards, or "U-turn," which switches while dealing damage) would be pruned by a naive power-based heuristic, but the LLM knows to include it when the strategic context demands it.
Learned policy network (RL approach): Training a neural network to predict high-value actions would require millions of games of self-play or expert data. This is the approach rejected by the paper's thesis of zero-shot, training-free deployment. The LLM serves as a pre-trained, generalist policy approximator — its knowledge of Pokémon strategy is imprecise but broadly correct, and the subsequent tree search corrects for its imprecisions by exploring multiple candidates and evaluating their consequences.

The LLM call as a black box. The paper emphasizes that "the LLM functions as a black box, allowing for flexibility in selecting any frontier model based on budget and computational resources" (Section 1). The prompt structure is model-agnostic — the same prompts work for GPT-4o, Llama 3.1 8B, or any future model that understands Pokémon mechanics. The performance degrades gracefully with model size (Table 2 shows PokéChamp with Llama 3.1:8b still achieving a 56% win rate against Abyssal vs. 84% with GPT-4o), but the architecture does not depend on any model-specific feature.

Opponent Modeling (LLM + Historical Data for Adversarial Reasoning)

This module is the second LLM call in the minimax pipeline, and it addresses the core challenge of partial observability: the opponent's actions depend on their hidden information (their benched Pokémon, movesets, items, stats), which the agent must infer from observable behavior. The opponent modeling module produces a set of predicted opponent actions $\{b_h^{(j)}\}_{j=1}^{m'} \subset B_h(y_h)$ for each node in the search tree where it is the opponent's turn to move.

Why opponent modeling is harder than action sampling. When generating candidate actions for itself, the agent has complete information about its own team and complete knowledge of its own strategic intent. The LLM is essentially reasoning: "Given my resources and my plan, what good moves do I have?" When modeling the opponent, the LLM must reason: "Given what I have observed about the opponent's team (which is incomplete), what would a rational opponent do?" This requires inference under uncertainty — the opponent might have a threatening Pokémon in the back that would alter their optimal action, but the agent cannot see it.

Stat estimation from historical data (revisited). Before the LLM is prompted, the system refines its estimate of the opponent's hidden stats using historical usage data, as described in the approximate world model section. This is more important for opponent modeling than for action sampling because the opponent's actions depend on what their Pokémon are capable of. For example, if the opponent's Landorus-Therian is commonly run with a Choice Scarf (an item that boosts Speed by 50% but locks the user into one move), the opponent's action space is significantly constrained — they can only use the first move they select, and they cannot switch freely. Inferring the item from the opponent's behavior (e.g., if Landorus moved before a naturally faster Pokémon, it likely has a Choice Scarf) allows the agent to narrow the set of possible opponent actions.

The paper describes this as: "we leverage the statistical data from Pokémon Showdown, including Pokémon move pools, EV spreads, item usage, etc., aggregated from gameplay over a given period" (Section 4.1). The dataset of 3 million battles provides the empirical distribution of these hidden variables conditioned on the Pokémon species and format.

The LLM-based action prediction. The opponent modeling prompt is structured similarly to the player action sampling prompt but framed from the opponent's perspective. It includes:

The observable state from the opponent's viewpoint (what the opponent sees of the player's team — essentially the mirror of the player's observation).
The battle history (the same public events, but interpreted from the opponent's interests).
The one-step lookahead heuristics from the opponent's side (which moves minimize turns to KO the player's Pokémon).
The opponent's legal actions as far as they can be inferred (the opponent's active Pokémon's known moves, plus the possibility of switching to unknown benched Pokémon).

What the LLM produces. The LLM outputs a set of likely opponent actions, representing the opponent's plausible strategies. Unlike player action sampling, where the system includes tool-generated fallbacks, the opponent modeling relies entirely on the LLM (supplemented by historical stat data). The paper does not describe an explicit non-LLM fallback for opponent action prediction — the min-operator in the minimax tree will consider all LLM-predicted opponent actions as the set over which the opponent minimizes.

The difficulty of opponent action prediction. Table 1 quantifies the challenge: even at high Elo (1800), the model's Top-1 opponent action prediction accuracy is only 15%, compared to 30% for player action prediction. The gap reflects the partial observability — without knowing the opponent's full team and hidden stats, predicting their exact action is inherently uncertain. However, Top-5 accuracy is 53% for opponents at 1800 Elo, meaning that while the exact action is hard to pinpoint, the set of 5 most likely actions captures the opponent's true action over half the time. For the minimax search, this is sufficient: the agent does not need to predict exactly what the opponent will do; it needs to consider what the opponent could do that would be worst for the agent, and the LLM's predicted set serves as a reasonable approximation of the opponent's feasible strategic options.

Why not learn an opponent model from data? The paper's dataset of 3 million battles could, in principle, be used to train a supervised opponent model that predicts actions from game state. The paper chooses not to do this for two reasons: (1) training an opponent model would violate the zero-shot, training-free principle that the paper advocates, and (2) a learned opponent model would be static — it would reflect the metagame at the time the training data was collected — while the LLM, combined with updated historical statistics (which can be refreshed periodically), can adapt to metagame shifts without retraining. The tradeoff is that the LLM-based predictions are noisier and less calibrated than a well-trained classifier, but the minimax search's conservative min-operator compensates by considering multiple opponent actions rather than relying on a single prediction.

LLM Value Function (Position Evaluation at Search Depth K)

This module is the third LLM call in the minimax pipeline. Its purpose is to evaluate leaf nodes of the search tree — game states that are reached after $K$ plies of forward simulation — without expanding the tree further to terminal states (which could be 50+ turns away). The value function maps an infoset at the cutoff depth to a scalar score representing the agent's probability of winning from that position, which is then backed up the tree by the minimax recursion.

Why an LLM-based value function? In classical game AI (e.g., Chess engines), the evaluation function is a hand-crafted linear combination of material and positional features (pawn structure, piece mobility, king safety), often tuned by expert knowledge or self-play. In deep RL (e.g., AlphaZero), the value function is a neural network trained to predict game outcomes from states. Neither approach is available to PokéChamp: hand-crafting an evaluation function for Pokémon would require encoding hundreds of type matchups, thousands of moves, and complex interactions between abilities, items, and stat changes — an enormous engineering effort that would be brittle to metagame changes. Training a neural value network would require millions of self-play games, violating the zero-shot constraint.

The LLM offers a third path: it has already internalized a rough evaluation function from its pre-training data, which includes Pokémon strategy guides, forum discussions, and competitive analyses. The LLM "leverages common sense and gameplay knowledge from the internet, serving as an effective approximation for value functions" (Section 4.1). It is not perfectly calibrated, but it provides a signal that is correlated with the true win probability, and the minimax search over multiple candidate actions hedges against its imprecisions.

The prompt structure. The value function prompt describes the game state at a leaf node — typically 2–4 plies into the future from the current turn — and asks the LLM to evaluate it. The prompt includes (Section 4.1):

The full projected state: which Pokémon are active on each side, their current HP, any status conditions, stat changes, field effects (weather, hazards, terrain), and which Pokémon are fainted or benched on each side.
The projected battle history up to that point (the events that occurred in the simulated plies leading to this leaf).
The same strategic context as the action-sampling prompt (team strategy, threat assessment).

Scoring criteria (explicit, from the paper). The LLM is asked to generate a score based on the following factors:

Positive factors (increase the score):
- Effectiveness of the player's current moves against the opponent's active Pokémon (high damage, super-effective types).
- Number of remaining Pokémon on the player's side relative to the opponent (having more Pokémon is a resource advantage, as switching and sacs are more available).
- General win probability assessment (a holistic judgment of momentum and position).
Negative factors (decrease the score):
- Excessive switching by the player (frequent switching suggests the player is being forced into unfavorable matchups and is losing momentum).
- Effectiveness of the opponent's moves against the player's active Pokémon (the opponent has strong offensive pressure).
- Speed disadvantage (the opponent moves first, giving them priority and potentially KO-ing before the player can act).
- Number of remaining Pokémon on the opponent's side (more opponent Pokémon means more resources to wear down the player).
- Strength of remaining opponent Pokémon (the opponent's bench contains threatening sweepers or walls that the player has not yet neutralized).

What the LLM produces. The exact format of the output is not specified in detail in the paper, but it can be inferred as a numerical score (or a text description that is parsed into a score) that is comparable across leaf nodes. The score is used directly in the minimax backup — the max-player selects the leaf with the highest score among their options, and the min-player selects the leaf with the lowest score among the opponent's options.

Why the specific criteria were chosen. The criteria balance immediate tactical advantage (move effectiveness, speed) with long-term strategic resource advantage (remaining Pokémon, bench strength). This is important because Pokémon battles have a strong resource-management component — sacrificing a Pokémon early to gain momentum is often a winning strategy, but an evaluation function that only considers the current matchup (who is ahead on HP right now) would undervalue the player who has more healthy Pokémon in reserve. The LLM's ability to consider both immediate and long-term factors is a key advantage over a pure one-step lookahead (which only sees the current turn).

Limitations acknowledged by the paper. The value function's accuracy is bounded by the LLM's knowledge, which is frozen at the training cutoff date and may not reflect the current metagame. Additionally, the value function evaluates a snapshot of the game state — it does not simulate further beyond the cutoff depth. For strategies that play out over many turns (like stall teams that rely on gradual chip damage and status), the cutoff-depth evaluation may fail to capture the long-term trajectory. The paper observes that PokéChamp struggles against stall tactics and excessive switching (Appendix C.1, C.2), which is consistent with a value function that overweights immediate damage and undervalues positional endurance.

Full Minimax Tree Search Algorithm

This section describes how the three LLM modules — action sampling, opponent modeling, and value function — are assembled into the full minimax search, and how the resulting action is selected and executed in the live battle environment.

The minimax recursion. At the current infoset $x_h$ , the agent computes a recommended action $\hat{a}_h$ according to the standard depth-K minimax criterion:

$\hat{a}_h = \arg\max_{a_h} \min_{b_h} \mathbb{E}_{x_{h+1}} \ldots \max_{a_{h+K-1}} \min_{b_{h+K-1}} \mathbb{E}_{x_{h+K}} V(x_{h+K})$

where:

$a_h$ iterates over the player's LLM-generated candidate actions at the root.
$b_h$ iterates over the opponent's LLM-predicted actions at the opponent's response node.
$\mathbb{E}_{x_{h+1}}$ represents the (expected-value) transition through the approximate world model — given a pair of actions $(a_h, b_h)$ , the system simulates one turn of battle to produce the next observation $x_{h+1}$ .
The max-min alternation continues for $K$ layers (plies), where $K$ is typically 2–4 (1–2 full turns, since each turn involves two plies: player action then opponent action).
At depth $K$ , the LLM value function $V(x_{h+K})$ evaluates the leaf infoset without further expansion.

What this computes, operationally:

Root node (max-player turn): The LLM generates $m$ candidate player actions. For each candidate $a_h$ :
- Opponent response node (min): The LLM generates $m'$ predicted opponent actions. For each $b_h$ :
  - Transition: The world model simulates one turn with actions $(a_h, b_h)$ , producing an expected next observation $\mathbb{E}[x_{h+1}]$ (treating stochastic effects by their expected values).
  - Second player turn (if $K \geq 2$ ): At $x_{h+1}$ , the LLM generates candidate actions for the player again, and the recursion continues.
  - Leaf evaluation: At depth $K$ , the LLM value function scores the resulting observation.
  - Min backup: The opponent's effective response is the one that minimizes the backed-up score (including any deeper search outcomes). This captures the worst-case opponent behavior consistent with the LLM's opponent model.
- Max backup: The player's effective score for action $a_h$ is the minimum over opponent responses (the guaranteed value under worst-case opponent play).
Action selection: The root action $\hat{a}_h$ is the one that maximizes this guaranteed value — the action with the best worst-case outcome.

Why minimax and not expectimax? Expectimax would model the opponent as playing according to their true policy (the probability distribution over their actions), computing an expected value rather than a minimum. The paper chooses minimax — modeling the opponent as adversarial — for robustness: "The overall strategy corresponds to minimax tree search which is described at Figure 4," and "the Nash equilibrium is also the optimal policy against the adversarial opponent" (Section 4). In practice, real opponents are not fully adversarial — they play to win, not specifically to make PokéChamp lose — but the minimax assumption is a conservative heuristic that protects against the worst case and tends to produce safer, more robust play. The tradeoff is that minimax may be overly cautious: an action that performs well against most opponent responses but poorly against one unlikely response might be rejected, even though the unlikely response is not actually in the opponent's repertoire. The paper does not quantify this tradeoff empirically.

Depth K and computational constraints. The paper does not specify an exact value of $K$ (it says "we limit expansion to a depth of k steps" in Section 4, but the value of $k$ is not explicitly stated). However, the tight time constraints (15 seconds per turn, 150 seconds per match) imply that $K$ must be small — likely 1 or 2 full turns (2 or 4 plies). Evidence from the experiments supports this: PokéChamp lost approximately one-third of its online ladder games due to timeout (Section 5.3, "For about one third of the games, PokéChamp lost by exceeding the turn time limit"), indicating that even with aggressive pruning, the LLM API latency and the simulation cost make deeper search infeasible within the time limit.

The paper acknowledges that performance is "currently limited by the accuracy of opponent modeling and the method's online computational budget. By increasing the breadth and depth size of the search, we expect to see further improvement's to the performance of the method" (Section 6). This suggests that $K$ is constrained by practical latency, not by a principled decision about optimal search depth.

The simulation uses expected values, not Monte Carlo sampling. To reduce computational cost, the world model "computes the expected value within these transitions" (Section 4.1) rather than sampling multiple stochastic outcomes. This means that moves with accuracy less than 100% are treated as if they always hit but deal reduced expected damage (damage multiplied by accuracy). This simplification removes variance but introduces bias: a 70%-accurate move with 120 base power has the same expected damage as a 100%-accurate move with 84 base power, but the risk profiles are very different. The LLM value function can partially compensate by recognizing high-variance plays in its evaluation, but the tree search itself does not explicitly model risk.

How the selected action relates to the live game. After the minimax search selects $\hat{a}_h$ , the agent sends this action to the Pokémon Showdown server, which resolves it along with the real opponent's action (which may be different from any of the LLM's predicted opponent actions). The battle proceeds to the next turn, and the entire process repeats: the new observation is fed into the approximate world model, the action-sampling and opponent-modeling LLMs are prompted with the updated state, and a fresh minimax tree is built. There is no persistence of the search tree across turns — each turn's search starts from scratch, with all LLM calls made anew. This is a consequence of the zero-shot architecture: there is no learned value function or policy that persists across turns, only the frozen LLM weights and the historical statistics database.

The PokéChamp-Fast variant (speed-optimized). To address the timeout problem, the paper experimented with PokéChamp-Fast (Section 5.3), which uses the LLM to dynamically choose between two modes: (1) simply taking the heuristic action from the one-step lookahead (no search), or (2) performing a truncated minimax search. The LLM decides based on the current game state — presumably, complex strategic situations trigger the search, while straightforward situations default to the greedy heuristic. However, this variant performed worse (Elo 1150–1310, compared to 1300+ for the standard method), and the paper attributes the degradation to "a covariate shift in the competitive metagame over time. Specifically, the LLM's pre-training data, which has a fixed cutoff date, likely biases its decision-making even when provided with updated historical statistics about the current metagame" (Section 5.3). This is a revealing negative result: the LLM's frozen knowledge can override real-time statistical evidence, causing the agent to make decisions based on outdated metagame assumptions. It also suggests that the full minimax search, by explicitly exploring multiple futures, acts as a corrective to the LLM's biases — the search evaluates consequences objectively, while the "choose whether to search" meta-decision relies entirely on the LLM's (potentially biased) judgment.

Relationship to Nash equilibrium computation. The paper is careful not to claim that the minimax search computes a Nash equilibrium. The approximations — finite depth $K$ , action-space pruning via LLM sampling, expected-value simulation, LLM-based value function — mean that the computed action $\hat{a}_h$ is an approximation to the true minimax value. The relationship is: if the LLM's action sampling were perfect (always including the optimal action in the candidate set), the opponent modeling were perfect (always including the opponent's worst-case response), the value function were perfectly accurate, and the depth were sufficiently large, then the algorithm would compute the exact Nash equilibrium. In practice, each of these components introduces error, and the paper does not provide a theoretical bound on the resulting suboptimality. This is not a weakness of the paper — such bounds are notoriously difficult for POMGs — but it means the empirical results (win rates, Elo ratings) are the primary evidence for the approach's effectiveness, not theoretical guarantees.

4. Key Insights and Innovations

Innovation 1: LLMs as Modular Heuristic Replacements in Classical Tree Search — Not as End-to-End Decision Makers

The paper's most fundamental conceptual contribution is not any single module but rather the architectural insight that an LLM can serve as a drop-in heuristic for three distinct, traditionally engineered or learned components of a minimax tree search — action proposal, opponent prediction, and position evaluation — and that this decomposition is both sufficient for expert-level play and achievable with zero task-specific training. This is distinct from prior work that treats LLMs either as end-to-end planners (where the model directly outputs the chosen action) or as search-free reasoners (where chain-of-thought substitutes for lookahead).

Before PokéChamp, the dominant paradigm for LLM-based game agents — exemplified by PokéLLMon (Hu et al., 2024b) — was to prompt the model with the game state and have it output an action, possibly with self-consistency (sampling multiple outputs and taking the consensus). This treats the LLM as a policy network, albeit one implemented via text generation rather than learned weights. The planning failures documented in Section 1.1 — LLMs cannot play Nash Tic-Tac-Toe (Topsakal & Harper, 2024), LLM-powered Nethack agents underperform heuristic bots (Jeurissen et al., 2024) — are failures of this architecture: the LLM attempts to implicitly simulate future states and evaluate them within a single forward pass, a task it is demonstrably bad at.

PokéChamp reconfigures the LLM's role entirely. Instead of asking "what should I do?", it asks three narrower questions, each of which plays to the LLM's strengths while compensating for its weaknesses through structured algorithmic scaffolding:

"What are the plausible things I could do?" (action sampling). This is a generation task — the LLM produces a small set of strategically diverse candidates. The LLM does not need to rank them or choose between them; it just needs to recall relevant strategies from its training data. The tree search handles the evaluation and selection.
"What might the opponent do in response?" (opponent modeling). This is an inference under uncertainty task — the LLM must reason about hidden information from observable signals. Again, the LLM does not need to be perfectly accurate; it just needs to produce a plausible set of opponent moves that covers the adversary's likely strategic options, from which the min-operator selects the worst case.
"If the game reached this state, who is winning?" (value function). This is an evaluation task — the LLM judges a static snapshot using its pre-existing strategic knowledge. The evaluation does not need to be perfectly calibrated; it just needs to be correlated with the true win probability, and the minimax backup over multiple trajectories smooths out its errors.

What makes this decomposition intellectually distinctive is that it maps each LLM call to a well-defined computational role within a classical algorithm that has known theoretical properties (minimax), rather than asking the LLM to approximate the entire algorithm implicitly. The LLM provides the domain knowledge that makes the search tractable — pruning the branching factor from ~12 to ~4, evaluating leaves without simulating 50 more turns — while the algorithm provides the structured lookahead and adversarial robustness that the LLM lacks. Neither component alone would suffice: pure minimax is computationally intractable in a $10^{354}$ state space, and pure LLM reasoning fails at planning. The combination is what works.

This framing is significant beyond Pokémon. It suggests a general template for integrating LLMs into classical AI algorithms for partially observable, adversarial domains: identify the components of the algorithm that require domain-specific heuristics (pruning, evaluation, opponent reasoning), replace each with a prompted LLM call, and let the algorithm handle the structured computation. This is not an incremental improvement over LLM-only agents — it is a fundamentally different architecture that changes how we think about the division of labor between neural knowledge and symbolic reasoning.

Evidence for the decomposition's effectiveness comes from the ablation implicit in the baselines: PokéLLMon (LLM-only, no search) achieves a 40% win rate against Abyssal in Gen 9 OU, while PokéChamp (LLM + minimax) achieves 84% (Table 2). The 44-percentage-point gap cannot be attributed to better prompting alone — it reflects the structural advantage of explicit adversarial search over implicit LLM reasoning.

Innovation 2: Partial Observability as a First-Class Target for LLM-Based Opponent Modeling — Not an Afterthought

Most prior work on LLM game agents either operates in fully observable environments (Chess, Go, or simplified text games) or treats partial observability as something to be managed implicitly by the LLM's "understanding" of the game state description. PokéChamp elevates opponent modeling to a named, architecturally distinct module with its own prompt, its own data sources (historical usage statistics), and its own measurable accuracy (Table 1). This makes partial observability a first-class target for LLM integration rather than an implicit challenge left to the model's general reasoning capabilities.

The paper provides direct evidence that opponent modeling is a harder problem than self-action selection — and quantifies exactly how much harder. Table 1 shows Top-1 prediction accuracy of 30% for the player's own actions versus 15% for opponent actions across Elo ratings. This gap is not a failure of the LLM; it is a reflection of the genuine information asymmetry in the game. The player knows their own team completely; the opponent's team is partially hidden. No amount of prompting sophistication can overcome the fundamental uncertainty — there is a lower bound on prediction error imposed by the missing information.

What makes this contribution conceptually novel is that PokéChamp does not try to eliminate this uncertainty (by predicting a single opponent action) but rather operationalizes it within the minimax framework. The opponent modeling module produces a set of plausible opponent actions, and the min-operator in the tree search assumes the worst among them. This converts partial observability from a prediction problem (which is inherently noisy) into a robustness problem (which the minimax algorithm is designed to handle). The agent does not need to know what the opponent will do; it needs to know what the opponent could do that would be worst, and the LLM's predicted action set provides a bounded approximation of that space.

This approach stands in contrast to the dominant paradigm in competitive game AI, which has historically relied on two strategies for handling partial observability:

Bayesian inference over a known type space (as in Poker AI, e.g., Libratus; Brown & Sandholm, 2018). This requires enumerating all possible opponent "types" (e.g., all possible hands in Poker) and computing posterior beliefs over them from observed actions. In Pokémon, the type space — all possible movesets, EV spreads, items, and abilities for six Pokémon — is far too large to enumerate explicitly, making pure Bayesian approaches computationally infeasible.
Self-play from scratch (as in AlphaStar; Vinyals et al., 2019). The agent learns to handle partial observability implicitly through millions of games, developing internal representations of opponent state without an explicit opponent model. This requires massive training, which PokéChamp avoids.

PokéChamp's approach — using an LLM to propose a plausible set of opponent configurations based on historical metagame data and observable behavior, then planning conservatively against that set — is a novel middle ground. It leverages the LLM's knowledge of common strategies (which serves as an informative prior over the opponent's hidden state) while using minimax to protect against the cases where the opponent deviates from the most likely configuration. The negative result with PokéChamp-Fast (Section 5.3) reinforces the importance of this approach: when the LLM was asked to dynamically choose between search and a greedy heuristic, the LLM's frozen priors overrode real-time statistical evidence, suggesting that explicit opponent modeling within a search framework is more robust than delegating meta-decisions to the LLM.

Innovation 3: Verifier (Minimax) Over-Optimization Is Not the Bottleneck — Model Capability and Metagame Shift Are

The paper provides a diagnostic negative result that distinguishes the failure modes of LLM-augmented game agents from those commonly observed in other test-time compute scaling contexts. In the reference paper on compute-optimal test-time scaling (the example in the prompt), the primary bottleneck for further improvements was verifier over-optimization: search algorithms (beam search, lookahead search) found solutions that scored highly under the process reward model but were actually incorrect, causing performance to degrade at high budgets. The optimal strategy was to avoid aggressive search on easy problems where the verifier was reliable and apply it only where it could provide genuine guidance.

PokéChamp reveals a qualitatively different bottleneck. The two major failure modes documented in the paper are:

Timeout (computational budget). Approximately one-third of online ladder games were lost because the agent exceeded the turn time limit (Section 5.3). This is not a failure of the minimax search to find good actions — it is a failure to complete the search within the available time. The bottleneck is API latency and search tree size, not the accuracy of the evaluation function.
Strategic vulnerabilities to stall and excessive switching (Appendix C). PokéChamp struggles against opponents who employ long-term attrition strategies or frequent Pokémon switching. These strategies exploit the limited search depth imposed by the time constraint — stall teams win over 20+ turns, far beyond the 2–4 ply lookahead, and excessive switching punishes the agent's inability to recognize and exploit switching patterns. The bottleneck here is depth, not breadth: the LLM's value function at depth K cannot capture the cumulative effects of passive damage (entry hazards, status, weather) over many turns.

The PokéChamp-Fast experiment (Section 5.3) reveals a third bottleneck: metagame shift. When a speed-optimized variant was tested on March 3, 2025 — approximately six months after the main experiments — its Elo decreased to 1150–1310 (from 1300+ for the standard method), and the paper attributes this to "a covariate shift in the competitive metagame over time. Specifically, the LLM's pre-training data, which has a fixed cutoff date, likely biases its decision-making even when provided with updated historical statistics about the current metagame." This is a distinctive finding: the LLM's frozen knowledge can be a liability when the environment changes, overriding real-time data with outdated priors. In the reference compute-optimal scaling paper, the process reward model was trained on fresh data from the base model and did not face this temporal distribution shift; here, the LLM's value function and action proposals are frozen at training time, and the world changes around them.

These three bottlenecks — time, depth, and temporal staleness — collectively paint a picture of what does not limit PokéChamp: the quality of the action sampling (pruning) and the accuracy of the value function (evaluation) appear sufficient for strong play. The limiting factors are computational (search depth × breadth must fit in 15 seconds), not representational (the LLM knows enough to make good decisions if given enough time). This is a significant diagnostic finding because it tells the research community where to invest effort: faster inference (model distillation, speculative decoding, cached prompts) and deeper search (asynchronous evaluation, progressive deepening) are more likely to improve PokéChamp than better opponent modeling or value function prompting. It also suggests that the architecture's zero-shot constraint — no training — is what creates these bottlenecks; a small amount of fine-tuning (e.g., distilling the LLM's evaluations into a fast neural value network) could dramatically reduce latency and enable deeper search, though at the cost of the zero-shot property.

Innovation 4: Pokémon Battles as a POMG Benchmark with a Real-Player Dataset, Synthetic Puzzles, and Live Ladder Evaluation

While the paper's primary contribution is the PokéChamp agent architecture, a secondary but significant innovation is the infrastructure it establishes for benchmarking LLM-game-theory integration. The paper compiles the "largest real-player Pokémon battle dataset" with over 3 million games (500k+ high-Elo matches), develops synthetic puzzles to test specific skills (1v1 optimal sequences, Terastallization and Dynamax usage), and evaluates against live human opponents on the online ladder — creating a multi-faceted evaluation ecosystem that is more comprehensive than typical game AI benchmarks.

What makes this infrastructure distinctive is its dimensionality: it tests agents along three axes that are often conflated in simpler benchmarks.

Offline action prediction (Table 1) measures whether the agent understands what humans do in real games — a supervised learning / imitation learning metric that captures alignment with the metagame. This is a necessary but not sufficient condition for strong play: an agent that perfectly predicts human actions would still lose if it cannot handle opponents who deviate from meta.
Synthetic puzzles (Section 5.2, Appendix C) measure whether the agent can solve specific, isolated sub-problems — finding the optimal move sequence in a 1v1 scenario, leveraging generation-specific mechanics, handling stall and switching strategies. These are essentially unit tests for strategic reasoning, analogous to the endgame tablebases used in Chess AI evaluation.
Live ladder play (Section 5.3) measures whether the agent can compete against real, adaptive human opponents under genuine time pressure and partial observability. This is the hardest and most ecologically valid metric — humans will identify and exploit the agent's weaknesses (the paper notes that "games played against humans who knew they were playing against PokéChamp were able to determine the excessive switching limitation and tailor an adversarial strategy"), making it a true adversarial test rather than a static evaluation.

Few prior game AI benchmarks provide all three axes. Chess engines are typically evaluated on puzzle suites (tactical problems) and against other engines, but rarely against humans in real-time online play with Elo tracking. RL-based game agents (AlphaStar, OpenAI Five) are evaluated against humans but require massive training and do not release offline prediction benchmarks. The paper's combination — a public dataset for imitation learning, synthetic puzzles for targeted skill evaluation, and live ladder for adversarial stress-testing — provides a template for how game AI benchmarks should be constructed for the LLM era, where agents are expected to leverage pre-existing knowledge rather than learn from scratch.

The dataset itself has additional value beyond the PokéChamp evaluation. Figure 5 shows that higher-Elo games tend to be longer (right panel), which is a non-trivial empirical finding: it suggests that skill in Pokémon battles is expressed partly through the ability to extend games and manage resources over more turns, rather than through quick knockouts. This aligns with the paper's observation that PokéChamp struggles with long-game strategies (stall) — the very skill that distinguishes high-Elo human play from low-Elo play. The dataset enables future research to quantify this relationship and to train agents that explicitly model game-length dynamics.

The puzzle design warrants specific mention because it isolates mechanics that would be lost in aggregate win-rate statistics. The Terastallization and Dynamax puzzles (Figure 6) test whether the agent understands type-changing and power-increasing mechanics — capabilities that require both knowledge of the mechanic's rules and strategic reasoning about when to deploy it. A brute-force RL agent might learn to use these mechanics effectively through millions of trials, but an LLM agent must understand them from pre-training knowledge alone. The fact that PokéChamp solves these puzzles demonstrates that its LLM-based reasoning can handle generation-specific mechanics zero-shot, which is a capability that RL-from-scratch agents lack without retraining for each new game generation.

The paper's explicit recommendation to "limit use of the online ladder evaluation to avoid data contamination and overwhelming human users" (Section 7, Impact Statement) is itself a noteworthy contribution to AI evaluation ethics — acknowledging that benchmarking against humans imposes externalities on the human community and advocating for primarily offline, agent-vs-agent evaluation as the standard going forward.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper compiles a dataset of over 3 million Pokémon battles from the Pokémon Showdown platform, spanning various competitive formats, with more than 500,000 high-Elo matches (Elo > 1600). The dataset includes detailed information on team compositions, move choices, and battle outcomes. For action prediction experiments, replays are collected from a spectator's perspective, and key hidden information (EV spreads, item choices) is reverse-engineered by supplementing the observed data with historical likelihood estimates. For synthetic puzzle evaluation, the authors create 1,000 1v1 battle scenarios drawn from the Gen 8 Random Battles meta, filtered to include only matchups with a feasible win condition (i.e., solvable by the Abyssal bot). For full-game evaluation, the testbed is the Gen 9 OverUsed (OU) format on Pokémon Showdown, which includes the Terastallization mechanic and uses custom-built teams. A separate set of Gen 8 Random Battles experiments is also conducted, both with and without the Dynamax mechanic. All full-game evaluations consist of at least 25 matches between any two methods, yielding a minimum of 100 games per method for Elo calculation.
Base model(s). The primary LLM used is GPT-4o-2024-05-13 (Achiam et al., 2023), accessed via API as a black box. Ablation experiments also use Llama 3.1 8B (Dubey et al., 2024), an open-source model, to test performance scaling with model size. The models are chosen to span both frontier proprietary and smaller open-source capabilities, and neither model receives any task-specific fine-tuning — all prompting is zero-shot with respect to Pokémon-specific training. The paper argues that these models possess sufficient pre-existing Pokémon knowledge from internet-scale pre-training to serve as effective heuristics within the minimax framework.
Metrics. The primary metrics are: (1) Win rate — the percentage of games won against a specific opponent, computed as the fraction of matches where the agent's final action sequence leads to all opponent Pokémon fainting. (2) Elo rating — a relative skill rating computed from pairwise match outcomes using the standard Elo system, where a higher Elo indicates stronger performance against other rated opponents. The Elo scale is anchored to the Pokémon Showdown ladder, where human players have known Elo distributions. (3) Average number of turns — the mean game length for a given agent, used as a proxy for playstyle (shorter games suggest aggressive, knockout-focused strategies; longer games suggest defensive or stall-oriented play). For the online ladder experiments, the primary metric is Elo reached after a fixed number of games (50 games), with an additional breakdown of losses due to timeout versus losses due to gameplay. For action prediction experiments, the metric is Top-k accuracy — the fraction of turns where the true human action appears in the agent's top-k predicted actions, computed separately for player action prediction and opponent action prediction across different Elo brackets.
Baselines. The paper compares against six baselines: (1) PokéLLMon (Hu et al., 2024b) — the prior state-of-the-art LLM-based Pokémon agent, which uses self-consistency prompting (sampling multiple LLM outputs and taking the majority action) without any tree search. PokéLLMon is evaluated with GPT-4o. (2) Abyssal Bot — a rule-based heuristic bot used in official Pokémon games, which follows pre-defined heuristics for move selection and switching based on type matchups, base power, and simple threat assessment. This is the strongest non-LLM baseline. (3) One Step Lookahead Bot — the paper's own admissible heuristic bot, which selects actions purely based on the one-step lookahead described in Section 4.1 (choosing the move that minimizes turns to KO the opponent's current Pokémon or the top switch from a simple heuristic). This ablation tests the value of the minimax search and LLM modules above the raw heuristic. (4) Max Power Bot — a bot that selects moves based solely on power level, ignoring type matchups, switching, and strategic context. This serves as a weak lower bound. (5) Random Bot — a bot that selects actions uniformly at random from legal moves. This serves as the theoretical floor. (6) Human players on the online ladder — for the live ladder evaluation, PokéChamp faces real human opponents of varying skill levels on the Pokémon Showdown Gen 9 OU ladder, providing an ecologically valid benchmark against adaptive, adversarial play.
Generation budget / compute accounting. The paper does not measure compute in terms of FLOPs or LLM API calls — the primary resource constraint is wall-clock time. Each Pokémon battle has a total clock of 150 seconds per player, with 15 additional seconds added per turn. Exceeding the time limit results in an automatic loss. This means the relevant "budget" is not the number of LLM calls but whether the entire minimax search — including LLM API latency, world model simulation, and tree backup — can be completed within approximately 10–12 seconds per turn (allowing buffer for network latency). The paper does not report the exact number of LLM calls per turn, the API latency per call, or the depth K of the search tree — these are treated as implementation details constrained by the time budget. For the PokéChamp-Fast variant, the budget constraint is relaxed by allowing the LLM to dynamically skip the minimax search and use only the one-step lookahead, reducing latency at the cost of decision quality.
Cross-validation / statistical protocol. For head-to-head bot comparisons, each matchup consists of at least 25 games, and Elo ratings are computed from the full matrix of pairwise match outcomes using a minimum of 100 games per method. No cross-validation is used for bot-vs-bot or bot-vs-human evaluation — results are computed directly from the observed match outcomes. For the 1v1 puzzle benchmark, 1,000 scenarios are sampled from Gen 8 Random Battles and filtered for feasible win conditions, but no train/test split is used — all 1,000 scenarios are used for evaluation. The paper does not report confidence intervals, standard errors, or statistical significance tests for any win rate or Elo comparisons, which limits the ability to assess whether observed differences (e.g., 84% vs. 40% win rate) are statistically robust given the sample sizes. For the online ladder evaluation, PokéChamp played 50 games over a one-week period (September 1–7, 2024), and the Elo reported is the rating achieved after those 50 games — no uncertainty quantification is provided around this point estimate.

Main Quantitative Results

Head-to-Head Bot Comparisons in Gen 9 OU (Full Custom Teams with Terastallization)

The primary head-to-head results are reported in Table 2 (Gen 9 OU with Terastallization) and Table 3 (Gen 9 OU mirror matchups without Terastallization). PokéChamp with GPT-4o achieves the highest win rate and Elo across all comparisons.

Against heuristic and LLM baselines (Table 2):

PokéChamp (GPT-4o) vs. Abyssal: 84% win rate, with an Elo of 1268 and an average game length of 15.7 turns. This is the strongest result in the table — PokéChamp wins more than 4 out of 5 games against the strongest rule-based bot.
PokéChamp (Llama 3.1:8b) vs. Abyssal: 56% win rate, Elo 1204, average 16.9 turns. The open-source 8B model still achieves a positive win rate against Abyssal, demonstrating that the architecture's effectiveness is not dependent on frontier-scale models.
PokéLLMon (GPT-4o) vs. Abyssal: 40% win rate, Elo 1020, average 22.6 turns. This is the prior LLM state-of-the-art, and the 44-percentage-point gap between PokéChamp (GPT-4o) and PokéLLMon (GPT-4o) — both using the same underlying model — isolates the contribution of the minimax tree search architecture over prompting-only approaches. The much longer average game length for PokéLLMon (22.6 turns vs. 15.7 for PokéChamp) suggests that PokéLLMon struggles to close out games efficiently, consistent with a lack of strategic lookahead.
One Step Lookahead vs. Abyssal: 44% win rate, Elo 1107, average 17.9 turns. The fact that the pure heuristic bot (no LLM, no minimax search) achieves a higher win rate against Abyssal than PokéLLMon (44% vs. 40%) is a striking result — it suggests that the LLM's strategic reasoning in PokéLLMon is actively harmful in some situations, perhaps by suggesting sophisticated but suboptimal plays that a simple greedy heuristic avoids.
Abyssal (self-play baseline): N/A win rate (by definition), Elo 1117, average 17.9 turns.
Max Power vs. Abyssal: 16% win rate, Elo 885, average 19.5 turns. This confirms that raw power maximization is a weak strategy, as expected.
Random vs. Abyssal: 0% win rate, Elo 399, average 21.2 turns. The random bot never wins, establishing the floor.

The Elo ordering — PokéChamp (GPT-4o) at 1268, Abyssal at 1117, One Step Lookahead at 1107, PokéLLMon at 1020, Max Power at 885, Random at 399 — shows a clear hierarchy where the LLM-augmented minimax agent substantially outperforms all baselines.

Mirror matchups without Terastallization (Table 3):

To isolate the effect of team composition (removing the advantage of superior teambuilding), the paper evaluates mirror matchups where both sides use identical teams, and Terastallization is disabled:

PokéChamp (GPT-4o) vs. Abyssal: 90% win rate. This is higher than the 84% in Table 2, suggesting that team composition differences in the non-mirror setting slightly favor Abyssal (i.e., PokéChamp's team choices are not fully optimized, and when teams are equalized, PokéChamp's strategic advantage is even larger).
PokéChamp (Llama 3.1:8b) vs. Abyssal: 83% win rate. The 8B model performs much closer to GPT-4o in mirror matchups (83% vs. 90%) than in non-mirror (56% vs. 84%), suggesting that team composition understanding — which may require deeper Pokémon knowledge — is where the smaller model loses ground.
PokéLLMon (GPT-4o) vs. Abyssal: 60% win rate. This is higher than the 40% in Table 2, consistent with the interpretation that team composition quality is a confounding factor in the non-mirror comparisons.
One Step Lookahead vs. Abyssal: 56% win rate. This is higher than the 44% in Table 2, again suggesting that team composition effects are non-trivial.

The pattern across Tables 2 and 3 reveals that team composition is a significant confounding variable — all methods perform better in mirror matchups — but PokéChamp maintains a dominant advantage regardless.

Pairwise win rate matrix (Figure 7, left): The paper visualizes the pairwise win rates between methods in Gen 9 OU. The matrix confirms that PokéChamp (GPT-4o) has a winning record against every other method, and the hierarchy is largely transitive (A > B and B > C implies A > C), with no non-transitive cycles reported.

Impact of team composition (Figure 7, right): A separate analysis shows that win rates vary substantially depending on which specific teams are used, confirming that teambuilding skill — which is outside the scope of the battle agent itself — is a significant factor in overall performance. This motivates the mirror-matchup analysis in Table 3 as a cleaner evaluation of battle decision-making.

Gen 8 Random Battles (Without and With Dynamax)

The Random Battles format tests adaptability to unpredictable team compositions, removing the teambuilding confound.

Without Dynamax (Table 4):

PokéChamp (GPT-4o) vs. Abyssal: 70% win rate.
PokéChamp (Llama 3.1:8b) vs. Abyssal: 64% win rate.
PokéLLMon (GPT-4o) vs. Abyssal: 56% win rate.
One Step Lookahead vs. Abyssal: 44% win rate.

The 70% win rate for PokéChamp (GPT-4o) in Random Battles is lower than the 84% in Gen 9 OU (Table 2), which is expected given the increased uncertainty of random team assignments — the agent cannot rely on knowing its own team's synergies in advance. However, the 14-percentage-point gap over PokéLLMon (70% vs. 56%) remains substantial. Notably, the One Step Lookahead bot achieves the same 44% win rate as in Gen 9 OU, suggesting that its performance is insensitive to format (it relies only on the current matchup, not team composition), while PokéChamp's LLM-based reasoning provides a larger advantage in formats where team knowledge matters.

With Dynamax (Table 5):

PokéChamp (GPT-4o): 56% win rate vs. Abyssal, Elo 1273, average 17.1 turns.
PokéChamp (Llama 3.1:8b): 52% win rate vs. Abyssal, Elo 1184, average 19.1 turns.
PokéLLMon (GPT-4o): 36% win rate vs. Abyssal, Elo 1048, average 22.5 turns.
Abyssal: Elo 1213, average 19.0 turns.
One Step Lookahead: 16% win rate vs. Abyssal, Elo 998, average 18.9 turns.
Max Power: 4% win rate vs. Abyssal, Elo 787, average 23.2 turns.
Random: 0% win rate vs. Abyssal, Elo 493, average 24.3 turns.

Dynamax reduces PokéChamp's win rate against Abyssal from 70% to 56% (comparing Table 4 to Table 5), suggesting that Dynamax — which provides temporary HP doubling and move power increases — compresses the skill gap by introducing high-variance "swing" turns that are harder to plan around with shallow search. However, PokéChamp still maintains a 20-percentage-point lead over PokéLLMon (56% vs. 36%) and a 40-point lead over One Step Lookahead (56% vs. 16%). The Elo ordering is preserved: PokéChamp (GPT-4o) at 1273, Abyssal at 1213, PokéChamp (Llama 3.1) at 1184, PokéLLMon at 1048, One Step Lookahead at 998, Max Power at 787, Random at 493.

The shorter average game length for PokéChamp (17.1 turns vs. 22.5 for PokéLLMon and 24.3 for Random) indicates that PokéChamp closes out games more efficiently, even with Dynamax variance. This is consistent with the minimax search identifying knockout sequences that end games quickly rather than engaging in prolonged attrition.

Pairwise matchup matrix for Random Battles with Dynamax (Figure 9): The matrix visualizes all pairwise win rates. The rows and columns are ordered by Elo, producing an approximately upper-triangular matrix (higher-Elo methods tend to beat lower-Elo methods), confirming transitivity.

Action Prediction Against Human Players (Table 1)

The paper evaluates the player action sampling and opponent modeling modules in isolation by testing how well the LLM predicts human actions from real game data across different Elo ratings. The baseline is random prediction: 7% for player action prediction (since there are typically ~14 legal actions) and less than 1% for opponent prediction (since the hidden information makes the effective action space much larger).

Player action prediction accuracy (Top-1 through Top-5):

Elo	Top-1	Top-2	Top-3	Top-4	Top-5
1200	30%	40%	48%	53%	58%
1400	26%	23%	30%	32%	43%
1600	27%	30%	39%	44%	53%
1800	30%	42%	55%	62%	66%

Opponent action prediction accuracy (Top-1 through Top-5):

Elo	Top-1	Top-2	Top-3	Top-4	Top-5
1200	16%	30%	40%	46%	53%
1400	16%	17%	20%	26%	39%
1600	13%	21%	26%	35%	40%
1800	15%	29%	40%	50%	53%

Key patterns:

Top-1 accuracy is low for both player and opponent prediction (26–30% for player, 13–16% for opponent) — the LLM's point predictions are far from perfect. However, this does not necessarily indicate poor strategic understanding: the paper notes that "there may be multiple correct actions in many situations, as several strategies could be equally viable." The LLM might propose a strategically sound action that differs from the human's choice without being wrong.
Top-5 accuracy is substantially higher for both (43–66% for player, 39–53% for opponent) — when the LLM is allowed to propose a set of 5 candidate actions, it captures the human's action a majority of the time at most Elo levels. For the minimax search, this is the relevant metric: the action sampling module needs to include good actions in its candidate set, not identify the single best action. The Top-5 accuracy of ~60% suggests the candidate set is reasonably comprehensive.
Opponent prediction is systematically worse than player prediction (Top-1: 15% vs. 30% at Elo 1800) — this gap reflects the genuine partial observability in the game. Without seeing the opponent's benched Pokémon and hidden stats, predicting their exact action is inherently more difficult. However, the fact that the gap persists at all Elo levels suggests the LLM is not simply memorizing common patterns (which would apply equally to player and opponent) but is genuinely limited by the missing information.
Trends with Elo are inconsistent. For player prediction, Top-1 accuracy is roughly flat (26–30%) while Top-5 accuracy generally increases with Elo (from 58% at 1200 to 66% at 1800), suggesting that higher-Elo players' actions are more predictable in distribution (they play more "standard" strategies that are easier to capture in a broad candidate set). For opponent prediction, there is a dip at mid-Elo (Top-1 drops to 13% at 1600, Top-5 to 40%), with Elo 1800 recovering to levels similar to Elo 1200. The paper does not offer a detailed explanation for this non-monotonicity, but it may reflect that mid-level players use more varied and less predictable strategies than either beginners (who play straightforwardly) or experts (who converge on meta-optimal play).

The prediction results validate that the LLM's action sampling and opponent modeling — while imperfect — provide a candidate set that is rich enough to support tree search. The minimax framework does not require the LLM to predict the opponent's action correctly; it requires the opponent's true action to be somewhere in the candidate set or for the candidate set to include actions that are "close enough" in strategic impact. The Top-5 accuracy of 40–53% for opponent prediction suggests this condition is approximately met, though with significant room for improvement.

Puzzle Benchmarks: 1v1 Battles, Terastallization, Dynamax (Section 5.2)

1v1 Battles (1,000 scenarios): PokéChamp achieves an 86% win rate on the 1,000-scenario 1v1 puzzle benchmark, compared to PokéLLMon's 76% win rate. The 10-percentage-point advantage in a constrained setting without switching isolates the contribution of the one-step lookahead heuristic and the minimax search over self-consistency prompting. Since 1v1 battles remove the partial observability of benched Pokémon and the complexity of switching decisions, the gap is primarily attributable to better move selection within a single matchup — PokéChamp computes exact turns-to-KO and evaluates the tradeoff between offense and defense, while PokéLLMon relies on the LLM's implicit damage estimation.

Terastallization and Dynamax puzzles (Figure 6): The paper provides qualitative examples (not quantitative success rates) of PokéChamp correctly using generation-specific mechanics. In the Terastallization puzzle (Figure 6, left), PokéChamp recognizes that after Glimmora is revealed as the opponent's chosen Pokémon, Roaring Moon's type changes via Terastallization alter the matchup, and the agent correctly decides to switch rather than stay in and take a super-effective hit. In the Dynamax puzzle (Figure 6, right), PokéChamp uses Dynamax to increase HP and move power, enabling it to knock out two opponent Pokémon in succession — a strategic use of the mechanic that requires understanding both the immediate damage boost and the multi-turn implications (Dynamax lasts 3 turns, so planning the sequence of KOs matters).

The paper does not report quantitative win rates or solve rates for these specific puzzles, making it difficult to assess how robustly PokéChamp handles these mechanics compared to simpler scenarios. The examples serve as existence proofs that the LLM-based reasoning can handle the mechanics, but the lack of aggregate statistics leaves open the possibility that PokéChamp fails on a non-trivial fraction of similar puzzles.

Stall and excessive switching puzzles (Appendix C, Figure 8): These are diagnostic failure cases, not benchmarks with quantitative scores. In the stall puzzle (Figure 8, left), PokéChamp initially selects Darkrai against Blissey (correct: Darkrai's Focus Blast threatens Blissey), but then switches to Enamorus, which faints to entry hazards. After re-entering Darkrai, a missed Focus Blast increases uncertainty in the agent's decision-making, leading it to switch again rather than maintaining position. The paper diagnoses this as a failure of the value function to account for long-term attrition and the limited lookahead depth. In the excessive switching puzzle (Figure 8, right), PokéChamp repeatedly uses Focus Blast (a Fighting-type move) even as the opponent switches between two Pokémon that resist Fighting, demonstrating a failure to adapt to the opponent's switching pattern — the agent over-commits to a predicted optimal move rather than recognizing that the opponent is exploiting its predictability.

These failure cases are valuable diagnostics, but they are anecdotal — no quantitative frequency of such failures is reported, making it difficult to assess whether these are rare edge cases or common failure modes that significantly impact performance.

Online Ladder Against Human Players (Section 5.3)

PokéChamp was deployed on the Pokémon Showdown online ladder in the Gen 9 OU format between September 1 and September 7, 2024, playing against real human opponents under standard time controls (150 seconds total, 15 seconds per turn).

Timeout losses: Approximately one-third of the games were lost because PokéChamp exceeded the turn time limit. This means the win rates reported below are conditional on the agent not timing out — the unconditional win rate (including timeouts) is lower, though the exact figure is not reported. The paper estimates a "projected" Elo by removing timeout losses: "If removing losses due to timeout, we can estimate its Elo rating, where PokéChamp will achieve an Elo rating of 1300–1500, which places it in the top 30%–10% of players."

Win rate among non-timeout games: Within the remaining two-thirds of games, PokéChamp achieved a 76% win rate. This is the relevant metric for assessing the agent's strategic quality independent of the latency bottleneck, and it aligns with the head-to-head bot results (84% against Abyssal in Table 2, 76% against PokéLLMon — though against humans the win rate is naturally lower due to human adaptability).

Elo trajectory: After 50 games, PokéChamp reached an Elo above 1300 on the Showdown ladder. The paper projects that without timeouts, it would stabilize in the 1300–1500 range. Figure 1 visualizes this Elo range relative to the human player distribution, showing PokéChamp in the 70th–90th percentile. For context, the paper maps other methods' Elo onto this scale using their win rates against PokéChamp: PokéLLMon, Abyssal, and other bots are plotted at their estimated Elo based on the pairwise win rates from Tables 2 and 5.

PokéChamp-Fast variant (March 3, 2025): A speed-optimized variant where the LLM dynamically chooses between the one-step lookahead action (no search) and a truncated minimax search achieved an Elo of only 1150–1310 and did not converge. The paper attributes this to metagame covariate shift — the LLM's frozen pre-training data biases its meta-decisions despite updated historical statistics — and notes that performance was lower than the standard method tested six months earlier. This is a significant negative result: it suggests that the full minimax search's structured evaluation serves as a corrective to the LLM's potentially outdated priors, and that delegating the "whether to search" decision to the LLM itself introduces a new failure mode.

Adversarial exploitation by human opponents: The paper reports an important qualitative finding: "games played against humans who knew they were playing against PokéChamp were able to determine the excessive switching limitation and tailor an adversarial strategy to take advantage of these limitations." This highlights a key difference between bot-vs-bot evaluation (where both agents' strategies are fixed) and bot-vs-human evaluation (where humans adapt specifically to exploit the bot's known weaknesses). It also underscores the paper's recommendation for anonymous evaluation — if human opponents know they are facing an AI with specific vulnerabilities, they will exploit them, making win rates non-representative of performance against naïve opponents.

Ablation Studies and Robustness Checks

Model scale ablation (GPT-4o vs. Llama 3.1:8b): Across all formats, PokéChamp with the smaller open-source model underperforms the GPT-4o variant but still substantially outperforms the prior state-of-the-art LLM-based agent (PokéLLMon with GPT-4o). In Gen 9 OU (Table 2), Llama 3.1:8b achieves a 56% win rate against Abyssal vs. GPT-4o's 84%, while PokéLLMon with GPT-4o achieves only 40%. In Gen 8 Random Battles without Dynamax (Table 4), the Llama variant achieves 64% vs. GPT-4o's 70%, while PokéLLMon achieves 56%. In mirror matchups without Terastallization (Table 3), the gap narrows: Llama variant 83% vs. GPT-4o 90%, with PokéLLMon at 60%.

These results demonstrate that the architecture's performance scales with model capability — GPT-4o's larger size and better reasoning yield better strategic proposals and evaluations — but critically, the architecture itself provides a large enough boost that even an 8B model with minimax search outperforms a frontier model without it. This is a robustness check on the claim that the framework, not just the underlying model, is responsible for the performance gains: if the gains were purely from GPT-4o being a stronger model than whatever PokéLLMon used, then Llama 3.1:8b (a weaker model) should be worse than PokéLLMon with GPT-4o. The fact that it is better (64% vs. 56% in Random Battles without Dynamax, Table 4) supports the architectural contribution.

Team composition ablation (mirror matchups, Table 3 vs. Table 2): By removing team composition differences (both sides use identical teams, so any performance difference is attributable to battle decisions), the mirror matchup experiments control for the confounding effect of teambuilding quality. All methods improve in mirror matchups: PokéChamp (GPT-4o) from 84% to 90%, PokéChamp (Llama 3.1:8b) from 56% to 83%, PokéLLMon from 40% to 60%. The fact that the gap between GPT-4o and Llama 3.1 narrows in mirror matchups (from 28 points to 7 points) suggests that a significant portion of the GPT-4o advantage in non-mirror settings comes from superior understanding of team synergies and matchup assessment, rather than from better move-by-move decision-making. This is a subtle but important finding: the LLM's teambuilding knowledge (which Pokémon to bring, with which sets) is a separate capability from its in-battle strategic reasoning, and the two are confounded in standard win-rate comparisons.

Search mechanism ablation (PokéChamp vs. One Step Lookahead vs. PokéLLMon): The comparison between PokéChamp, the One Step Lookahead bot (no LLM, no search), and PokéLLMon (LLM, no search) provides an approximate ablation of the search component:

One Step Lookahead (no LLM, no search) achieves 44% against Abyssal in Gen 9 OU (Table 2). This is the performance of the pure admissible heuristic: greedy offensive moves and rule-based switches.
PokéLLMon (LLM, no search) achieves 40% against Abyssal. The LLM's strategic reasoning without search is actually slightly worse than a simple greedy heuristic — a counterintuitive result suggesting that the LLM sometimes outsmarts itself, proposing complex but suboptimal plays.
PokéChamp (LLM + search) achieves 84% against Abyssal. The jump from 44% to 84% represents the combined effect of LLM-guided pruning, opponent modeling, and value function evaluation within the minimax framework over the heuristic-only alternative.

The 8-percentage-point gap between One Step Lookahead and PokéLLMon is a negative result for prompting-only approaches: even with the same underlying LLM (GPT-4o), the self-consistency prompting in PokéLLMon underperforms a simple heuristic that just picks the highest-damage move. This strongly supports the paper's thesis that LLMs require structured search to translate their knowledge into effective play.

Format and mechanic ablation (Gen 9 OU vs. Gen 8 Random, with/without Terastallization and Dynamax): PokéChamp's performance varies across formats, testing the architecture's robustness to different game rules:

Gen 9 OU (custom teams, Terastallization): 84% win rate against Abyssal (Table 2).
Gen 8 Random (random teams, no Dynamax): 70% against Abyssal (Table 4). The drop from 84% to 70% when moving from custom to random teams reflects the loss of teambuilding advantage and the increased uncertainty of unfamiliar team compositions.
Gen 8 Random (random teams, with Dynamax): 56% against Abyssal (Table 5). The further drop from 70% to 56% when Dynamax is enabled suggests that the mechanic's high-variance swings partially neutralize PokéChamp's strategic advantages.
Mirror matchups (no Terastallization): 90% against Abyssal (Table 3). The increase from 84% to 90% when team composition is equalized and Terastallization removed indicates that both factors — team composition and the Terastallization mechanic — introduce variance that slightly reduces PokéChamp's advantage.

The pattern across these ablations shows that PokéChamp is most dominant in settings with low variance (mirror matchups, known teams, no game-changing mechanics) and its advantage shrinks as variance increases (random teams, Dynamax). This is consistent with the minimax framework: when outcomes are more predictable, the search's evaluation is more accurate, and the advantage over reactive or heuristic strategies is larger. When outcomes are noisy (e.g., Dynamax turns swing the game state unpredictably), the benefit of lookahead is diluted.

Speed-accuracy tradeoff (PokéChamp vs. PokéChamp-Fast, Section 5.3): The PokéChamp-Fast variant is an ablation on the search mechanism itself — it uses the LLM to dynamically decide whether to search at all, falling back to the one-step lookahead when the LLM judges the situation to be straightforward. The result is that PokéChamp-Fast achieves a lower Elo (1150–1310) than the standard PokéChamp (1300+), despite (presumably) having fewer timeout losses. This ablation demonstrates that the full minimax search is not just a latency tax — it provides genuine strategic value that cannot be recovered by a faster heuristic, and the LLM is not reliable at judging when search is necessary versus when it can be skipped. The paper's interpretation — that the LLM's frozen priors override real-time statistics in the meta-decision — suggests that a learned "search-or-not" classifier trained on post-hoc game analysis might outperform the LLM-based dynamic selection, though this would violate the zero-shot constraint.

Online vs. offline evaluation (Section 5.3 vs. Tables 2–5): The online ladder results provide a robustness check against the concern that bot-vs-bot win rates are inflated by the bots' inability to adapt to PokéChamp's specific weaknesses. Human opponents, especially those who know they are facing an AI, can identify and exploit vulnerabilities like excessive switching (Appendix C.2) and timeout susceptibility. The paper's finding that PokéChamp achieves a 76% win rate among non-timeout games against humans — roughly comparable to its 84% win rate against Abyssal — suggests that its strategic advantages against heuristic bots translate to the human setting, at least when the time constraint is not the limiting factor. However, the one-third timeout rate is a significant practical limitation that the lab evaluations (which presumably do not enforce the same strict time limits) may mask.

Critical Assessment

Does the 76% win rate against PokéLLMon and 84% against Abyssal demonstrate that LLM-augmented minimax search outperforms prior approaches?

The central empirical claim — that PokéChamp substantially outperforms both the prior LLM-based state-of-the-art (PokéLLMon) and the strongest heuristic bot (Abyssal) — is supported by the head-to-head results in Table 2 and Table 5, with consistent patterns across formats. The win rate gaps are large (84% vs. 40% against Abyssal for GPT-4o, 64% vs. 56% for the Llama variant in Random Battles without Dynamax), and the Elo ordering is transitive (PokéChamp > Abyssal > PokéLLMon > One Step Lookahead > Random). The mirror matchup results in Table 3 confirm that the advantage persists even when team composition is equalized.

However, the experiments demonstrate a narrower claim than "LLM-augmented minimax search outperforms prior approaches in general." Specifically:

The comparison is against a single LLM-based baseline (PokéLLMon) that uses self-consistency prompting. There are many other ways to integrate LLMs with planning — Tree of Thoughts (Yao et al., 2024), Reasoning via Planning (Hao et al., 2023), or ReAct-style interleaved reasoning (Yao et al., 2022). The paper does not compare against any of these alternatives, even though they are cited in the related work. It is possible that a simpler integration (e.g., using the LLM to generate a plan and then executing it, without explicit adversarial search) would achieve comparable performance at lower latency. The claim is specifically "PokéChamp outperforms PokéLLMon," not "minimax search outperforms all LLM-planning integrations."
The GPT-4o variant and the Llama 3.1:8b variant are not compared to PokéLLMon with Llama 3.1:8b. The ablation that would cleanly isolate the architecture from the model is: PokéChamp (Llama 3.1:8b) vs. PokéLLMon (Llama 3.1:8b). Instead, the paper compares PokéChamp (Llama 3.1:8b) against PokéLLMon (GPT-4o), which confounds architecture and model. The fact that the 8B model with minimax search outperforms GPT-4o with self-consistency is evidence that the architecture provides a large boost, but a same-model comparison would quantify that boost precisely. It is unclear whether PokéLLMon with Llama 3.1:8b would be even weaker (making the architectural advantage larger) or whether the gap between GPT-4o and Llama 3.1:8b is smaller within the PokéLLMon framework (making the architectural advantage smaller).
The Abyssal bot is a "strongest rule-based bot" but its exact capabilities are not described in detail. The paper does not specify what heuristics Abyssal uses, whether it performs any search, or how it handles partial observability. Without this information, it is difficult to assess whether the 84% win rate represents a genuine strategic triumph or whether Abyssal has known weaknesses (e.g., predictable switching patterns, inability to handle Terastallization) that PokéChamp exploits. A comparison against a stronger heuristic baseline — perhaps a bot that uses expectiminimax with a hand-crafted evaluation function — would provide a clearer picture of how much the LLM components contribute beyond what careful engineering could achieve.

Does the live ladder performance (Elo 1300–1500, top 10–30%) demonstrate expert-level play?

The claim that PokéChamp achieves "expert-level" performance is based on its projected Elo of 1300–1500 on the Pokémon Showdown ladder, which places it in the 70th–90th percentile of human players (Figure 1). This claim requires several qualifications:

The Elo is "projected" by removing timeout losses. The paper states that approximately one-third of games were lost due to timeout, and the 1300–1500 range is the Elo the agent would have achieved without those losses. This is not the Elo it actually achieved — the actual Elo, including timeouts, is lower (the paper does not report it). For a human player, "managing the clock" is part of skill, and an agent that times out in one-third of its games would not be considered expert-level regardless of its strategic quality in the remaining games. The projection is useful for diagnosing whether the strategic reasoning (as opposed to the latency) reaches expert level, but it overstates the agent's practical performance.
The sample size is small (50 games) and the time window is short (one week). Elo ratings converge slowly, and 50 games is a relatively small sample — the 95% confidence interval on an Elo estimate from 50 games is roughly ±100–150 points, depending on the rating system's K-factor. The reported range of 1300–1500 may partially reflect this estimation uncertainty rather than genuine performance variability. A longer evaluation period with more games would be needed to narrow the estimate.
The human opponents were not randomly sampled from the ladder distribution. Players at different times of day, days of the week, and Elo ranges have different skill levels and playstyles. The paper does not report the Elo distribution of the human opponents PokéChamp faced — if it was matched primarily against lower-Elo players (as is common when a new account climbs the ladder), the win rate would overestimate performance against the broader player population. The projected Elo of 1300–1500 should be interpreted as the rating the agent would stabilize at after playing a representative sample of opponents, but whether the 50 games constitute such a sample is unclear.
Adversarial adaptation is acknowledged but not quantified. The paper notes that human opponents who knew they were playing against an AI exploited PokéChamp's weaknesses (e.g., excessive switching). This implies that the 76% win rate in non-timeout games may not be sustainable — if PokéChamp becomes widely used, opponents will learn its weaknesses and develop counter-strategies, driving the win rate down. The paper's recommendation for anonymous evaluation underscores this concern but does not address it empirically.

The "expert-level" characterization is therefore best understood as: PokéChamp's strategic decision-making, when it does not time out, is competitive with human players in the 70th–90th percentile of the Showdown ladder, based on a limited sample of games. This is a meaningful achievement — it demonstrates that zero-shot LLM-augmented minimax can reach the top third of human players in a complex game — but it is not evidence of consistent top-10% performance under sustained, adaptive competition.

Does the ablation architecture (PokéChamp vs. One Step Lookahead vs. PokéLLMon) cleanly isolate the contribution of each component?

The paper implicitly decomposes PokéChamp's performance into three contributions: the LLM's strategic knowledge (present in PokéLLMon but not One Step Lookahead), the minimax search structure (present in PokéChamp and One Step Lookahead but not PokéLLMon), and the LLM-guided pruning, opponent modeling, and evaluation (present only in PokéChamp). The available comparisons provide partial support:

One Step Lookahead (44% vs. Abyssal) represents the contribution of a simple admissible heuristic with no LLM and no deep search. Adding the LLM as a direct action selector (PokéLLMon) reduces performance to 40%, suggesting that raw LLM strategic reasoning is less reliable than a greedy damage-maximization heuristic for move selection.
Adding structured minimax search with LLM components (PokéChamp, 84%) yields a 44-percentage-point improvement over the heuristic baseline. This is the combined effect of all three LLM modules plus the search structure, and the paper cannot separate their individual contributions.

The key missing ablation is a PokéChamp variant that uses the minimax search structure but with non-LLM heuristics for pruning, opponent modeling, and evaluation — for example, using the one-step lookahead for action pruning, a simple type-effectiveness model for opponent action prediction, and a hand-crafted evaluation function based on HP ratios and remaining Pokémon counts. This would quantify how much of the 40-point gap between PokéChamp (84%) and the heuristic baseline (44%) comes from the search structure alone and how much comes from the LLM's domain knowledge. Without this ablation, it is possible that a well-engineered non-LLM minimax agent — with more sophisticated heuristics than the greedy one-step lookahead — would close much of the gap, meaning the LLM's contribution is primarily in replacing engineering effort, not in providing capabilities that are otherwise inaccessible.

What experiments would strengthen the paper's claims?

Several experiments are conspicuously absent and would significantly strengthen the paper:

Same-model comparison (PokéChamp vs. PokéLLMon, both using Llama 3.1:8b). This would isolate the architectural contribution from the model capability, quantifying exactly how much the minimax framework adds over self-consistency prompting for a fixed LLM.
Search depth ablation (K = 1, 2, 3, 4). The paper does not report how performance varies with search depth. If deepening the search from 1 to 2 plies provides large gains but 2 to 3 provides minimal gains, that would characterize the value of lookahead versus the accuracy of the LLM value function. If deeper search is constrained primarily by latency, this would motivate investment in faster inference infrastructure.
LLM component ablation (remove opponent modeling, remove value function, remove action sampling individually). The paper describes three LLM modules but evaluates them only in combination. A component ablation would reveal which modules contribute most to performance and where effort should be focused for improvement. For example, does the opponent modeling module (which has only 15% Top-1 accuracy in Table 1) actually improve the minimax search outcomes, or does the min-operator's conservatism make even inaccurate opponent models sufficient?
Latency profiling. The paper reports that approximately one-third of online games are lost to timeout, but does not break down where the time is spent: LLM API calls, world model simulation, tree search backup, or network latency. This information is crucial for diagnosing the timeout problem and identifying the most impactful optimization targets.
Statistical significance testing. No confidence intervals, standard errors, or hypothesis tests are reported for any win rate or Elo comparisons. With 25–100 games per matchup, some of the smaller differences (e.g., PokéChamp Llama 3.1 vs. PokéLLMon GPT-4o at 64% vs. 56% in Table 4) may not be statistically significant. Reporting uncertainty would help readers assess whether the observed rankings are reliable or could be due to sampling noise.
Broader LLM baseline comparisons. The related work section cites several LLM-planning frameworks — Tree of Thoughts, ReAct, RAP — but none are implemented as baselines. Comparing against a ReAct-style agent (interleaved reasoning and action) or a Tree-of-Thoughts agent (exploring multiple reasoning paths with LLM-based evaluation) would position PokéChamp within the broader LLM-planning landscape rather than against a single self-consistency baseline.
Evaluation against non-LLM minimax with hand-crafted heuristics. A minimax agent that uses the one-step lookahead for action pruning, a simple type-effectiveness table for opponent modeling, and a weighted sum of HP ratios and Pokémon counts for evaluation would be a strong ablation baseline. If this agent achieved, say, 70% against Abyssal (compared to PokéChamp's 84%), the LLM's incremental contribution would be ~14 percentage points — much smaller than the 40-point gap over the pure heuristic, and a more honest assessment of what the LLM specifically provides.

Do the puzzle benchmarks (1v1, Terastallization, Dynamax) support the claim that PokéChamp can handle generation-specific mechanics and complex game states?

The 1v1 puzzle result (86% for PokéChamp vs. 76% for PokéLLMon) is a clean, quantitative comparison on a well-defined subtask, and the 10-percentage-point advantage supports the claim that PokéChamp's move selection is superior in constrained settings. However, the Terastallization and Dynamax puzzles are qualitative examples (Figure 6) without aggregate statistics — they demonstrate that PokéChamp can use these mechanics correctly in some instances, not that it reliably does so. Without a quantitative solve rate on a larger set of Terastallization/Dynamax puzzles, the claim that PokéChamp "understands and uses these generation-specific game mechanics effectively" (Section 5.2) is supported only anecdotally. The stall and excessive switching puzzles (Appendix C) are framed as failure cases, which is valuable for diagnosis but underscores that the puzzle evaluation is not systematic — it highlights successes and failures post-hoc rather than providing a benchmark that future agents can be scored against.

Does the 3-million-game dataset represent a significant contribution beyond enabling PokéChamp's opponent modeling?

The dataset is described as "the largest real-player Pokémon battle dataset" and a contribution in its own right (Section 1, Section 5). It is used to estimate opponent stats and provide historical usage data for the approximate world model. However, the paper does not demonstrate that the dataset's scale is necessary — would a smaller dataset (e.g., 100k games) produce similar stat estimates and opponent model accuracy? No ablation on dataset size is performed, so the marginal value of 3 million games over a more modest collection is unquantified. The dataset's value as a research resource for the community depends on whether it is publicly released with documentation and standardized splits for benchmarking — the paper indicates it will be available but does not describe the release format, licensing, or whether it includes standardized train/validation/test splits for action prediction and other tasks.

6. Limitations and Trade-offs

The Difficulty and Opponent Estimation Pre-Computation Is Not Accounted for in Performance Metrics

The assumption or constraint. PokéChamp's three LLM modules — action sampling, opponent modeling, and value function evaluation — all depend on the approximate world model, which itself requires estimating the opponent's hidden stats (A, D in the damage formula) from historical Showdown usage data and LLM-based inference from observable behavior. The paper states that "to infer hidden information and approximate latent state, we leverage the statistical data from Pokemon showdown, including Pokemon move pools, EV spreads, item usage, etc., aggregated from gameplay over a given period" (Section 4.1). This statistical pre-computation — and the LLM calls used to refine stat estimates from turn-order observations — consumes computational resources that are entirely unaccounted for in the headline performance metrics (win rates, Elo ratings). The paper provides no measurement of the latency or compute cost of this pre-computation relative to the 15-second turn budget.

The consequence. In the online ladder evaluation, approximately one-third of games were lost due to timeout (Section 5.3). The paper attributes these losses generically to "exceeding the turn time limit," but does not break down how much time is consumed by the approximate world model versus the LLM API calls versus the minimax tree search. If stat estimation and one-step lookahead damage calculations consume a non-trivial fraction of the 15-second turn budget, then the current architecture cannot accommodate more sophisticated opponent modeling or deeper search even with faster LLM inference — the pre-computation itself is a bottleneck. Additionally, the quality of the stat estimates depends on the freshness of the historical usage data and the accuracy of the LLM's stat inference from behavior. If the metagame shifts (as the PokéChamp-Fast experiment in Section 5.3 demonstrates is a real concern), the historical data may become stale, degrading the accuracy of the one-step lookahead and the opponent model without any warning signal. The paper does not characterize how sensitive the downstream win rate is to errors in the stat estimates — if a 10% error in estimating the opponent's Speed stat causes the turns-to-KO heuristic to be wrong, does the minimax search's evaluation compensate, or does the agent make actively harmful decisions based on incorrect admissible heuristics?

What evidence exists in the paper. The paper provides no latency profiling, no ablation on the accuracy of stat estimates, and no sensitivity analysis of win rate to errors in the approximate world model. The one-third timeout rate is a clear indicator that end-to-end latency — including all pre-computation, LLM calls, and search — is a binding constraint, but the lack of decomposition makes it impossible to diagnose which component is the bottleneck. The PokéChamp-Fast experiment (Section 5.3) shows that skipping the minimax search and falling back to the one-step lookahead reduces performance, suggesting that the one-step lookahead alone is insufficient and the search contributes genuine value. But this experiment does not isolate the cost or accuracy of the pre-computation itself.

Mitigation status. The paper does not attempt to address this limitation. It does not report the cost of the approximate world model, does not propose a faster stat estimation method, and does not suggest caching or amortizing pre-computation across turns (even though stats, once estimated, could potentially be reused for multiple turns if no new information is revealed). This is an unacknowledged practical barrier to deployment: a practitioner attempting to replicate PokéChamp would discover that a significant fraction of the 15-second turn budget is consumed before the first LLM call is even made, and might find that the system cannot reliably complete a minimax search of any depth within the time limit.

The assumption or constraint. PokéChamp builds a fresh minimax tree from scratch on each turn, with no persistence of plan, search tree, or evaluation across turns. The LLM is prompted anew each turn with the current state and history; there is no mechanism to continue refining a planned sequence of actions from the previous turn, to learn from the outcome of the previous minimax search (e.g., "the opponent did not play the worst-case action I predicted, so my opponent model may be miscalibrated"), or to commit to a multi-turn strategy that plays out over several turns. The depth-K cutoff means the value function must evaluate positions that are K plies deep, but many Pokémon strategies — particularly defensive stall tactics and setup sweeps — play out over 5–15 turns and require evaluating cumulative effects (entry hazard damage, status chip damage, weather attrition, PP stalling) that are not captured in a 2–4 ply snapshot. The paper explicitly documents this failure in Appendix C.1: "PokéChamp struggles with stall strategies due to the uncertainty they introduce in the current matchup. This uncertainty often causes PokéChamp to switch its current Pokémon frequently, which can be counter-productive."

The consequence. The agent exhibits a specific, exploitable behavioral pattern: it overvalues immediate tactical advantage (dealing damage now, avoiding a bad type matchup now) and undervalues long-term positional advantage (maintaining a favorable matchup even if it means taking chip damage, preserving a key Pokémon for later, accumulating entry hazard pressure). The excessive switching failure documented in Appendix C.2 — where PokéChamp repeatedly uses Focus Blast against Pokémon that resist Fighting, failing to recognize the opponent's switching pattern — is a direct consequence of the per-turn search independence. Each turn, the minimax tree evaluates the current board and selects the action with the best worst-case outcome, but this greedy-per-turn optimization has no mechanism to recognize that the opponent is exploiting the agent's predictability. A human player (or an RL agent with a recurrent policy) would notice "I've used Focus Blast three turns in a row and the opponent keeps switching to a Fighting resist — I should predict the switch and use a different move," but PokéChamp's turn-independent search starts each new observation from scratch and reaches the same conclusion each time.

The one-third timeout rate compounds this problem: if the search were deeper (more plies), it might capture longer-term strategic dynamics, but deeper search is blocked by the latency constraint. This creates a vicious cycle — the agent cannot plan far enough ahead to handle stall and switching strategies, and it cannot allocate more time to planning because it already times out in one-third of games. The paper acknowledges (Section 6) that performance is "currently limited by the accuracy of opponent modeling and the method's online computational budget. By increasing the breadth and depth size of the search, we expect to see further improvement's to the performance of the method," but this aspiration is in direct tension with the already-binding latency constraint.

What evidence exists in the paper. The stall and excessive switching puzzles in Appendix C (Figure 8) provide qualitative evidence of the failure mode. The average game length for PokéChamp (15.7 turns in Gen 9 OU, Table 2) is shorter than for PokéLLMon (22.6 turns) and Abyssal (17.9 turns), which is consistent with an agent that plays aggressively and closes games quickly rather than engaging in prolonged positional battles — but also suggests it may struggle when forced into longer games by defensive opponents. The one-third timeout rate further indicates that the per-turn search already saturates the available time budget, leaving no slack for deeper search that might mitigate this limitation.

Mitigation status. The paper acknowledges the limitation indirectly (Appendix C) but proposes no mechanism to address it within the current architecture. Possible mitigations — such as maintaining a persistent plan across turns (e.g., the LLM generates a 3-turn strategy at the start of a sequence and the agent follows it for several turns before re-planning), using the previous minimax tree as a warm start for the current turn's search (reusing evaluated subtrees when the opponent's action matches a predicted branch), or incorporating a recurrent opponent model that learns from observed patterns during the battle — are not explored. The paper frames the limitation as a consequence of "limited lookahead depth necessitated by time constraints" (Appendix C.1) and leaves it as an open challenge.

The LLM's Frozen Knowledge Conflicts with Real-Time Statistical Data, Causing Performance Degradation Under Metagame Shift

The assumption or constraint. PokéChamp relies on the LLM's pre-existing knowledge of Pokémon strategy, which is frozen at the model's training cutoff date. The paper explicitly identifies this as a limitation in the context of the PokéChamp-Fast experiment (Section 5.3): "the LLM's pre-training data, which has a fixed cutoff date, likely biases its decision-making even when provided with updated historical statistics about the current metagame." The approximate world model provides real-time statistical data (usage percentages for moves, items, EV spreads, and team compositions from recent Showdown games), but the LLM's action proposals and value function evaluations are shaped by its pre-training knowledge, which may reflect an outdated metagame. For example, if a Pokémon was considered weak during the LLM's training period but has become dominant in the current metagame due to a new strategy or team archetype, the LLM may undervalue it in opponent modeling and fail to prepare appropriate counters.

The consequence. The agent's strategic judgment degrades over time as the metagame evolves away from the LLM's training distribution, even when the historical data module is updated with fresh statistics. The PokéChamp-Fast experiment provides direct evidence: six months after the main experiments, a speed-optimized variant achieved an Elo of only 1150–1310, compared to 1300+ for the standard method tested earlier. The paper attributes this degradation to "a covariate shift in the competitive metagame over time." More subtly, the LLM's frozen knowledge may cause confirmation bias in its use of the statistical data — the LLM may discount real-time evidence that contradicts its pre-existing beliefs about which strategies are effective, leading to systematically suboptimal action sampling and opponent modeling in a shifting metagame. This problem is structural, not incidental: any LLM-based agent with a frozen training cutoff will face unavoidable staleness as the environment changes, and the rate of degradation depends on how fast the metagame evolves.

The limitation also affects the value function. The scoring criteria listed in Section 4.1 (positive factors: move effectiveness, remaining Pokémon, win probability; negative factors: excessive switching, speed disadvantage, opponent's remaining Pokémon) are evaluated by the LLM based on its frozen strategic understanding. If the metagame has shifted such that certain Pokémon or strategies are now valued differently — for example, a defensive Pokémon that was considered mediocre during training becomes a meta-defining threat — the value function's assessments of board states involving that Pokémon will be systematically miscalibrated.

What evidence exists in the paper. The PokéChamp-Fast experiment (Section 5.3) is the primary evidence. The standard PokéChamp, which always performs a full minimax search, was tested in September 2024 and achieved an Elo above 1300. PokéChamp-Fast, tested on March 3, 2025, achieved 1150–1310 without convergence. The paper explicitly identifies metagame shift as the likely cause, not architectural differences: "the LLM's inherent prior knowledge can outweigh real-time statistical information, particularly when deciding between strategic exploration (minimax search) and exploitation (one-step lookahead)." This suggests that the full minimax search partially corrects for the LLM's outdated priors by evaluating the consequences of actions using the (up-to-date) world model, rather than relying solely on the LLM's strategic judgment. But the degradation in PokéChamp-Fast reveals that when the LLM's priors are allowed to dominate (as in the "choose whether to search" meta-decision), performance suffers.

Mitigation status. The paper partially mitigates this through the full minimax search, which the PokéChamp-Fast comparison suggests acts as a corrective — the search evaluates outcomes using the approximate world model (which can be updated with fresh statistics), and the LLM's role is limited to proposing candidate actions and evaluating leaf nodes, where its outdated knowledge is tempered by the structured adversarial reasoning. However, the mitigation is incomplete: the LLM's action proposals still determine which branches of the game tree are explored, and if the LLM systematically omits metagame-relevant actions from its candidate set because its training data predates the current meta, the minimax search cannot recover them. The paper does not propose a mechanism for updating the LLM's strategic knowledge post-deployment (e.g., through retrieval-augmented generation from recent high-Elo games, or periodic fine-tuning on new battle data), which would be the most direct mitigation.

The Single Benchmark / Single Model Family Design Limits Confidence in Generalization

The assumption or constraint. All experiments in the paper use a single game environment (Pokémon Showdown, primarily Gen 9 OU with supplementary Gen 8 Random Battles results) and two model families (GPT-4o and Llama 3.1). The paper's claims about the effectiveness of LLM-augmented minimax search — that it achieves "expert-level play," that it "significantly outperforms existing bots and AI agents," that it demonstrates "the potential of integrating LLMs with game-theoretic planning algorithms" — are supported exclusively within this narrow experimental domain. Section 6 gestures toward generality: "we provide a generalized framework of action sampling, one step lookahead world modeling, opponent modeling, and value that may be easily applied to other frameworks," but no evidence of such application is presented.

The consequence. Several aspects of PokéChamp's performance may be specific to Pokémon battles in ways that do not transfer to other partially observable competitive games. First, the LLM's pre-existing knowledge of Pokémon strategy is a crucial enabler — GPT-4o's training data includes substantial Pokémon-related content (strategy guides, forum discussions, competitive analyses) that provides the domain knowledge for action sampling and value function evaluation. In a game where the LLM has less pre-existing knowledge (a novel game, a less popular game, or a game with mechanics not well-documented online), the quality of the action proposals and value function estimates would degrade substantially, and the paper provides no evidence about the lower bound of LLM knowledge required for the framework to function.

Second, the one-step lookahead heuristic is enabled by a well-defined damage formula (Equation 3) that can be computed exactly given stat estimates. Many competitive games — even turn-based ones — do not have such a clean, computable one-step transition model that provides admissible heuristics. In a game with more complex or less mathematically specified transitions (e.g., a card game with hidden information and complex rule interactions), the approximate world model would be harder to build, and the paper provides no guidance on how to construct one.

Third, the specific failure modes identified — struggle against stall strategies (Appendix C.1), excessive switching (Appendix C.2), timeout under tight time controls — may not generalize to games with different temporal dynamics. In a game with longer turn times or no strict time limit, PokéChamp's latency problems would be less binding; in a game without switching mechanics, the excessive switching failure would be irrelevant. But new failure modes would likely emerge in different environments, and the paper's diagnostic methodology (anecdotal puzzle analysis) provides no systematic way to anticipate them.

What evidence exists in the paper. The paper provides zero cross-domain or cross-model-family generalization evidence. All results are on Pokémon Showdown, and while two model families are tested (GPT-4o and Llama 3.1), both are autoregressive transformer LLMs with similar pre-training paradigms. The paper does not test whether the framework works with a model that has less Pokémon knowledge (e.g., a smaller model, a model trained primarily on code, a model with a different pre-training corpus), which would help establish the minimum knowledge threshold required.

Mitigation status. The paper does not claim cross-domain generalization; it presents the work as a demonstration within Pokémon battles and suggests (Section 6) that the framework "may be easily applied to other frameworks." This is an aspiration, not a supported claim. The limitation is acknowledged implicitly by the narrow scope of the evaluation, but the paper's language occasionally overclaims (e.g., "our work opens new avenues for research in competitive multi-agent settings," Section 1) without the cross-domain evidence that would justify such breadth. A practitioner considering applying the framework to a different game — a different competitive Pokémon format with unfamiliar mechanics, or a non-Pokémon game entirely — would need to assume that the LLM possesses sufficient domain knowledge and that a computable one-step heuristic can be constructed, with no empirical guidance from this paper on how to validate those assumptions.

The Opponent Model Has No Mechanism for Handling Adversarial Deception or Off-Meta Play

The assumption or constraint. PokéChamp's opponent modeling module predicts opponent actions using two sources: (1) historical usage statistics from the Showdown dataset (the most common EV spreads, movesets, and items for each Pokémon species in the current metagame), and (2) LLM-based reasoning about what a rational opponent would do given the observable state. Both sources assume the opponent plays standard, meta-conforming strategies. The historical data captures the average or modal opponent; the LLM prompt asks what a rational opponent would do. Neither mechanism accounts for an opponent who is deliberately playing off-meta — using unusual stat distributions, unexpected moves, or non-standard items — with the specific intent of exploiting PokéChamp's reliance on statistical norms. The paper acknowledges a related concern in Section 6: "our work can be taken advantage of adversarially due to static opponent modeling," but does not explore the implications.

The consequence. An opponent who knows (or infers) that PokéChamp uses modal stat estimates and common-moveset assumptions can construct a team that systematically violates these assumptions — for example, using a Pokémon commonly run as a physical attacker but training it for special attacks, or equipping an item that changes the Speed tier in unexpected ways. The approximate world model's damage calculations would be wrong (because the A, D, or Speed stats are mis-estimated), and the one-step lookahead would provide misleading turns-to-KO heuristics. The LLM's opponent action predictions would systematically fail because the opponent's action space includes moves or strategies that the LLM considers "irrational" or "non-standard" and therefore excludes from its candidate set. The minimax search's min-operator, which assumes the opponent chooses the worst action for the agent from among the predicted actions, would not protect against this because the truly worst action (the one exploiting the agent's stat mis-estimation) might not be in the predicted set at all.

The paper's report that human opponents who knew they were playing against PokéChamp "were able to determine the excessive switching limitation and tailor an adversarial strategy" (Section 5.3) demonstrates that human opponents do adapt to exploit the agent's known weaknesses. Stat-based deception is a natural extension: if the Pokémon Showdown community identifies that PokéChamp relies on modal stat estimates, players could begin running off-meta EV spreads specifically to break the agent's damage calculations and force suboptimal decisions. The paper's recommendation for anonymous evaluation acknowledges this vulnerability but does not address it architecturally.

What evidence exists in the paper. Table 1 provides indirect evidence of the opponent model's fragility: Top-1 opponent action prediction accuracy is only 13–16% across all Elo ranges, and even Top-5 accuracy is only 40–53%. While the paper attributes this to partial observability, some fraction of the error likely comes from opponents using non-standard strategies that the LLM does not predict. The action prediction experiments use historical replay data, not adversarially constructed teams, so they do not directly measure the degradation that would occur against a deliberately deceptive opponent. The mirror matchup results (Table 3), where both sides use identical teams, partially control for stat estimation error (the teams are identical, so modal estimates should be accurate), but do not test off-meta scenarios.

Mitigation status. The paper mentions the limitation in passing (Section 6: "our work can be taken advantage of adversarially due to static opponent modeling") and leaves it as an "open challenge for future work." No architectural mitigation is proposed. Potential mitigations — such as maintaining a distribution over opponent stats rather than a point estimate and using the minimax search's min-operator over that distribution, periodically updating the opponent model based on observed behavior during the battle (online inference of the opponent's stats from damage rolls and Speed interactions), or incorporating an explicit "unknown/off-meta" opponent type in the action prediction prompt — are not explored. A practitioner deploying PokéChamp in a competitive setting where opponents may adapt would need to develop these mitigations independently.

7. Implications and Future Directions

How This Work Changes the Landscape

PokéChamp represents a reframing of how LLMs should be integrated into classical game-theoretic algorithms, rather than a paradigm shift in either LLM capabilities or game AI. Its primary conceptual contribution is the demonstration that an LLM can serve as a modular, zero-shot heuristic replacement for three distinct components of minimax tree search — action pruning, opponent modeling, and position evaluation — and that this decomposition is sufficient to achieve expert-level play in a complex partially observable game without any task-specific training. This is not a new algorithm (minimax is decades old) nor a new capability for LLMs (they could already discuss Pokémon strategy). It is a new architecture for combining them that changes the default approach to building game agents.

Before PokéChamp, the landscape for LLM-based game agents was split between two unsatisfying poles. On one side, prompting-only approaches (exemplified by PokéLLMon) treated the LLM as an end-to-end policy: describe the game state in text, ask the model what to do, and perhaps sample multiple times for self-consistency. The paper's results demonstrate that this approach is not merely suboptimal — it can be actively worse than a greedy heuristic. PokéLLMon with GPT-4o achieves a 40% win rate against the Abyssal bot, while the One Step Lookahead bot (a purely heuristic move selector with no LLM and no search) achieves 44% (Table 2). The LLM's strategic knowledge, when applied without structured lookahead, reduces performance. On the other side, RL-from-scratch approaches (AlphaZero, AlphaStar, OpenAI Five) achieve superhuman performance but require millions of games of self-play and do not generalize across environments. PokéChamp occupies a previously empty point in this design space: competitive performance (top 30% of human players) with zero environment-specific training, by using the LLM not as a decision-maker but as a heuristic provider within a classical planning algorithm.

This reframing resolves a tension in the literature that the paper explicitly documents in Section 1.1. Several prior works had shown that LLMs possess extensive game knowledge but fail at structured planning — they cannot play Nash Tic-Tac-Toe (Topsakal & Harper, 2024), and LLM-powered Nethack agents underperform heuristic bots (Jeurissen et al., 2024). Other works had shown that structured search algorithms (Tree of Thoughts, RAP) can improve LLM reasoning on puzzle-like tasks but had not been tested in real-time, partially observable, adversarial environments. PokéChamp synthesizes these threads by showing that the LLM's knowledge and the search algorithm's structure are complementary: the LLM provides the domain-specific heuristics that make search tractable in a $10^{354}$ state space, and the search provides the adversarial lookahead that the LLM lacks. Neither component alone works (PokéLLMon: 40%; One Step Lookahead: 44%); together they achieve 84% (Table 2). This is more than additive — it is a qualitative change in capability that emerges from the architecture, not from either component.

The paper also provides a diagnostic framework that redirects research attention toward specific bottlenecks. The three documented failure modes — timeout (one-third of online games, Section 5.3), depth-limited inability to handle long-horizon strategies (Appendix C.1, C.2), and metagame staleness (PokéChamp-Fast degradation, Section 5.3) — collectively identify computational budget and temporal distribution shift as the primary barriers to further improvement, not the quality of the LLM's strategic knowledge or the accuracy of its position evaluations. This is a non-obvious finding: the natural assumption might be that an LLM's imperfect Pokémon knowledge would be the limiting factor, but the evidence suggests that GPT-4o already knows enough about Pokémon strategy to support expert-level play if given sufficient time to search. The implication for the field is that research investment should shift from improving LLM strategic reasoning (better prompting, more domain-specific fine-tuning) toward reducing inference latency and enabling deeper search within real-time constraints. This is a concrete, actionable redirection: model distillation, speculative decoding, cached prompts, and asynchronous evaluation are likely more impactful for PokéChamp-style agents than better opponent modeling prompts.

Finally, the paper establishes Pokémon battles as a credible benchmark for LLM-planning integration, comparable to how Chess and Go served as benchmarks for classical game AI and how NetHack and Minecraft serve for open-ended RL. The three-axis evaluation framework — offline action prediction (Table 1), synthetic puzzles (Section 5.2, Appendix C), and live adversarial play against humans (Section 5.3) — provides a template for how game AI benchmarks should be constructed in the LLM era, where agents are expected to leverage pre-existing knowledge rather than learn from scratch. The dataset of 3 million battles (500k high-Elo) lowers the barrier to entry for other researchers. This infrastructure contribution, while secondary to the agent architecture, may have longer-term impact by standardizing evaluation and enabling controlled comparisons across different LLM-planning approaches.

Follow-Up Research This Work Enables

Latency-ablation study to isolate the binding constraint on search depth. The paper identifies timeout as the cause of one-third of online losses (Section 5.3) but provides no breakdown of where the 15-second turn budget is spent — LLM API calls, world model simulation, tree search backup, or network latency. A direct follow-up would instrument PokéChamp to log per-component timing across 50+ games (LLM prompt construction, API round-trip, damage calculation, minimax backup), then plot the distribution of time per component to identify the bottleneck. If LLM API latency dominates (e.g., 8–10 seconds per turn for GPT-4o), the experiment would quantify the performance gain from switching to a faster model (GPT-4o-mini, a locally hosted Llama 3.1) at the cost of decision quality. If world model simulation dominates, optimization of the damage calculation (precomputed lookup tables, vectorized batch computation) becomes the priority. The key metric is the depth-K achievable within 10 seconds (leaving buffer for network latency) as a function of optimization target. This study would directly inform whether PokéChamp's ceiling is hardware-limited or algorithm-limited.

Search-depth scaling analysis with offline evaluation. The paper does not report how PokéChamp's win rate varies with search depth K, leaving open the question of whether deeper search provides diminishing returns (K=1 is nearly as good as K=4) or whether the current depth (likely 1–2 full turns) is far from the performance ceiling. A controlled experiment would run PokéChamp against Abyssal and PokéLLMon at K=1, 2, 4, 8 plies, removing the time constraint (allowing as much wall-clock time as needed) to isolate the strategic value of depth from the latency bottleneck. If win rate plateaus at K=2, the value function is accurate enough that deeper search adds no benefit; if it continues improving at K=8, the current performance is severely depth-limited and investment in faster inference is strongly motivated. This experiment also tests whether the LLM value function remains calibrated at deeper nodes (where the state diverges further from the types of positions the LLM has seen in training data) or degrades in accuracy, which would indicate a need for better evaluation functions before deeper search becomes useful.

Component ablation: which LLM module contributes most to the 84% win rate? The paper credits PokéChamp's performance to the combination of LLM-based action sampling, opponent modeling, and value function evaluation, but evaluates them only as a package. A component ablation would replace each module with a non-LLM alternative — action sampling with the top-3 moves from the one-step lookahead, opponent modeling with a simple type-effectiveness table, value function with a hand-crafted linear combination of HP ratios and Pokémon counts — and measure the win rate degradation against Abyssal in Gen 9 OU for each ablation. If replacing opponent modeling causes a small drop (say, 84% → 78%) but replacing the value function causes a large drop (84% → 55%), the value function is the critical LLM contribution and should be the focus of improvement efforts. Conversely, if opponent modeling has minimal impact (consistent with its low Top-1 accuracy of 15% in Table 1), the agent may be robust to poor opponent models, and effort spent improving opponent prediction would be wasted. This experiment also tests the paper's implicit claim that all three modules are necessary — a finding that one module can be replaced with a simple heuristic without performance loss would simplify the architecture and reduce latency.

Adversarial robustness evaluation: can human-designed off-meta teams systematically defeat PokéChamp? Section 6 acknowledges that "our work can be taken advantage of adversarially due to static opponent modeling," and Section 5.3 reports that humans who knew they were playing against PokéChamp exploited its excessive switching weakness. A systematic follow-up would recruit 10–20 experienced Pokémon players, inform them of PokéChamp's architectural assumptions (modal stat estimates, LLM-based opponent modeling, limited search depth), and ask them to design teams and strategies specifically to defeat it. The experiment would measure: (1) the win rate of human-designed adversarial teams versus human-designed standard meta teams against PokéChamp, (2) which specific assumptions are exploited most frequently (stat deception, off-meta movesets, stall tactics, switching patterns), and (3) whether the LLM's action predictions degrade measurably against adversarial teams (replicating the Table 1 analysis on adversarial data). This stress-test would quantify the fragility of the zero-shot approach and establish whether adversarial robustness requires moving beyond frozen LLM knowledge — for example, through online adaptation during the battle or through explicit uncertainty modeling in the opponent module.

Cross-domain transfer: does the LLM-minimax decomposition generalize to a non-Pokémon POMG? The paper claims "we provide a generalized framework ... that may be easily applied to other frameworks" (Section 6) but provides only Pokémon evidence. A strong follow-up would replicate the architecture in a different partially observable competitive game where the LLM has less pre-existing knowledge — for example, a simplified war game, a negotiation game, or a custom-designed game with documentation provided in the prompt rather than assumed from pre-training. The experiment would measure: (1) the performance of the LLM-minimax agent vs. a self-consistency prompting baseline vs. a heuristic bot in the new game, (2) whether providing game rules in the prompt (rather than relying on pre-training knowledge) is sufficient for the action sampling and value function modules, and (3) whether the one-step lookahead heuristic is constructible from first principles when a damage formula is not available. A negative result — the framework fails in a new domain — would bound its generality and suggest that PokéChamp's success depends on the LLM's unusually deep Pokémon knowledge. A positive result would strengthen the case for the framework as a general architecture.

Online opponent model adaptation from observed damage and speed interactions. PokéChamp currently uses a static point estimate of opponent stats (modal values from historical data), which makes it vulnerable to off-meta stat distributions. A natural extension is online Bayesian updating: start with the historical prior over possible EV spreads for the opponent's Pokémon, then update the posterior after each turn based on observed damage rolls (which reveal information about A and D in Equation 3) and Speed interactions (which reveal information about the Speed stat). The updated posterior would be used in the approximate world model's damage calculations for the next turn's search. The experiment would measure: (1) the accuracy of stat estimates after 1, 3, 5 turns of observation versus the static prior, (2) the win rate improvement against opponents using off-meta EV spreads (constructed adversarially), and (3) the computational overhead of Bayesian updating relative to the 15-second turn budget. A positive result would directly address the adversarial robustness vulnerability while maintaining the zero-shot training constraint (Bayesian updating requires no learning, only inference).

Practical Applications and Downstream Use Cases

Competitive game AI for titles with frequent balance patches and new content. The zero-shot, training-free property of PokéChamp's architecture is most valuable in games where the environment changes frequently — new characters, balance adjustments, mechanical reworks — making RL-from-scratch approaches economically infeasible because retraining would be required after every update. Modern competitive games (MOBAs, hero shooters, digital card games) release balance patches every 2–4 weeks, and each new character or expansion shifts the metagame. PokéChamp's architecture could be deployed for in-game AI opponents, automated playtesting, or coaching tools: the LLM provides strategic knowledge from pre-training on community discussions and patch notes, the approximate world model is updated with the new numerical parameters (damage values, cooldowns, stats), and the minimax search handles adversarial planning. The Pokemon Showdown results suggest that even with a low-latency open-source model (Llama 3.1:8b achieved 56% win rate against Abyssal, Table 2), the framework produces competent play, making on-device or low-cost deployment feasible.

Automated playtesting for game balance. Game developers currently rely on human playtesters and heuristic bots to evaluate whether new characters, items, or mechanics are balanced. Human playtesting is expensive and slow; heuristic bots do not explore off-meta strategies that creative human players might discover. A PokéChamp-style agent could be deployed in batches to simulate thousands of games with new content, using the LLM's strategic creativity (its ability to propose non-obvious actions beyond the greedy heuristic) to surface balance issues that heuristic bots would miss. The paper's evidence that the One Step Lookahead bot (purely greedy) achieves only 44% against Abyssal while PokéChamp (LLM-guided) achieves 84% (Table 2) demonstrates the gap between heuristic and strategically-aware play — an automated playtesting system that relied solely on heuristic bots would systematically miss the strategies that human experts would discover. The dataset of 3 million human games provides a baseline for calibrating whether the AI's play patterns match human distributions.

LLM-planning integration benchmark for the research community. The paper's infrastructure — the 3-million-game dataset, the 1v1 puzzle suite, the Gen 9 OU evaluation protocol, and the online ladder connection — can serve as a standardized benchmark for comparing different approaches to combining LLMs with planning algorithms. Researchers developing new LLM-planning integrations (Tree of Thoughts variants, RAP-style world models, ReAct-based interleaving) can evaluate on the same set of opponents (Abyssal, PokéLLMon) and report the same metrics (win rate, Elo, average turns, action prediction accuracy) against a known performance ceiling (PokéChamp with GPT-4o at 84% and Elo 1268). The puzzle benchmarks (1v1, Terastallization, stall, switching) provide unit tests for specific capabilities. This is analogous to how the Arcade Learning Environment standardized evaluation for deep RL — it converts a complex, heterogeneous research area into a common set of metrics and baselines, accelerating progress by enabling direct comparisons. The paper's impact statement urging researchers to "limit use of the online ladder evaluation" and instead "compete AI agents against other AI agents" provides a sustainable evaluation protocol that does not degrade the human player experience.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives, but the experimental results support the following conditional preference:

Prefer zero-shot LLM-augmented minimax (PokéChamp-style) when: the game environment has a computable one-step transition model for admissible heuristics (even if approximate), the LLM possesses non-trivial pre-training knowledge of the game's strategies and mechanics, partial observability requires explicit opponent modeling rather than full-information search, the deployment timeline or cost prohibits environment-specific training (fine-tuning or self-play), and real-time constraints permit at least a small number of LLM API calls per decision. The Gen 9 OU results (84% win rate against the strongest heuristic bot, Elo 1300–1500 on the human ladder) demonstrate that this combination achieves expert-level play with zero training, but the one-third timeout rate (Section 5.3) shows that latency is a binding constraint that must be managed.
Prefer self-consistency prompting (PokéLLMon-style) when: latency is so tight that even a single round of LLM calls followed by search is infeasible (sub-second decision requirements), the game's branching factor is small enough that exhaustive search is possible without LLM pruning, or the LLM's pre-training knowledge is so shallow that its action proposals are no better than random sampling. The paper's results argue against this approach in general — PokéLLMon with GPT-4o underperforms even a greedy heuristic (40% vs. 44% against Abyssal, Table 2) — suggesting that prompting-only LLM agents should be a fallback, not a default.
Prefer RL-from-scratch (AlphaZero-style) when: superhuman performance is the goal and massive training compute is available, the environment permits self-play with fast simulation (thousands of games per second), and the game mechanics do not change over time. The paper explicitly does not compete with this paradigm — it positions itself as a zero-shot complement, not a replacement — and the failure of test-time compute on the hardest problems (Section 5.3: PokéChamp achieves near-zero win rates on specific adversarial puzzles) suggests that for the most challenging strategic scenarios, learned approaches may still be necessary.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Generalist Agents Lack Strategic Planning Capabilities

Why Pokémon Battles Are an Ideal — and Unusually Difficult — Testbed

Prior Approaches and Their Specific Shortcomings

Reinforcement Learning from Scratch (Tabula Rasa)

LLM-Only Approaches (Prompting Without Planning)

The Specific Gap: No Framework Combines LLM Knowledge with Classical Planning

How This Paper Positions Itself Relative to Existing Work

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Mathematical Formalization: Pokémon as a Partially Observable Markov Game

Approximate Game Transition (World Model)

Player Action Sampling (LLM as Search-Space Pruner)

Opponent Modeling (LLM + Historical Data for Adversarial Reasoning)

LLM Value Function (Position Evaluation at Search Depth K)

Full Minimax Tree Search Algorithm

4. Key Insights and Innovations

Innovation 1: LLMs as Modular Heuristic Replacements in Classical Tree Search — Not as End-to-End Decision Makers

Innovation 2: Partial Observability as a First-Class Target for LLM-Based Opponent Modeling — Not an Afterthought

Innovation 3: Verifier (Minimax) Over-Optimization Is Not the Bottleneck — Model Capability and Metagame Shift Are

Innovation 4: Pokémon Battles as a POMG Benchmark with a Real-Player Dataset, Synthetic Puzzles, and Live Ladder Evaluation

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Head-to-Head Bot Comparisons in Gen 9 OU (Full Custom Teams with Terastallization)

Gen 8 Random Battles (Without and With Dynamax)

Action Prediction Against Human Players (Table 1)

Puzzle Benchmarks: 1v1 Battles, Terastallization, Dynamax (Section 5.2)

Online Ladder Against Human Players (Section 5.3)

Ablation Studies and Robustness Checks

Critical Assessment

Does the 76% win rate against PokéLLMon and 84% against Abyssal demonstrate that LLM-augmented minimax search outperforms prior approaches?

Does the live ladder performance (Elo 1300–1500, top 10–30%) demonstrate expert-level play?

Does the ablation architecture (PokéChamp vs. One Step Lookahead vs. PokéLLMon) cleanly isolate the contribution of each component?

What experiments would strengthen the paper's claims?

Do the puzzle benchmarks (1v1, Terastallization, Dynamax) support the claim that PokéChamp can handle generation-specific mechanics and complex game states?

Does the 3-million-game dataset represent a significant contribution beyond enabling PokéChamp's opponent modeling?

6. Limitations and Trade-offs

The Difficulty and Opponent Estimation Pre-Computation Is Not Accounted for in Performance Metrics

The Revision Model Analogue — Iterative Refinement of Plans Within a Turn — Is Absent, Limiting Performance on Long-Horizon Strategies

The LLM's Frozen Knowledge Conflicts with Real-Time Statistical Data, Causing Performance Degradation Under Metagame Shift

The Single Benchmark / Single Model Family Design Limits Confidence in Generalization

The Opponent Model Has No Mechanism for Handling Adversarial Deception or Off-Meta Play

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

Kimi-VL Technical Report

Why Do Multi-Agent LLM Systems Fail?