G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning

ArXiv: 2505.13426

🎯 Pitch

Leading VLMs flunk simple games like 2048 despite flawlessly explaining the rules—this paper shows pure RL training spontaneously makes vision and reasoning co-evolve, boosting Shisen-Sho scores from 1.9 to 17.5 without any supervised perception data.

1. Executive Summary

This paper introduces VLM-Gym, a curated reinforcement learning environment with multiple visual games (2048, Shisen-Sho, Shisen-Sho-Cifar10, Swap) featuring unified interfaces and adjustable compositional difficulty, and uses it to train VLMs via pure RL-driven self-evolution. Training a weak base model (Qwen2.5-VL-7B) with GRPO on this environment produces the G0 models, which spontaneously develop effective perception and reasoning patterns — a bootstrapping phenomenon where localization behaviors in perception (systematically identifying tiles with exact coordinates) and enumeration patterns in reasoning (methodically analyzing game states row by row) mutually reinforce each other without any supervision. The subsequent G1 models incorporate a perception-enhanced cold start (knowledge distillation from Claude-3.7-Sonnet-Thinking using ground-truth perception provided by the programmable environment) prior to RL, achieving the highest scores across all four games and outperforming the teacher model itself, with Shisen-Sho performance improving from a base score of 1.9 to 17.5. The analysis reveals that this co-evolution of perception and reasoning is effective only when the game provides a large action space that prevents incorrect perception–reasoning chains from accidentally receiving positive rewards, establishing that verifiable outcome signals alone can drive mutual improvement of both capabilities without explicit process supervision.

2. Context and Motivation

The Core Problem: VLMs Know What to Do but Can't Execute in Interactive Environments

The paper addresses a specific and counterintuitive failure mode in modern Vision-Language Models: the "knowing-doing" gap. A VLM like Qwen2.5-VL-72B or OpenAI-o1 can describe an image perfectly, answer questions about it accurately, and even explain the rules of a game in detail — yet when placed in that same game as a player making real-time decisions, it performs catastrophically. The authors report that leading VLMs "can struggle profoundly with straightforward games like Shisen-Sho and 2048, sometimes even achieving scores lower than random choice" (Section 1). This is not a failure of knowledge — the model knows what the tiles are and what the rules prescribe — but a failure to translate that static understanding into a sequence of effective actions under the time pressure and state-evolution dynamics of an interactive environment.

This problem matters for several reasons the paper surfaces explicitly and implicitly:

• Autonomous agent deployment: If VLMs are to serve as the perceptual-reasoning core of embodied robots, game-playing agents, or autonomous computer-use systems, they must do more than answer questions about images. They must act on visual inputs in real time, with consequences for wrong actions. A model that can describe a game board but cannot play the game is useless as an agent.
• Broad AI capability evaluation: Games have historically driven progress in AI by providing well-defined, verifiable, progressively difficult tasks (Atari for deep RL, Go and chess for AlphaGo/AlphaZero, StarCraft for strategic reasoning). Extending this tradition to VLMs tests whether the recent advances in multimodal understanding generalize to sequential decision-making — and the paper's evidence suggests they do not, at least out of the box.
• Understanding the relationship between perception and reasoning: The paper identifies perception and reasoning as "two core abilities of VLMs" (Section 1) but observes that prior work treats them as separable and independent — evaluating perception accuracy and reasoning accuracy in isolation. In interactive settings, however, these abilities are deeply coupled: you cannot reason effectively about a game state you perceive incorrectly, and you have no incentive to improve your perception unless doing so leads to better reasoning and better rewards. This coupling is both a challenge and an opportunity that prior work had not systematically explored.

Conflicting Successes: VLMs Excel at Static Tasks, Struggle at Dynamic Ones

The paper is motivated by a stark asymmetry in the literature. On one side, VLMs have achieved remarkable results on static, single-turn multimodal benchmarks: image captioning, visual question answering (VQA), mathematical reasoning from visual inputs, and even medical image interpretation. The authors cite models like GPT-4V, Claude-3.7-Sonnet, Qwen2.5-VL, Gemini, and Kimi-VL as evidence of rapid progress (Section 1, first paragraph). These tasks share a common structure: the model receives an image and a question, produces one answer, and is evaluated for correctness.

On the other side, when the same models are dropped into interactive, multi-turn environments — where each action changes the state, the next observation depends on the previous action, and rewards are sparse or noisy — performance collapses. The paper explicitly references Balrog (Paglieri et al., 2024) and Atari-GPT (Waytowich et al., 2024) as evidence that "VLM performance in such settings often falls considerably short of human capabilities" (Section 1). The authors' own probing confirms this: Qwen2.5VL-72B, a model with strong static benchmark performance, achieves a score of only 2.6 on Shisen-Sho (Table 1) compared to a random baseline of 0.4 — meaning it is barely better than chance, and dramatically worse than the 15.3 achieved by Claude-3.7-Sonnet-Thinking.

This disconnect is theoretically significant because it reveals that the representations and capabilities learned during VLM pretraining do not automatically transfer to the sequential decision-making domain. Something is missing — perhaps the ability to plan under uncertainty, to recover from perceptual errors, to balance exploration and exploitation, or to learn from sparse outcome signals. The paper frames this as the "knowing-doing" gap, borrowing the term from psychology where it describes the disconnect between declarative knowledge (knowing that) and procedural knowledge (knowing how).

Where Existing Approaches Fall Short

The paper identifies several limitations in prior attempts to close this gap:

1. Manual chain-of-thought (CoT) curation doesn't scale. One natural approach is to collect expert demonstrations: have humans (or stronger models) annotate game states with detailed perception–reasoning–action chains, then fine-tune the VLM on these trajectories. The paper references work like PCA-Bench (Chen et al., 2024) and Zhai et al. (2024) which take this approach. The problem, as the authors state, is that "this method faces scalability limitations" (Section 1): you need human experts or strong teacher models for every game, every difficulty level, and every edge case. For a general-purpose VLM agent deployed across many environments, the annotation cost becomes prohibitive.

2. Reinforcement learning for VLMs lacks infrastructure. RL with verifiable rewards (RLVR) has recently driven breakthroughs in LLM reasoning — the authors explicitly cite DeepSeek-R1 and GRPO (Shao et al., 2024) as evidence that reinforcement learning can induce complex reasoning behaviors without supervised data. However, extending this paradigm to VLMs in interactive games faces a practical obstacle: standard RL environments (like OpenAI Gym) are designed for single-action-at-a-time interaction and do not support the parallel action sampling required by group-advantage RL algorithms like GRPO. GRPO needs to sample multiple candidate actions from the same observation, evaluate each against the environment, and compute advantages from the reward distribution — but a standard Gym environment processes one action, advances the state, and cannot rewind to evaluate alternatives. This infrastructure gap has prevented systematic exploration of VLM-RL in games.

3. Existing game benchmarks for VLMs are too narrow or too static. The paper surveys the landscape of VLM-in-games research (Section 5, "Vision-Language Models in Games") and finds that prior work has either: (a) converted game states to text descriptions (e.g., SmartPlay, GTBench), losing the visual richness that makes games challenging; (b) used text-only games (e.g., 24-points in Zhai et al., 2024), avoiding the perception challenge entirely; or (c) benchmarked static screenshots without interactive multi-turn play (e.g., Balrog), which evaluates single-step reasoning but not sequential decision-making. The paper cites Schmied et al. (2025), who "examined LLM failures in text games and RL-based solutions, but this research also remained within text-only confines" (Section 5). No prior work provided a scalable, visually rich, multi-turn, multi-game RL training environment with adjustable difficulty — which is precisely the gap VLM-Gym fills.

4. The relationship between perception and reasoning in RL is unexplored. The paper's related work section on "Reinforcement Learning for Enhancing Vision-Language Models" surveys recent efforts like R1-V (object counting via GRPO), VisualThinker-R1-Zero ("aha moment" emergence in visual reasoning), MMEureka (RLOO on instruction-tuned VLMs), and Vision-R1 / R1-OneVision (multimodal CoT datasets). These works all demonstrate that RL can improve VLM performance on specific visual tasks. However, the authors identify a critical blind spot: "most existing work examines perception and reasoning separately in RL, leaving it unclear how these core VLM abilities might mutually improve" (Section 5). No prior work asks whether improving perception causes better reasoning, whether better reasoning creates pressure for better perception, or whether these dynamics depend on properties of the environment itself (action space size, reward density, perceptual difficulty). The paper positions itself as providing "initial evidence that such a bootstrapping pattern indeed occurs during the RL process" (Section 5).

How This Paper Positions Itself

The paper frames its contribution as bridging three separate research threads that had not previously intersected:

1. VLM game-playing agents (the "what" domain): The need for VLMs that can act effectively in visually rich interactive environments.
2. Reinforcement learning for foundation models (the "how" methodology): The recent success of GRPO and RLVR in inducing emergent reasoning in LLMs, extended here to the multimodal setting.
3. Understanding of VLM capabilities (the "why" analysis): The decomposition of VLM behavior into perception and reasoning components, and the discovery that these components bootstrap each other under appropriate reward structures.

The paper's central innovation is not the RL algorithm itself (GRPO is used off-the-shelf) nor the game selection (2048 and Shisen-Sho are well-known), but rather the infrastructure–method–analysis triad: (a) VLM-Gym as a platform that makes multi-game VLM-RL training practical at scale, (b) the G0 and G1 training recipes that show how pure RL and cold-start SFT interact differently depending on game properties, and (c) the systematic decomposition of training dynamics into perception accuracy and reasoning accuracy, which reveals the bootstrapping phenomenon.

The paper explicitly distinguishes its approach from supervised methods: "G0 models are trained using pure RL-driven self-evolution" (abstract), meaning no human demonstrations, no teacher-generated trajectories, no ground-truth perception labels during training. The success of G0 on Shisen-Sho (improving from 1.9 to 12.8, surpassing both GPT-4o and o1) is presented as evidence that verifiable game rewards alone can drive the acquisition of sophisticated perception and reasoning patterns — an "emergent" outcome analogous to DeepSeek-R1's "aha moment" in math reasoning, but here occurring in a multimodal, interactive context.

The cold-start approach in G1 is positioned not as a replacement for pure RL, but as a solution to specific failure modes identified in G0: the perception prior gap (the base model literally cannot recognize the tiles well enough to get started), inaccurate reward credit assignment (in games like 2048, random actions often succeed, so the reward signal doesn't distinguish good perception from bad), and sparse rewards (in games like Swap, the model rarely receives any positive reward at all during early training, so there's no learning signal). The G1 recipe uses ground-truth perception — which VLM-Gym provides natively because it is a programmable environment — to give the policy a "warm start" via knowledge distillation from a strong teacher (Claude-3.7-Sonnet-Thinking). The teacher receives the ground-truth perception explicitly in its prompt and generates reasoning and action sequences conditioned on that accurate perception. This alleviates the cold-start problem without requiring human annotation: the perception labels come from the game engine itself (e.g., the engine knows which tiles are where), and the reasoning trajectories come from the teacher model, creating a scalable data generation pipeline.

The paper also positions itself as contributing to the broader question of when and why RL works for VLMs. The analysis in Section 4.1 identifies three game-specific properties that govern RL effectiveness: perception prior gap (how far the model's raw visual understanding is from what's needed), reward credit assignment quality (whether the reward signal correctly attributes success to the action that caused it), and reward density (whether the model encounters positive rewards frequently enough to learn). This taxonomy — while not formalized as a theorem — provides a practical diagnostic framework for predicting whether pure RL will succeed on a new game or whether a cold start is necessary. It's a contribution to engineering methodology rather than theory, but one that directly guides practitioners in applying these techniques.

3. Technical Approach

3.1 Reader Orientation

The paper builds a training pipeline — not a single model, but a system of environment, algorithms, and staged training phases — that takes a weak, off-the-shelf Vision-Language Model (Qwen2.5-VL-7B) and progressively turns it into a competent game-playing agent that can perceive game boards, reason about valid moves, and output correctly formatted actions. The problem it solves is the "knowing-doing gap": VLMs can describe game states but cannot play the games effectively. The solution is to use reinforcement learning with verifiable game rewards in a purpose-built, scalable environment (VLM-Gym), optionally preceded by a perception-enhanced cold start where the model is fine-tuned on trajectories distilled from a strong teacher that receives ground-truth perception from the game engine. The key insight is that perception and reasoning abilities are not independent — they bootstrap each other during RL, and the training recipe must account for game-specific properties (perception difficulty, reward density, action space size) that determine whether pure RL can succeed or whether a warm start is needed.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

1. VLM-Gym Environment: A curated, multi-game RL suite (2048, Shisen-Sho, Shisen-Sho-Cifar10, Swap) providing standardized interfaces for observation, action, and reward. It supports parallel game execution, parallel action sampling from a single observation, and configurable compositional difficulty. The environment generates game screenshots (observations), processes model actions, computes rewards, and provides ground-truth perception information (tile positions, identities) since it is a programmable engine.

2. Policy Model (VLM): A Qwen2.5-VL-7B base model that takes a game screenshot and a text prompt as input, and produces structured text output containing a <perception>...</perception> block describing what it sees, a thinking... block with its reasoning, and an <answer>...</answer> block with the action to take. This model is the object of all training.

3. Training Algorithms: Two distinct training stages are used alone or in combination: (a) Supervised Fine-Tuning (SFT) on perception-enhanced teacher trajectories (the G1 cold start), and (b) Group Relative Policy Optimization (GRPO) — the RL algorithm that trains the policy by sampling groups of candidate actions per observation, computing advantages from game rewards, and updating the model to favor actions that outperform their group average.

4. Teacher Model (for G1): Claude-3.7-Sonnet-Thinking, prompted with the ground-truth perception provided by VLM-Gym, generates high-quality reasoning and action trajectories. These trajectories are paired with the ground-truth perception to form the SFT dataset for the G1 cold start.

Information flow during G0 training:

1. VLM-Gym presents a game state as a 640×840 pixel screenshot + a text prompt to the policy model.
2. The policy model generates a group of G = 5 candidate responses (each containing perception, reasoning, and action).
3. For each candidate action, VLM-Gym evaluates it against the game engine (using its parallel action support, rewinding state as needed), computes the game reward (GR), format reward (FR), and — if perception labels are available — perception reward (PR).
4. The rewards for the group are normalized into advantages (mean 0, std 1), and the GRPO objective is used to update the policy model parameters via gradient descent.
5. To ensure diverse state exploration, a random policy is interleaved: a fixed number of random steps are executed in each game to push the environment into varied states before the policy model takes over. This is done offline — the random steps generate the initial observations that the policy sees.

Information flow during G1 cold start (before RL):

1. VLM-Gym generates game states and extracts ground-truth perception (e.g., "Tile at (0,0): Red square, Tile at (0,2): Red square").
2. Claude-3.7-Sonnet-Thinking is prompted with the screenshot and this ground-truth perception in a structured template, and generates thinking and <answer> blocks.
3. The ground-truth perception, teacher's reasoning, and teacher's action are concatenated into a complete training example, and the base VLM is fine-tuned on 1,000 such examples per game via standard SFT.

3.3 Roadmap for the Deep Dive

We will proceed in five stages:

1. VLM-Gym Environment Design: The observation/action/reward interfaces, the parallel execution and action-sampling capabilities, and the game-specific rules that define what the model must learn. Without understanding the environment, the training objectives and emergent behaviors make no sense.

2. The Policy Model's Input-Output Structure: What exactly the VLM sees (the prompt and image), and what it must produce (the structured output with perception, reasoning, and action blocks). This structure is critical because format rewards and perception rewards depend on parsing these blocks.

3. Reinforcement Learning Objective and GRPO: The reward decomposition (Game, Format, Perception rewards), the final reward function, the GRPO advantage computation and policy gradient, and the ϵ-greedy exploration mechanism. This is the engine that drives self-evolution in G0 and post-cold-start improvement in G1.

4. The G1 Cold-Start SFT Pipeline: How teacher trajectories are constructed using ground-truth perception, the prompt template, and the SFT hyperparameters. This is the bridge between pure RL's limitations and the full G1 performance.

5. Implementation and Training Configuration: Concrete hyperparameters, data sizes, training steps, and evaluation protocols that make the results reproducible. Understanding these numbers is essential for assessing the practical cost and scalability of the approach.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that RL with verifiable rewards, when applied in a properly designed interactive environment with parallel action support, can cause perception and reasoning abilities to co-evolve from a weak base model — and that a perception-enhanced cold start can extend this to games where pure RL fails due to reward credit assignment or sparsity problems.

VLM-Gym Environment Design

VLM-Gym is not just a collection of games; it is a reinforcement learning platform designed from the ground up to serve the unique requirements of training foundation-model VLMs with group-advantage algorithms like GRPO. The authors explicitly state that existing RL environments like OpenAI Gym lack three capabilities essential for VLM-RL training: "parallel execution across numerous game states simultaneously," "parallel sampling of multiple actions for any given observation," and "adjustable difficulties across multiple dimensions" (Section 2). Each game in VLM-Gym implements a standardized interface that provides these capabilities.

Observation space. Every game state is rendered as a 640×840 pixel RGB screenshot (Section 3.3, Training Details). This is the raw visual input the VLM receives — there is no symbolic state representation, no pre-extracted features, no text description of the board. The model must learn to extract all relevant information (tile positions, tile identities, numeric values, board structure) directly from pixels. This makes the perception challenge real and non-trivial: the VLM must develop visual grounding abilities specific to each game's visual language (colored shapes in Shisen-Sho, numerals in 2048, CIFAR-10 images in Shisen-Sho-Cifar10).

Action space. The action format is game-specific but standardized across games of similar types:

• 2048: Actions are direction choices from the set {up, down, left, right}, representing the four possible swipe directions. The action space size is 4.
• Shisen-Sho and Shisen-Sho-Cifar10: Actions are coordinate pairs in the format <answer>(x1,y1),(x2,y2)</answer>, specifying two tiles to attempt to match. With an 8×8 grid, the action space size is 64 × 63 / 2 = 2,016 possible unordered pairs, though the model must also recognize which pairs are valid matches.
• Swap: Actions are coordinate pairs in the same format, specifying two adjacent tiles to swap. The action space size is substantial because the model must identify adjacencies and valid swap pairs.

This is a critical design choice with consequences that reverberate through the entire paper. The action space size directly determines whether games are amenable to pure RL (G0). The authors identify that Shisen-Sho's large action space (2016 possible pairs) is a key reason G0 succeeds: "A large action space that prevents incorrect perception and reasoning process from gaining reward is also critical in the RL process" (Section 4.1, Bootstrapping subsection). In contrast, 2048's tiny action space (4 directions) means that even a random policy or a model with completely wrong perception can stumble into correct actions and receive positive rewards, which the authors call the "Inaccurate Reward Credit" problem.

State evolution and multi-turn play. Games proceed as a Markov process: the current state depends on the previous state and the action taken. The paper implements this as a standard environment step function, but crucially, VLM-Gym supports rewinding state to evaluate multiple hypothetical actions from the same observation (the "Parallel Actions" feature). This is non-trivial because physical game engines advance state linearly; VLM-Gym must snapshot and restore state for each candidate action evaluation, a capability the authors identify as "often absent in standard Gym-style environments" (Section 2).

Reward functions. Each game provides a reward signal defined as a function of the action a and state s. The rewards are sparse and binary across all games, taking values in {-1, +1}:

• 2048: Reward(a,s) = +1 if action a leads to at least one tile merge in state s; -1 otherwise. This means the reward does not care about how many tiles merged or their values, only whether a merge occurred.

• Shisen-Sho and Shisen-Sho-Cifar10: Reward(a,s) = +1 if action a leads to a valid tile match (identical tiles connected by a path with at most two 90° turns); -1 otherwise.

• Swap: Reward(a,s) = +1 if action a leads to at least one match-3 disappearance; -1 otherwise.

The unified {-1, +1} reward space is explicitly noted by the authors as important for fair comparison across games during RL training (Section 4.1, "Given the unified reward space of {-1, 1}, these comparisons are fair"). A positive reward means the action was correct, a negative reward means it was invalid or ineffective. This simple reward structure means the model is essentially doing binary classification at each step: predict which actions will succeed. There is no reward shaping, no partial credit, no curriculum.

Compositional difficulty. VLM-Gym allows difficulty to be adjusted along multiple axes simultaneously. The authors illustrate this with Shisen-Sho vs. Shisen-Sho-Cifar10: the game rules are identical, but the perceptual difficulty is raised by replacing simple colored shapes with CIFAR-10 images. This is an example of compositional difficulty — the reasoning challenge is unchanged (same path-finding logic), but the perception challenge is substantially harder because CIFAR-10 images require fine-grained visual discrimination (telling a cat from a dog from a deer) rather than just shape-and-color matching. This design enables controlled experiments: by comparing Shisen-Sho and Shisen-Sho-Cifar10 training dynamics, the authors can isolate the effect of perceptual difficulty on RL learning efficiency.

Ground-truth perception access. Because VLM-Gym is a programmable environment (not a black-box simulator), it natively knows the ground-truth state of every game board: which tiles are where, what their values or identities are, which tiles are adjacent, etc. This information is available to the training pipeline as a privileged oracle — it is not given to the VLM during normal gameplay, but can be used during cold-start data construction (to prompt the teacher model with accurate perception) and during evaluation (to compute perception accuracy metrics). This is a critical enabler for the G1 pipeline: without ground-truth perception from the engine, the cold-start distillation would require human annotators to label tile positions, which would not scale.

Random policy as exploration baseline. To ensure the training policy encounters diverse game states (rather than repeatedly seeing the same initial board), the paper implements an ϵ-greedy-inspired exploration strategy (Section 3.1, "Exploring Diverse and Dependent Game States"). A random policy executes a predetermined number of random steps for each game: "2048: 100 steps, Shisen-Sho: 250 Steps, Shisen-Sho-Cifar10: 250 Steps, Swap: 250 Steps." These random steps push the environment into varied states before the policy model's turn begins. The authors note that while a random policy "serves this purpose, exploration could be further enhanced by implementing a more sophisticated searching algorithm or a stronger model, which we reserve for future investigation."

The Policy Model's Input-Output Structure

The policy model is Qwen2.5-VL-7B, a 7-billion parameter open-source Vision-Language Model. During training and evaluation, it receives two inputs: (1) the game screenshot (640×840 pixels), and (2) a text prompt specific to each game. The prompt describes the game rules, the action format, and the required output structure. The exact prompts are provided in Appendix C.3, but the critical structural element is the enforced output format:

<perception>...</perception>
thinking...
<answer>...</answer>

The perception block is meant to contain the model's description of what it sees on the game board. For Shisen-Sho, the paper shows the model developing a "localization pattern" that systematically enumerates tiles with their coordinates, e.g., "(0, 0): Yellow square, (0, 2): Blue triangle, ..." (shown in Figure 4). For 2048, the perception block should describe the numeric grid. The model is not explicitly told how to format its perception — the G0 model discovers effective formatting patterns on its own during RL. In the G1 cold start, the teacher model's perception block is replaced with the ground-truth perception from VLM-Gym (Section 3.2: "we then used the resulting response concatenated with the groundtruth perception information to finetune the base model"), meaning the G1 base model is fine-tuned to output the engine-provided perception verbatim.

The reasoning block (between thinking and response) is where the model performs its strategic analysis. The paper's Figure 4 shows examples: "Row 0: Yellow square at (0,0), Blue triangle at (0,1)... All tiles have partners except..." — this is the "enumeration pattern" in reasoning, where the model methodically checks each tile against possible matches. This reasoning is generated by the model itself; there is no ground-truth reasoning to supervise against. In G0, reasoning patterns emerge spontaneously during RL. In G1, reasoning trajectories are distilled from Claude-3.7-Sonnet-Thinking.

The answer block contains the action in the game-specific format: a direction for 2048, or coordinate pairs for Shisen-Sho/Shisen-Sho-Cifar10/Swap. The format reward (FR) evaluates whether the model's output parses correctly according to this structure.

Why structure matters. The authors state that the Format Reward is "An auxiliary 0-1 reward that evaluates whether the model's output adheres to the required format" (Section 3.1, Rewards Setting). This reward incentivizes the model to produce parseable, structured output so that: (a) the action can be extracted and executed in the game, (b) the perception block can be compared against ground truth for evaluation purposes, and (c) the structured thinking process is interpretable to researchers analyzing the model's behavior.

Reinforcement Learning Objective and GRPO

The RL training is the core engine of both G0 and the post-cold-start phase of G1. The paper uses Group Relative Policy Optimization (GRPO) following the approach of DeepSeek-R1, and defines a composite reward function that combines game success, format compliance, and optional perception accuracy.

Reward decomposition. The total reward for a model output o given observation q is a weighted sum of three components:

$\text{Final Reward} = \text{GR} + \alpha \cdot \text{FR} + \beta \cdot \text{PR}$

where GR is the Game Reward from VLM-Gym (the {-1, +1} signal defined per game), FR is the Format Reward, PR is the Perception Reward, and α (default 1) and β (default 0) are hyperparameters controlling their relative importance.

What each component computes:

• Game Reward (GR): The verifiable outcome signal from the environment. GR = +1 if the action succeeded (merge/matched/swapped correctly), GR = -1 otherwise. This is the primary learning signal — all improvement in game-playing ability is ultimately driven by GR. The paper emphasizes that this reward is "verifiable" in the RLVR sense: the game engine can compute it deterministically without any learned model or human judgment.

• Format Reward (FR): A binary 0-1 reward that checks whether the model's output text parses into the expected <perception>...</perception> thinking... <answer>...</answer> structure. The paper specifies "PR is 1 only when the model follows the predefined output perception structure and also produces fully correct results; otherwise, the reward is 0" — but note that this quote appears under the PR definition, and the FR definition is simpler: "An auxiliary 0-1 reward that evaluates whether the model's output adheres to the required format." The distinction is that FR checks structural compliance (are the tags present and parsable?), while PR checks content correctness within the perception block.

• Perception Reward (PR): A binary 0-1 reward that checks whether the model's output in the <perception></perception> tags exactly matches the ground-truth perception information provided by VLM-Gym. PR is 1 only when "the model follows the predefined output perception structure and also produces fully correct results; otherwise, the reward is 0" (Section 3.1). This reward compares the content (are the tiles correctly identified?), not just the format.

Why β = 0 by default. The paper sets β = 0 in all main experiments (Section 3.1: "β (default 0)"). This means the Perception Reward is not used during standard G0 and G1 training. The model does not receive any direct reward for perceiving the board correctly; it only receives the game reward for taking correct actions. The authors deliberately omit PR from the standard recipe to test whether game rewards alone can drive perception improvement (the bootstrapping hypothesis). Section 4.3 explores the effect of setting β > 0 in a controlled experiment and finds that while PR accelerates perception accuracy improvement, it does not increase the final game reward — suggesting that the game reward signal alone is sufficient, and that perception improvement is a byproduct of the model's attempts to increase its game reward, not a separately incentivized objective.

GRPO objective. The policy model π_θ is optimized to maximize:

$\mathcal{J}_{GRPO}(\pi_\theta) = \mathbb{E}\left[\{q_s\}_{s=1}^{B} \sim p_{\mathcal{Q}}, \{o_i\}_{i=1}^{G} \sim \pi_{\theta_{old}}(\cdot|q_s)\right] \frac{1}{B}\sum_{s=1}^{B}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \left\{ \min\left[\frac{\pi_\theta^{s,i,t}}{\pi_{\theta_{old}}^{s,i,t}} \hat{A}_{s,i,t}, \text{clip}\left(\frac{\pi_\theta^{s,i,t}}{\pi_{\theta_{old}}^{s,i,t}}, 1-\epsilon, 1+\epsilon\right) \hat{A}_{s,i,t} \right] - \beta \,\mathbb{D}_{KL}\left[\pi_\theta||\pi_{ref}\right] \right\}$

where:

• B is the batch size (number of distinct game states/observations, set to 128).
• G is the group size (number of candidate outputs sampled per observation, set to 5).
• q_s is the s-th observation in the batch, consisting of a game screenshot and its prompt.
• o_i is the i-th sampled output (token sequence) from the old policy π_{θ_old} for observation q_s. Each o_i contains the full <perception>...thinking...<answer>...</answer> text.
• |o_i| is the number of tokens in the i-th output.
• π_θ^{s,i,t} is the new policy's probability of the t-th token in the i-th output for the s-th observation.
• π_{θ_old}^{s,i,t} is the old policy's probability of that same token.
• ε is the clip range, set to 0.2.
• β (confusingly reused — not the same β from the reward function) is the KL penalty coefficient, set to 0.01.
• 𝔻_KL[π_θ||π_ref] is the Kullback-Leibler divergence between the new policy and a reference policy (keeps the model from drifting too far from its starting distribution).

The advantage computation. For each group of G outputs generated from the same observation, the advantage is the reward normalized within the group:

$\hat{A}_{s,i,t} = \frac{R(q_s, o_i) - \text{mean}(\{R(q_s, o_1), \ldots, R(q_s, o_G)\})}{\text{std}(\{R(q_s, o_1), \ldots, R(q_s, o_G)\})}$

where R(q_s, o_i) is the Final Reward for the i-th output on the s-th observation, computed as GR + α·FR + β·PR as defined above.

What this computes in operational terms: For a given game state, the policy generates 5 different responses. Each response gets a final reward. The average and standard deviation of these 5 rewards are computed. Each response's advantage is how many standard deviations its reward is above or below the group mean. A response with an advantage of +1.5 was 1.5 standard deviations better than the average response in its group; a response with advantage -0.8 was below average. The GRPO objective then increases the probability of tokens that appeared in high-advantage responses and decreases the probability of tokens from low-advantage responses, subject to the clipping constraint (don't change probabilities by more than factor 1 ± ε) and the KL penalty (don't deviate too far from the reference policy).

Why this form — the group-relative advantage. The key innovation of GRPO (borrowed from DeepSeekMath and DeepSeek-R1) is that it does not require a learned value function (critic) to estimate advantages. Instead, it uses the empirical reward distribution within the group as a baseline. This is computationally motivated: training a critic is expensive and unstable for large models, and the group-relative normalization automatically adapts to the difficulty of the observation. If all 5 responses to a very hard state produce rewards around -1, their advantages will all be near zero (because the standard deviation is small), meaning the model gets no update for that state — which is correct, because it had no good options to learn from. If one response to an easy state scores +1 while the rest score -1, that response gets a large positive advantage and the model learns to prefer the tokens that led to it.

Why the clipping and KL penalty. The min with clip is the standard PPO-style conservative update: it prevents the policy from changing too much in a single update by capping the probability ratio at 1 ± ε. The KL penalty provides additional regularization against catastrophic forgetting — without it, the model might overfit to recent game experiences and lose the general language and vision capabilities from pretraining. The authors set β_KL = 0.01 and ε = 0.2, following the standard GRPO recipe.

The G1 Cold-Start SFT Pipeline

The G1 training recipe introduces a supervised phase before RL that addresses three failure modes observed in G0: the perception prior gap (base model cannot recognize tiles), inaccurate reward credit assignment (random actions get rewarded in small action spaces), and sparse rewards (too few positive rewards to drive learning). The cold-start pipeline has three stages.

Stage 1: Collect game states and ground-truth perception. Using the same random policy that seeds exploration during RL, the authors collect 1,000 observations per game (Section 3.3, Training Details). For each observation, VLM-Gym provides the ground-truth perception — a structured description of every tile on the board, its position, and its identity (color/shape for Shisen-Sho, number for 2048, CIFAR-10 class for Shisen-Sho-Cifar10, color/shape for Swap). Since VLM-Gym is a programmable engine, this information is available for free — no human annotation needed.

Stage 2: Knowledge distillation from teacher. The ground-truth perception is inserted into a prompt template and sent to Claude-3.7-Sonnet-Thinking (the "teacher" model). The template, as described in Section 3.2 and illustrated in Figure 9, provides: (a) the game prompt with rules and action format, (b) the screenshot, and (c) the ground-truth perception explicitly, e.g., "Here is the accurate perception of the current board: [ground-truth perception text]". The teacher model then generates the thinking and <answer> content — reasoning about the game state given accurate perception, and selecting the action.

What the teacher provides that the environment doesn't. The game engine provides the what (tile positions, identities) and the outcome (reward for an action), but not the why (reasoning connecting perception to action). The teacher model fills this gap: it produces a reasoning chain like "Row 0 has a yellow square at (0,0) and a yellow square at (0,3). These are the only yellow squares. The path between them goes right-right-right with zero turns, which is valid. Therefore, I should select (0,0) and (0,3)." This reasoning is the "missing link" that the base model lacks — the translation from static perception to actionable decision. By distilling this into the base model via SFT, the G1 recipe gives the policy a head start on reasoning patterns that pure RL would need many steps to discover (or might never discover, as in the 2048 and Swap cases).

Stage 3: Supervised fine-tuning on concatenated trajectories. The final training example for SFT is constructed by concatenating: (1) the ground-truth perception text (from the engine) inside <perception>...</perception> tags, (2) the teacher's reasoning (from Claude), and (3) the teacher's action inside <answer>...</answer> tags. The paper specifies: "We then use the resulting response concatenated with the groundtruth perception information to finetune the base model" (Section 3.2). Critically, the perception block in the training data is the engine's ground truth, not the teacher's visual analysis — because the teacher may also make perception errors, and the goal is to give the base model a "perception-enhanced" start with accurate visual grounding. The base model is fine-tuned via standard next-token prediction on 1,000 examples per game, using a learning rate of 2e-5 for 1 epoch.

Why 1,000 examples and 1 epoch. The authors specify these numbers precisely. The small dataset (1,000 examples) and single epoch prevent overfitting: the goal is to give the model a warm start, not to memorize specific game states. After SFT, the model should have learned: (a) the output format (perception → reasoning → action), (b) how to produce accurate perception in the engine's format, and (c) basic reasoning patterns for the game. The subsequent RL phase (500 steps of GRPO) then refines and extends these capabilities through interaction, allowing the model to surpass the teacher on all games (Table 1).

Where the teacher's perception is used vs. replaced. An important subtlety: the teacher model does see the screenshot, so it could potentially use its own visual perception alongside the provided ground-truth. However, the training data is constructed so that the perception block in the SFT example is the engine's ground truth, not the teacher's visual interpretation. This ensures the student learns to output correct perception. The teacher's visual capacity is leveraged only for the reasoning and action selection — an interesting design choice that separates "knowing what's on the board" (provided by the engine) from "knowing what to do about it" (provided by the teacher).

Implementation and Training Configuration

The paper provides specific hyperparameters and infrastructure choices that determine the practical cost and reproducibility of the method.

Model and framework. All training uses Qwen2.5-VL-7B as the base policy model. The GRPO implementation is based on EasyR1 (Zheng et al., 2025), described as "an efficient, scalable, multi-modality RL training framework," and SFT uses LLaMA-Factory (Zheng et al., 2024). The VLM-Gym environment is built on Gymnasium (Towers et al., 2024), a standard RL environment interface.

RL hyperparameters. The paper specifies (Section 3.3, Training Details):

• Learning rate: 1.0e-6 (constant, no schedule mentioned)
• Weight decay: 0.02
• Batch size B: 128 parallel games (128 distinct observations per gradient step)
• Group size G: 5 candidate outputs per observation
• Training steps per game: 500
• GRPO clip range ε: 0.2
• KL penalty coefficient β_KL: 0.01
• Observation resolution: 640×840 pixels

What these hyperparameters imply. With B = 128 observations and G = 5 candidates per observation, each gradient step processes 128 × 5 = 640 total model outputs. Each output is a multi-token text sequence (perception + reasoning + action). For 500 training steps, the total number of game interactions is 640 × 500 = 320,000 — but note that many of these are parallel evaluations from the same observation, not sequential gameplay steps. The actual number of distinct game states seen is 128 × 500 = 64,000. For comparison, a full evaluation run on 2048 is 100 steps × 10 runs = 1,000 steps total, so the training data volume is substantial relative to evaluation.

SFT hyperparameters. The cold-start SFT uses:

• Dataset size: 1,000 examples per game
• Learning rate: 2e-5
• Epochs: 1
• Data collection: same random policy as RL exploration, with step counts of 100 (2048) or 250 (other games)

Reward hyperparameters. The reward weights are α = 1 (Format Reward) and β = 0 (Perception Reward) in all standard experiments. This means the final reward is GR + FR — game success plus format compliance. The absence of perception reward is deliberate, as discussed above: it isolates whether game rewards alone can drive perception improvement.

Evaluation protocol. The paper evaluates models in a multi-turn setting "adhering to the original game's scoring rules" (Section 3.4). Key evaluation parameters:

• 2048: 100 steps per run, 10 independent runs, reporting the average cumulative score over 100 steps × 10 runs. The score per step is "the sum of the values of merged tiles" — so merging two "8" tiles yields a score increment of 16.
• Shisen-Sho and Shisen-Sho-Cifar10: 36 steps per run, 10 independent runs, reporting average cumulative score. Score is equivalent to the game reward (number of successful matches minus failures).
• Swap: 1 step × 100 runs (this is notable because Swap is evaluated on single-step accuracy rather than multi-turn play, suggesting multi-turn Swap evaluation may be infeasible or the game quickly becomes too hard).

Why different evaluation lengths? The authors do not explicitly justify the step counts, but they align with game characteristics: 2048 is a longer game where progress accumulates over many merges; Shisen-Sho is a finite puzzle that can be completed in fewer steps; Swap is evaluated per-step because the base models perform so poorly that multi-turn evaluation would be dominated by repeated failures.

Baseline models for comparison. The paper benchmarks against:

• Qwen2.5-VL-7B (the base model itself, untrained)
• Qwen2.5-VL-72B (a 10× larger open-source model)
• GPT-4o (proprietary, latest OpenAI multimodal model)
• OpenAI-o1 (proprietary, reasoning-augmented model)
• Claude-3.7-Sonnet-Thinking (proprietary, the teacher model for G1)
• Random policy (takes uniformly random valid actions)

All proprietary models are queried through APIs with the same game prompts.

Training is per-game, not multi-game. The paper explicitly states: "Each game was trained independently" (Section 3.3, Training Details). This means there is a separate G0 model for 2048, a separate G0 model for Shisen-Sho, etc. — the models do not share parameters or experiences across games. This is an important limitation: it means the approach requires training from scratch for each new game, and the paper provides no evidence on whether skills transfer between games with similar mechanics (e.g., Shisen-Sho → Shisen-Sho-Cifar10).

Cross-validation and statistical rigor. Unlike the compute-optimal scaling paper (which used two-fold cross-validation within difficulty bins), this paper does not report a formal cross-validation protocol. Evaluation is on a fixed set of 10 random runs per game, and training uses a fixed 500 steps. There are no confidence intervals, standard errors, or significance tests reported. This is a significant methodological difference from the reference example and means the reported scores should be interpreted as point estimates with unknown variance.

4. Key Insights and Innovations

Innovation 1: The Bootstrapping Hypothesis — Perception and Reasoning Co-Evolve Under Outcome-Only Rewards

This is the paper's most intellectually distinctive contribution. The dominant assumption in the VLM field — reinforced by evaluation benchmarks and training pipelines — is that perception and reasoning are separable, independently improvable capabilities. PCA-Bench (Chen et al., 2024) evaluates them as distinct axes; MMEvalPro (Huang et al., 2025) calibrates multimodal benchmarks by isolating perception errors from reasoning errors; and most RL-for-VLMs work (R1-V, Vision-R1, MMEureka) applies RL to improve reasoning while assuming perception is a fixed input pipeline. The G0 and G1 experiments upend this assumption with a specific, mechanistic claim: perception and reasoning are not merely correlated but causally coupled in a positive feedback loop during RL, where improvements in one drive improvements in the other, and this coupling can be triggered by nothing more than a verifiable outcome reward.

The evidence for this claim is not a single ablation but a pattern across multiple experiments. In G0's Shisen-Sho training (Figure 4, Figure 5), the model spontaneously develops localization patterns in perception — systematically enumerating tiles with exact coordinates like "(0,0): Yellow square" — before any improvement in game reward is visible. The paper quantifies this in Figure 5: "A distinct increase in localization patterns appears exclusively in Shisen-Sho and Shisen-Sho-Cifar10 variants," and crucially, "The proliferation of localization patterns consistently precedes improvements in game rewards." This temporal ordering — perception patterns emerging first, game rewards rising later — is the signature of a bootstrapping dynamic, not mere correlation. The model does not get better at perception because it's explicitly rewarded for it (β = 0); it gets better at perception because doing so enables better reasoning, which enables higher game rewards, which reinforces the perception–reasoning chain.

The mechanism is laid out clearly: "Reasoning patterns cannot develop without localization patterns, as they rely on sufficient perception information. Similarly, optimal perception patterns are incentivized only through correct reasoning patterns that lead to actions yielding rewards" (Section 4.1). This is a structural claim about the reward landscape, not just an empirical observation. It says that in environments with large action spaces (like Shisen-Sho's ~2000 possible tile pairs), the reward function naturally discriminates between good and bad perception–reasoning chains: if you perceive the board incorrectly, your reasoning will almost certainly select an invalid action, and you'll receive a negative reward. The game reward thus serves as an implicit perception supervisor — it doesn't tell you which tiles you misidentified, but it reliably tells you that you got something wrong, and the credit assignment problem is tractable because the action space is large enough that random guessing rarely succeeds.

What makes this a fundamental rather than incremental advance is that it reconceptualizes what RL is doing to VLMs. Prior work (DeepSeek-R1, Kimi K1.5) framed RLVR as a way to improve reasoning by rewarding correct final answers. This paper argues that in multimodal interactive settings, RLVR is actually performing joint optimization of a coupled perception–reasoning system — and that the coupling is the reason it works, not a side effect. The paper's speculation that "a dedicated cold-start process to stabilize the RL process may become less critical, or potentially redundant, if the base model possesses sufficient intrinsic strength" (Section 4.3) suggests a future where pure RL can bootstrap both capabilities from scratch, provided the base model crosses some minimum threshold of initial perception ability.

This insight also explains when bootstrapping fails, which is equally important. The G0 model fails on 2048 and Swap for different reasons that both illuminate the bootstrapping hypothesis. In 2048, the action space is tiny (4 directions), so even completely wrong perception–reasoning chains stumble into correct actions and receive positive rewards: "The policy model can produce entirely incorrect perception and reasoning responses while still receiving a positive reward, which biases the learning process" (Section 4.1). In Swap, the reward is too sparse — "the policy model can hardly gain positive reward from the environments as the game is too hard for the base model" (Section 4.1). Both are failures of the reward signal to discriminate between good and bad perception–reasoning chains — in 2048 because it's too permissive, in Swap because it's too absent. This diagnostic framework (perception prior gap, inaccurate reward credit, sparse reward) is a practical contribution in its own right, giving practitioners a checklist for predicting whether pure RL will work on a new game.

Innovation 2: Perception-Enhanced Cold Start as a Scalable Alternative to Human Demonstration

The G1 training recipe — using ground-truth perception from a programmable environment to prompt a teacher model, then distilling the resulting reasoning–action trajectories into a student via SFT — is a methodological innovation in data generation for VLM training. It occupies a previously empty niche in the supervision spectrum between two extremes.

At one extreme is pure RL (G0): no human labels, no teacher trajectories, just environment rewards. This is maximally scalable but fails when the reward signal is too noisy or sparse, as the 2048 and Swap results demonstrate. At the other extreme is manual chain-of-thought curation: human experts annotate game states with detailed perception–reasoning–action chains, as in PCA-Bench and Zhai et al. (2024). This provides rich supervision but faces "scalability limitations" (Section 1) — you need annotators for every game, every difficulty level, and every edge case. The paper explicitly rejects this approach as a general solution.

The G1 cold start occupies a middle ground that exploits a specific property of programmable environments: the game engine already knows the ground-truth state (tile positions, identities, values), but this information is normally hidden from the agent. The paper's insight is that you can surface this information to a strong teacher model (Claude-3.7-Sonnet-Thinking), have the teacher produce reasoning and actions conditioned on accurate perception, and then use the resulting trajectories as SFT data for a weaker student. The teacher doesn't need to perceive the board — it receives perception as text from the engine — so the distillation is decoupled from the teacher's own visual limitations. The teacher contributes strategic reasoning, not visual grounding.

What makes this distinctive is that it inverts the usual relationship between perception and reasoning in training data. In standard supervised approaches, the model must learn perception and reasoning simultaneously from human demonstrations — if the human makes a perception error in their annotation, the model learns that error. In the G1 pipeline, perception is guaranteed correct because it comes from the engine, and reasoning is produced by a model that only has to reason (not perceive) because perception is given. This separation of concerns means the SFT phase teaches the student: "When the board looks like this, here is what the correct perception would be, and here is how you reason from that perception to an action." The subsequent RL phase then refines: "Given the board, can you actually perceive it correctly, and can you reason from your own perception?"

The significance of this innovation extends beyond games. Any domain with a simulated or instrumented environment — robotic simulation, UI automation, programmatic data generation — can potentially provide ground-truth state information that can be surfaced to a teacher model for scalable distillation. The paper demonstrates this for visual games, but the pattern (environment provides state → teacher provides reasoning → student learns both via SFT+RL) generalizes to any setting where a simulator knows more about the world than the agent can observe.

Innovation 3: VLM-Gym as Infrastructure That Enables a New Class of Experiments

This is not a conceptual advance in the traditional sense, but a practical innovation in research infrastructure that the paper convincingly argues did not previously exist and that enables experiments (like the bootstrapping analysis) that were previously infeasible. The paper's Section 2 explicitly enumerates three capabilities missing from standard RL environments that are necessary for VLM-RL training: parallel state execution, parallel action sampling from a single observation, and compositional difficulty. Each capability addresses a specific bottleneck.

Parallel action sampling is the most technically significant of the three because it directly enables the GRPO algorithm. Standard Gym environments process one action, advance the state, and have no built-in mechanism to rewind and evaluate alternatives. To implement GRPO on a standard environment, you would need to either: (a) run multiple copies of the environment in parallel (expensive and synchronization-heavy), (b) save and restore state manually for each candidate action (error-prone and game-specific), or (c) approximate advantages without multiple samples (loses the group-relative normalization that makes GRPO effective). VLM-Gym provides this capability natively, which is why the paper can train with G = 5 candidates per observation at B = 128 observations in parallel — a training throughput that would be impractical without purpose-built infrastructure.

Compositional difficulty enables the controlled experiments that isolate perception from reasoning challenges. The Shisen-Sho vs. Shisen-Sho-Cifar10 comparison is the clearest example: identical game rules, identical action space, identical reward function, but CIFAR-10 images replace simple colored shapes as tiles. This allows the paper to attribute the slower G0 learning curve on Shisen-Sho-Cifar10 (Figure 3) to perceptual difficulty alone, since nothing else changed. Without this capability, you can observe that some games are harder than others but cannot isolate why.

Ground-truth perception access — a consequence of VLM-Gym being programmable rather than black-box — is what makes the G1 cold start scalable and the perception accuracy metric (P_acc) computable. The paper uses this for three distinct purposes: (1) constructing the G1 SFT dataset (providing ground-truth perception to the teacher), (2) computing P_acc during G1 training to track perception improvement (Figure 6), and (3) computing the Perception Reward in the ablation experiment (Section 4.3). None of these would be possible without engine-level access to the true game state.

This innovation is incremental in the sense that each individual capability exists in some form elsewhere — parallel environments in RLlib or Sample Factory, difficulty adjustment in Procgen or Minigrid, state access in any open-source simulator. But the combination of all three in a VLM-compatible interface (producing screenshots, accepting text actions, supporting the GRPO training loop) is novel and, based on the paper's related work survey, fills a genuine gap. The release of VLM-Gym as open source (with the paper's code) makes this a contribution to the research ecosystem, not just to the paper's own results.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use the four games in VLM-Gym — 2048, Shisen-Sho, Shisen-Sho-Cifar10, and Swap — as the evaluation benchmarks. Because VLM-Gym is a programmable environment, the "test set" is not a fixed set of pre-collected states but rather a protocol of multi-turn rollouts from fresh game instances. For 2048, each evaluation run consists of 100 consecutive game steps; for Shisen-Sho and Shisen-Sho-Cifar10, 36 steps; for Swap, single-step evaluations repeated 100 times (Section 3.4). Each model is evaluated over 10 independent runs per game, and the paper reports the average cumulative score across these runs. The games themselves serve as both training and evaluation environments, with no held-out game states — the distinction between train and test comes from the random initialization of each new game instance and the stochastic state transitions (e.g., random tile spawns in 2048, random tile layouts in Shisen-Sho).

• Base model. All training starts from Qwen2.5-VL-7B, a 7-billion parameter open-source Vision-Language Model. The authors choose this model because it is "representative of the capabilities of many contemporary VLMs" (not stated explicitly but implied by its positioning against larger and proprietary models) and because its relatively weak initial game performance (e.g., score of 1.9 on Shisen-Sho, Table 1) leaves substantial room for RL-driven improvement to be measurable. A larger variant, Qwen2.5-VL-72B, is included as a baseline for scale comparison but is not used for training.

• Metrics. The primary metric throughout is cumulative game score averaged over multiple evaluation runs. The score computation is game-specific and reflects the scoring rules of each original game:

  • 2048: Each step's score is "the sum of the values of merged tiles" (Section 3.4). Merging two "4" tiles adds 8 to the score; merging two "1024" tiles adds 2048. The reported metric is the total score accumulated over 100 steps, averaged across 10 runs.
  • Shisen-Sho and Shisen-Sho-Cifar10: The score per step equals the game reward (Section 2.1), which is +1 for a valid match and -1 for an invalid attempt. The cumulative score over 36 steps is reported.
  • Swap: The score per step equals the game reward (+1 for a valid swap creating a match-3, -1 otherwise). Evaluation is single-step × 100 runs, so the reported metric is effectively the success rate (proportion of +1 outcomes).

  For the G1 training analysis (Figure 6), the paper introduces two additional diagnostic metrics tracked during training: Perception Accuracy (P_acc), defined as I(p_model = p_gt) — whether the model's output within <perception> tags exactly matches the VLM-Gym-provided ground-truth perception; and Reasoning Accuracy (R_acc), defined as I(r > 0 | P_acc = 1) — the probability of obtaining a positive game reward given that perception was correct, measuring whether the model reasons correctly when it sees the board accurately (Section 4.2). These metrics are tracked only for G1 models because the base models are fine-tuned to output perception in a format comparable to ground-truth, enabling exact matching.

• Baselines. The paper compares G0 and G1 against five baselines:

  • Random policy: Takes uniformly random valid actions at each step. Serves as a lower bound to establish that the base models, despite their poor performance, are doing more than chance in some cases.
  • Qwen2.5-VL-7B (the untrained base model): Establishes the starting point before any RL or SFT.
  • Qwen2.5-VL-72B: A model from the same family with roughly 10× more parameters, included to test whether scale alone closes the knowing-doing gap.
  • GPT-4o (OpenAI, 2024): A proprietary multimodal model included as a representative state-of-the-art system.
  • OpenAI-o1 (OpenAI, 2024): A proprietary reasoning-augmented multimodal model that uses extended chain-of-thought at inference time.
  • Claude-3.7-Sonnet-Thinking (Anthropic, 2024): The strongest proprietary baseline and also the teacher model used for G1's cold-start distillation.

  All proprietary models are queried through APIs using the same game prompts provided to the trained models. All baselines are evaluated under the identical multi-turn protocol.

• Generation budget / compute accounting. Training compute is measured in RL training steps (500 steps per game for all experiments), with each step processing a batch of B = 128 observations and sampling G = 5 candidate outputs per observation, for a total of 640 model outputs per step and 320,000 outputs over a full training run (Section 3.3). The paper does not convert this to FLOPs or wall-clock time, and does not report the computational cost of SFT (1,000 examples, 1 epoch) in comparable terms. For evaluation, compute is measured indirectly through the number of game steps and runs, but no attempt is made to equalize inference budgets between models — larger models (GPT-4o, o1, Claude) may use more inference compute per response. The GRPO algorithm itself incurs the cost of generating multiple candidates per observation during training (group size 5), which represents a 5× increase in training-time inference compared to single-sample RL algorithms.

• Cross-validation / statistical protocol. The paper does not report formal cross-validation, confidence intervals, standard errors, or significance tests. Evaluation uses a fixed protocol of 10 random seeds per game per model, and the reported scores are point estimates (averages) of these 10 runs. There is no held-out set of game instances used exclusively for evaluation — the game environments are generative (random initial configurations), so test-time performance is evaluated on fresh instances not seen during training. However, the absence of reported variance makes it difficult to assess whether differences between models (e.g., G0-7B at 12.8 vs. Claude at 15.3 on Shisen-Sho) are statistically reliable or within the range of run-to-run variation. All 10 runs for each model presumably use different random seeds, providing some measure of variance, but these are not quantified.

Main Quantitative Results

Game Score Performance: G1 Surpasses All Baselines, G0 Competes with Top Proprietary Models

Table 1 reports the headline results across all four games and all models. The paper organizes the comparison in three tiers:

Tier 1: The G1-7B model achieves the highest scores on every game, including surpassing its teacher model (Claude-3.7-Sonnet-Thinking). The specific numbers from Table 1:

• 2048: G1-7B scores 1070 vs. Claude's 892 (a 20% improvement over the teacher) and G0-7B's 759. The random baseline achieves 701, and the untrained base Qwen2.5-VL-7B scores only 246.
• Shisen-Sho: G1-7B scores 17.5 vs. Claude's 15.3 (a 14% improvement over the teacher) and G0-7B's 12.8. The untrained base scores 1.9.
• Shisen-Sho-Cifar10: G1-7B scores 14.1 vs. Claude's 8.7 (a 62% improvement over the teacher) and G0-7B's 8.0. The untrained base scores 0.4.
• Swap: G1-7B scores 0.78 vs. Claude's 0.43 (an 81% improvement over the teacher) and G0-7B's 0.05. The untrained base scores 0.02.

These results establish the paper's central performance claim: a 7B model, trained with perception-enhanced cold start followed by RL, can consistently outperform its much larger and more capable teacher model, including on games where the teacher itself struggles (Swap, Shisen-Sho-Cifar10).

Tier 2: G0-7B, trained with pure RL from a weak base, surpasses or matches the strongest proprietary models on Shisen-Sho and Shisen-Sho-Cifar10. From Table 1:

• On Shisen-Sho, G0-7B (12.8) outperforms o1 (3.9), GPT-4o (1.5), and Qwen2.5-VL-72B (2.6), and approaches Claude (15.3).
• On Shisen-Sho-Cifar10, G0-7B (8.0) outperforms o1 (5.4), GPT-4o (4.7), and Qwen2.5-VL-72B (0.8), and roughly matches Claude (8.7).
• On 2048, G0-7B (759) surpasses o1 (539) and GPT-4o (620), but trails Claude (892) and G1-7B (1070).
• On Swap, G0-7B (0.05) performs at chance level, substantially below Claude (0.43) and G1-7B (0.78).

Tier 3: Proprietary models and scale alone do not close the knowing-doing gap. The baseline comparisons reveal several noteworthy patterns:

• Qwen2.5-VL-72B, with 10× more parameters than the 7B base, shows minimal improvement on Shisen-Sho (2.6 vs. 1.9) and remains at 0.8 on Shisen-Sho-Cifar10, 0.06 on Swap. Scale alone does not translate static VLM capabilities to interactive gameplay.
• OpenAI-o1, despite its reasoning-augmented architecture, scores only 3.9 on Shisen-Sho and 5.4 on Shisen-Sho-Cifar10 — better than the untuned base models but far below G0-7B after RL training. This suggests that o1's extended chain-of-thought, while effective for static reasoning tasks, does not compensate for the perception–action coupling required in interactive environments.
• GPT-4o performs inconsistently: competitive on 2048 (620) but near-random on Shisen-Sho (1.5) and Swap (0.01). This pattern suggests GPT-4o's visual perception may be adequate for the large-numeral 2048 grid but inadequate for the fine-grained shape discrimination required by Shisen-Sho, consistent with the paper's emphasis on perceptual difficulty as a key variable.
• The random baseline on 2048 (701) outperforms Qwen2.5-VL-7B (246), Qwen2.5-VL-72B (243), and even some proprietary models. This counterintuitive result highlights the "Inaccurate Reward Credit" problem the paper identifies: with only 4 possible actions, a random policy that occasionally stumbles into merges can accumulate substantial score, while a model that tries to reason but does so incorrectly may select systematically wrong moves and perform worse than chance.

RL Training Dynamics: G0 Shows Emergent Perception–Reasoning Patterns, but Only on Certain Games

Figure 3 displays the average game reward curves during the 500-step G0 RL training process for all four games. The curves reveal dramatically different learning trajectories:

• Shisen-Sho (top-left panel): Average reward rises efficiently from approximately -1.0 (near-step 0) to +0.8 by step 400, with steady monotonic improvement throughout training. This is the gold-standard case for pure RL. The final evaluation score (12.8, Table 1) represents a 6.7× improvement over the base model (1.9).

• Shisen-Sho-Cifar10 (top-right panel): Average reward also rises but more slowly and to a lower asymptote than Shisen-Sho, reaching approximately +0.3 to +0.4 by step 400. The slower learning curve is attributed to the "Perception Prior Gap" — CIFAR-10 images are harder to discriminate than colored shapes, so the model takes longer to develop accurate localization patterns. Despite the slower training, the final evaluation score (8.0) is a dramatic improvement over the base model (0.4), confirming that RL can work even with high perceptual difficulty, just less efficiently.

• 2048 (bottom-left panel): The reward curve shows minimal improvement and high variance, oscillating around -0.6 to -0.4 throughout training with no clear upward trend. The final evaluation score (759) represents improvement over the base (246) but is driven more by the base model initially being worse than random on this task than by meaningful RL-driven skill acquisition. As discussed in the case study (Figure 10 in Appendix B), the model "crashed, completely ignored the screenshot image, and took random actions" — the RL process learned to discard perception rather than improve it, because random actions are rewarded often enough under the 4-action space that the model's flawed perception–reasoning chains are reinforced.

• Swap (bottom-right panel): The reward curve remains flat near -1.0 throughout training, with essentially no improvement. The final evaluation score (0.05) is indistinguishable from the base model (0.02) and the random baseline (0.01). The paper attributes this to the "Sparse Reward" problem: the base model "can hardly gain positive reward from the environments as the game is too hard."

Emergence of Localization Patterns Precedes Game Reward Improvement

Figure 4 presents qualitative evidence of emergent perception and reasoning patterns during G0's Shisen-Sho training. Before RL (left panel), the model's perception output consists of "primarily vague board descriptions without precise coordinate information for shapes" — descriptions like "the board has various colored shapes" rather than structured tile enumeration. After RL (right panel), the model develops two distinct patterns: a localization pattern in perception (systematically listing tiles with exact coordinates, e.g., "(0,0): Yellow square, (0,2): Blue triangle..."), and an enumeration pattern in reasoning (methodically analyzing the game state row by row to identify matching pairs).

Figure 5 quantifies this emergence by tracking the average number of localization patterns per model output at each RL training step, parsed via regex. The key findings stated in the paper (Section 4.1):

"1) A distinct increase in localization patterns appears exclusively in Shisen-Sho and Shisen-Sho-Cifar10 variants (notably, 2048 was excluded from plotting due to the complete absence of such patterns throughout all steps), and 2) The proliferation of localization patterns consistently precedes improvements in game rewards, suggesting this adaptation serves as a precursor to enhanced performance."

The temporal ordering — perception patterns emerging before game reward improvement — is the key mechanistic evidence for the bootstrapping hypothesis. It suggests the causal direction: the model first learns to perceive accurately, which then enables better reasoning, which then yields higher rewards. The perception improvement is not itself rewarded (β = 0 throughout G0 training) but is an instrumental sub-goal that the RL process discovers because it unlocks the ability to distinguish correct from incorrect actions in the large action space of Shisen-Sho.

G1 Training: Cold Start + RL Outperforms Either Alone

Figure 6 shows the G1 training curves for all four games, tracking three metrics: game reward (the standard RL objective), perception accuracy (P_acc), and reasoning accuracy (R_acc). The curves reveal several patterns:

The cold-start SFT provides a substantial initial boost but RL is necessary to reach peak performance. Across all games, the starting point after SFT (step 0 on the x-axis of Figure 6) is already well above the random baseline and the base model's starting point in G0 (compare with the step-0 positions in Figure 3). For Swap, the post-SFT game reward starts at approximately -0.8, compared to G0's flat -1.0 throughout training. The subsequent RL phase then improves performance further: on Swap, game reward rises from -0.8 to approximately +0.6 by step 500; on 2048, 2048 shows substantial improvement during RL after the cold start (Figure 7, bottom-left), in contrast to G0's flat curve.

Perception accuracy remains high and stable for perceptually simple games, but co-evolves with game reward for the perceptually challenging Shisen-Sho-Cifar10. From Figure 6:

• 2048 (top-left): Perception accuracy is near 1.0 throughout RL training. The perceptual task (reading large numerals on a 4×4 grid) is trivially solved by the SFT phase.
• Shisen-Sho (top-right): Perception accuracy starts above 0.9 and remains high, with slight fluctuations.
• Swap (bottom-right): Perception accuracy is near 1.0 throughout. Like 2048, the perceptual complexity is low.
• Shisen-Sho-Cifar10 (bottom-left): Perception accuracy starts lower (around 0.5–0.6 after SFT) and improves during RL alongside game reward, rising to approximately 0.8 by step 500. This is direct evidence of the bootstrapping dynamic in G1: even with a perception-enhanced cold start, the model's perception of CIFAR-10 images is imperfect, and the RL process — driven only by game rewards — further refines perception accuracy. This mirrors the G0 finding but in a setting where the model starts from a stronger baseline.

Reasoning accuracy (conditioned on correct perception) improves during RL across all games. The R_acc curves in Figure 6 (third row of each subplot) show consistent upward trends during RL for all four games, even for 2048 where G0's pure RL failed. This demonstrates that the cold start's primary contribution is in providing a perception and reasoning foundation from which RL can effectively improve reasoning — the bottleneck in G0 for 2048 was that poor perception prevented any reasoning improvement, but with perception fixed by SFT, RL can now optimize the decision-making component.

G1 vs. G0 Comparison: Cold Start Benefits Are Largest Where Pure RL Fails

Figure 7 directly compares the game reward curves of G0 and G1 during RL training for each game:

• 2048 (bottom-left): This is where G1 shows its most dramatic advantage over G0. G0's reward curve is flat and noisy near -0.5 throughout training. G1's reward curve (starting after SFT) is initially slightly above G0 (around -0.2) and steadily rises to approximately +0.6 by step 500. The gap between the two curves widens monotonically. This directly validates the paper's claim that the cold start addresses the "Inaccurate Reward Credit" problem: by providing accurate perception from the start, G1's RL phase receives a clean credit assignment signal (good reasoning leads to good actions, bad reasoning leads to bad actions), whereas G0's RL phase is contaminated by false positives (incorrect perception–reasoning chains that accidentally succeed).

• Swap (bottom-right): Similar pattern — G0 is flat at -1.0, G1 starts at approximately -0.8 after SFT and rises to +0.6. The cold start transforms Swap from an impossible RL problem to a tractable one by providing enough prior knowledge that the model occasionally receives positive rewards, breaking the sparse-reward barrier.

• Shisen-Sho (top-left): The trajectories are qualitatively similar — both G0 and G1 (post-SFT) show steady improvement during RL, with G1 starting from a higher baseline and maintaining a consistent advantage throughout. By step 500, the two curves appear to converge to similar final values, consistent with the paper's observation that "G0 and G1 finally reach the same game reward during RL training" (Section 4.3) for Shisen-Sho. This suggests that on games where pure RL is feasible, the cold start accelerates learning but does not change the asymptotic performance ceiling — the environment's reward structure, not the initialization, determines the maximum achievable performance.

• Shisen-Sho-Cifar10 (top-right): G1 starts with a significant advantage over G0 and maintains or slightly increases this gap through RL. G0's curve rises slowly from near -1.0; G1's curve starts higher and rises to a higher asymptote. The perception-enhanced cold start provides both a head start and, apparently, a higher ceiling on this perceptually challenging game — unlike Shisen-Sho, G0 does not catch up to G1 by step 500.

FLOPs-Matched or Scale Comparison: 7B with RL Outperforms 72B Without

While the paper does not perform a formal FLOPs-matched analysis (as the compute-optimal reference paper does), Table 1 provides an implicit scale comparison that supports one of the paper's key implications: RL training can enable a smaller model to substantially outperform a much larger untrained model. Specifically:

• Qwen2.5-VL-7B (base) vs. Qwen2.5-VL-72B (base): The 10× larger model shows only marginal improvements — 246 vs. 243 on 2048, 1.9 vs. 2.6 on Shisen-Sho, 0.4 vs. 0.8 on Shisen-Sho-Cifar10, 0.02 vs. 0.06 on Swap. Scale alone does not meaningfully close the knowing-doing gap.
• G0-7B vs. Qwen2.5-VL-72B: The RL-trained 7B model dramatically outperforms the untrained 72B model on Shisen-Sho (12.8 vs. 2.6, a 3.9× improvement) and Shisen-Sho-Cifar10 (8.0 vs. 0.8, a 10× improvement), and is competitive on 2048 (759 vs. 243) and Swap (0.05 vs. 0.06). This is approximately a 4–10× effective compute multiplier, though the paper does not quantify it in FLOPs terms.
• G1-7B vs. Claude-3.7-Sonnet-Thinking: The 7B model, after cold start + RL, outperforms the proprietary teacher model on all four games, with margins ranging from modest (1070 vs. 892 on 2048) to dramatic (14.1 vs. 8.7 on Shisen-Sho-Cifar10, 0.78 vs. 0.43 on Swap).

These comparisons are not FLOPs-controlled — the Claude model's parameter count and inference cost are unknown, and the training cost of G1 is not amortized over evaluation runs — so the efficiency claims are implicit rather than rigorously quantified. However, the direction of the results is consistent: post-training with environment interaction provides capabilities that pretraining scale alone does not confer.

Ablation Studies and Robustness Checks

Perception Reward as auxiliary process reward (Section 4.3, Figure 8): The paper tests whether adding an explicit perception reward (β > 0 in Equation 4) accelerates or improves RL training on Shisen-Sho. To isolate the effect, the authors first fine-tune only the visual encoder of the base model on 50 distilled examples containing ground-truth perception alone (no reasoning or action content), ensuring the model can output perception in the ground-truth format without being taught game strategy. They then run RL with and without PR. The result (Figure 8): adding PR "significantly accelerates improvement in perception accuracy" (left panel) but "does not impact the overall game reward" (right panel). The game reward curves with and without PR are nearly identical. The paper interprets this as evidence that perception improvement is a natural byproduct of optimizing the game reward — the model learns to perceive better because it helps win, not because it's separately rewarded. This is an important negative result for the auxiliary reward hypothesis: explicit perception supervision during RL does not improve final performance, even though it changes the learning trajectory.

G0 vs. G1 across games with different reward properties (Figure 7): This is not a controlled ablation in the traditional sense, but serves as an ablation over game properties. By comparing the same training recipes (pure RL vs. cold-start + RL) across four games with systematically different characteristics, the paper ablates the necessity of cold start as a function of game properties. The finding: cold start is essential when the game has a small action space (2048) or sparse rewards (Swap), but less critical when the game has a large action space that naturally discriminates good from bad perception (Shisen-Sho). This is a robustness check on the generality of the pure RL approach: it works on the right kind of game, but not all games.

Perception accuracy tracking during G1 RL (Figure 6): For Shisen-Sho-Cifar10, perception accuracy is observed to co-evolve with game reward during RL, even without explicit perception reward (β = 0 in standard training). This serves as a within-experiment replication of the bootstrapping hypothesis: it shows the same perception–reasoning co-evolution observed in G0's Shisen-Sho training, now occurring in a different game (Shisen-Sho-Cifar10) and a different training regime (post-SFT RL rather than pure RL from scratch). The cross-method replication strengthens the claim that bootstrapping is a general phenomenon, not an artifact of a specific training configuration.

G0 2048 case study (Appendix B, Figure 10): The paper includes a qualitative analysis of why G0 fails on 2048, showing rollouts before and after RL. Before RL, the model produces incorrect perception (e.g., misidentifying tile values) and flawed reasoning, yet sometimes receives positive rewards because the action space is small enough that random directions occasionally cause merges. After RL, the model's outputs become completely decoupled from the visual input — "the model crashed, completely ignored the screenshot image, and took random actions" (Section 4.1). This negative result validates the Inaccurate Reward Credit hypothesis: when wrong perception–reasoning chains are reinforced by accidental successes, the policy degenerates rather than improves. It also provides an existence proof that "more RL training" does not always help — the quality of the reward signal matters more than the quantity of training.

Teacher model perception vs. engine ground-truth perception in G1 cold start: An implicit ablation in the G1 data construction: the teacher model (Claude-3.7-Sonnet-Thinking) does see the screenshot and could potentially use its own visual perception instead of the provided ground-truth. However, the SFT data uses the engine's ground-truth perception in the <perception> tags, not the teacher's visual interpretation. This design choice — providing the teacher with ground-truth perception as text while still showing the screenshot — is an implicit ablation of "teacher visual perception quality" vs. "engine oracle perception quality." The paper does not report results using teacher-generated perception in the SFT data, so we cannot quantify how much this choice matters, but it represents a deliberate separation of perception (from the engine) from reasoning (from the teacher) that is central to the G1 recipe.

Critical Assessment

Does the paper demonstrate that "G0 models demonstrate emergent perception and reasoning patterns" (abstract claim)?

Yes, but only on a subset of games. The evidence for emergent patterns is strongest on Shisen-Sho, where Figure 4 qualitatively shows the transition from vague descriptions to systematic coordinate-based perception, and Figure 5 quantitatively tracks the increasing frequency of localization patterns during training. The evidence on Shisen-Sho-Cifar10 is weaker — Figure 5 shows increasing localization patterns, but the paper does not provide the qualitative before-and-after comparison it does for Shisen-Sho. For 2048, the paper explicitly states there is a "complete absence of such patterns throughout all steps." For Swap, the training curve is flat, implying no patterns emerged. The claim is therefore true but conditional: emergence occurs on games with large action spaces that naturally penalize incorrect perception–reasoning chains. This conditionality is acknowledged in the paper but is somewhat buried in Section 4.1 under the Reward Dynamics discussion; the abstract's broad statement ("G0 models... demonstrate emergent perception and reasoning patterns") does not communicate this qualification, which could lead to overgeneralization by readers who do not reach Section 4.1.

Does the paper demonstrate that "perception and reasoning abilities mutually bootstrap each other" (core analytical claim)?

The paper provides correlational and temporal evidence, not causal proof. The strongest evidence for the bootstrapping claim is: (1) localization patterns appear before game reward improvement in G0's Shisen-Sho training (Figure 5), (2) perception accuracy co-evolves with game reward in G1's Shisen-Sho-Cifar10 training (Figure 6, bottom-left), and (3) the paper's mechanistic explanation — that large action spaces create a structural coupling where bad perception leads to bad actions which lead to negative rewards — is logically coherent and consistent with the observed failures on small-action-space (2048) and sparse-reward (Swap) games.

However, a genuine causal demonstration of bootstrapping would require an intervention experiment: e.g., artificially degrading perception mid-training and showing that reasoning performance drops; or fixing perception at various accuracy levels and measuring the rate of reasoning improvement. The perception reward ablation (Figure 8) comes closest to such an intervention but tests a different question (whether explicit perception reward helps). The evidence for the "mutual" direction — that reasoning improvements cause perception improvements — is weaker than the evidence for the "perception enables reasoning" direction. The paper argues that "optimal perception patterns are incentivized only through correct reasoning patterns that lead to actions yielding rewards," but this is a logical inference, not an experimentally isolated mechanism. The perception accuracy in G1's Shisen-Sho-Cifar10 does improve during RL without explicit perception reward, which is consistent with reasoning driving perception improvement, but could also be explained by the model learning better visual features as a byproduct of processing more game images, independent of reasoning quality. A controlled experiment where reasoning quality is artificially manipulated while measuring perception improvement would strengthen this claim considerably.

Does the paper demonstrate that G1 "consistently surpass their teacher across all games" (abstract claim)?

Yes, this is directly supported by Table 1. G1-7B scores higher than Claude-3.7-Sonnet-Thinking on all four games, with margins ranging from 20% (2048) to 81% (Swap). This is a clear, quantitative result. However, the comparison is not necessarily fair in terms of total compute: Claude-3.7-Sonnet-Thinking is evaluated zero-shot (no environment-specific training), while G1-7B undergoes both SFT (1,000 teacher-distilled examples) and RL (500 steps × 128 observations × 5 candidates = 320,000 environment interactions). The paper does not claim these are compute-matched — it presents G1 as a method that can produce a model that outperforms the teacher, not as evidence that the student is more compute-efficient. However, readers unfamiliar with the distinction between "outperforms in evaluation" and "achieves better performance per unit of compute" may interpret the claim as implying efficiency, which the paper does not establish.

Additionally, the paper does not report an ablation where Claude-3.7-Sonnet-Thinking itself is fine-tuned or given RL training on VLM-Gym. It is possible — perhaps likely — that the teacher model would also benefit from environment-specific training and might maintain or extend its lead over the student if given equal training opportunities. The G1 result demonstrates effective knowledge distillation and refinement, not necessarily that the 7B architecture is superior to Claude's architecture.

Does the paper's diagnostic framework (perception prior gap, inaccurate reward credit, sparse reward) hold up?

The framework is descriptive and post-hoc, not predictive. The paper identifies three factors that explain why RL succeeded or failed on each game, but all three are identified after observing the results, and no experiment is conducted where one of these factors is systematically varied while holding others constant. The Shisen-Sho vs. Shisen-Sho-Cifar10 comparison does vary perceptual difficulty while holding rules constant, which isolates the "perception prior gap" factor — and indeed, RL is slower on the perceptually harder variant, consistent with the framework. However, the other two factors (inaccurate reward credit in 2048, sparse reward in Swap) are identified based on a single game each, with no within-game variation to confirm they are the causal drivers. A stronger validation would systematically vary action space size within a single game (e.g., a simplified 6×6 Shisen-Sho variant with smaller action space) or vary reward density within a single game (e.g., providing intermediate rewards for partial progress in Swap) and show that the G0 vs. G1 effectiveness gap changes as predicted.

What is missing?

• No transfer or multi-task experiments. All models are trained on individual games. The paper does not test whether RL on Shisen-Sho transfers to Shisen-Sho-Cifar10 (shared rules, different visuals), or whether training on multiple games jointly yields more general perception–reasoning skills. The "multi-game parallel training" capability of VLM-Gym is described as a feature but is never experimentally exploited.

• No statistical rigor. All reported scores are point estimates without confidence intervals. With 10 runs per evaluation and substantial stochasticity in game outcomes (especially 2048, where tile spawns are random), the variance of these estimates could be large. The claim that G1-7B outperforms Claude on 2048 (1070 vs. 892) might or might not be statistically significant — the paper provides no way to assess this.

• No scaling analysis. The paper only trains 7B models. It does not report how G0 or G1 performance scales with model size, leaving open the question of whether the bootstrapping phenomenon is size-dependent (e.g., requires a minimum model capacity to emerge) or whether the relative benefit of RL vs. scale changes at larger model sizes.

• Limited game diversity. All four games are tile-matching puzzles with similar visual structures (grids of icons). The paper's claims about VLM-Gym supporting "diverse visual games" are accurate relative to text-only environments but narrow compared to the range of game genres (platformers, strategy games, RPGs, first-person environments). The findings about bootstrapping may be specific to the tile-matching puzzle domain.

• No evaluation of generalization to out-of-distribution game states. Because VLM-Gym uses random initialization and random exploration during training, the model likely encounters a wide variety of states. However, the paper does not systematically test whether the learned skills generalize to, e.g., larger boards, new tile types, or modified rules — the kind of systematic generalization testing that would distinguish genuine perception–reasoning from memorized patterns.

• Teacher model not fine-tuned. As noted above, the comparison between G1-7B and Claude-3.7-Sonnet-Thinking would be more informative if both models received the same RL training. The current comparison confounds model architecture, training data, and training algorithm — we cannot attribute G1's superiority to any single factor.

• No comparison to behavior cloning from the teacher. The G1 recipe uses the teacher for cold-start SFT followed by RL. A natural ablation would be to skip RL and simply use more SFT data from the teacher (behavior cloning). The paper does not report how far SFT alone can go — we see the post-SFT starting points in Figure 6 (game reward values at step 0) but not evaluation scores in the same format as Table 1 for the SFT-only checkpoint. This makes it impossible to quantify how much RL adds beyond what the teacher already provides.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in Training Efficiency Claims

The assumption or constraint. The paper's entire reinforcement learning pipeline — both G0 and G1 — depends on access to VLM-Gym, a programmable environment that provides ground-truth game states, verifiable rewards, and the ability to snapshot and rewind state for parallel action evaluation. The headline results in Table 1 and the training efficiency claims throughout Section 4 implicitly assume that such an environment exists for the target domain and that its computational overhead is negligible relative to model training. The paper does not report the wall-clock time, FLOP count, or infrastructure cost of running VLM-Gym during RL training (500 steps × 128 parallel games × 5 candidate actions = 320,000 environment interactions per game), nor does it compare this cost to the model inference cost.

The consequence. A practitioner attempting to deploy this method on a new game or interactive domain would face a substantial infrastructure engineering burden before any model training can begin. VLM-Gym must be purpose-built for each new game: the observation rendering pipeline, action parser, reward function, state snapshot/restore mechanism, and ground-truth perception extractor must all be implemented correctly. This is feasible for simple grid-based puzzle games (where the paper operates) but becomes considerably harder for complex 3D environments, games with physics simulation, or domains where ground-truth state information is not easily extracted from the engine. The paper's claim that VLM-Gym enables "scalable multi-game parallel training" (abstract) is accurate for the four implemented games but leaves unaddressed the cost of scaling to new game types. Furthermore, the G1 cold-start pipeline requires the teacher model (Claude-3.7-Sonnet-Thinking) to generate 1,000 reasoning–action trajectories per game — an API cost that is not quantified. For a team without access to proprietary reasoning models, this distillation step may be impractical.

What evidence exists in the paper. The paper reports hyperparameters for environment interaction (Section 3.3: batch size 128, group size 5, 500 RL steps) but provides no FLOP accounting or infrastructure cost analysis. The open-source release of VLM-Gym is mentioned (abstract: "Source code including VLM-Gym and RL training are released"), which partially mitigates the reproducibility concern for these specific games, but does not address the cost of adapting VLM-Gym to new domains.

Mitigation status. The paper does not address this limitation directly. The release of VLM-Gym as open source is a partial mitigation in that it provides a reference implementation, but the fundamental reliance on a purpose-built programmable environment is inherent to the approach. The paper frames VLM-Gym as a research platform — "we believe VLM-Gym and the RL training framework could also serve as useful resources for the community for future research" (Section 6) — which is an honest positioning, but it means the method's applicability to arbitrary interactive visual environments is not established.

Training Is Per-Game with No Evidence of Cross-Game Transfer or Multi-Task Learning

The assumption or constraint. All experiments in the paper train separate models for each game. The authors explicitly state: "Each game was trained independently" (Section 3.3, Training Details). There is a separate G0 model trained on 2048, a separate G0 model trained on Shisen-Sho, a separate G1 model trained on Shisen-Sho-Cifar10, etc. The models do not share parameters, co-train on multiple games, or leverage experience from one game when learning another. The paper does not test whether skills learned in one game transfer to a related game — for instance, whether an RL-trained Shisen-Sho model performs above chance on Shisen-Sho-Cifar10 before any CIFAR-specific training, or whether co-training on both simultaneously yields a more robust perception system.

The consequence. This is not merely a missing experiment; it undermines a key claimed contribution of VLM-Gym itself. The paper states that VLM-Gym "supports parallel execution across numerous game states simultaneously, as well as across multiple distinct games" and "facilitates efficient large-batch training and enables research into multi-task reinforcement learning for VLMs" (Section 2, bullets). Yet this multi-game capability is never used for its stated purpose. A reader expecting evidence that the framework enables multi-task VLM agents — agents that can play multiple games, or that learn general perception–reasoning skills transferable across games — will find none. The paper demonstrates that the framework can train per-game specialists efficiently, but leaves completely open the question of whether it can produce a single model that plays multiple games, or whether skills transfer between visually or structurally similar games. The "bootstrapping" finding — that perception and reasoning co-evolve during RL — is demonstrated within individual games and may or may not generalize across games.

What evidence exists in the paper. The absense of transfer or multi-task experiments is explicitly noted in Section 3.3 and implicitly in all result tables (each model is game-specific). The Shisen-Sho and Shisen-Sho-Cifar10 pair would be the natural testbed for a transfer experiment (shared rules, different visual complexity), but no such experiment is conducted. The paper's related work section (Section 5) does not mention transfer or multi-task learning as a goal, suggesting this limitation is a deliberate scope constraint rather than an oversight, but it significantly narrows the practical applicability claim.

Mitigation status. The paper does not address this limitation. Future work would need to demonstrate: (a) whether a single model can be co-trained on multiple games via multi-task RL, (b) whether co-training yields better per-game performance than single-game training (positive transfer), and (c) whether skills acquired in one game (e.g., grid-based coordinate reasoning in Shisen-Sho) improve learning speed or asymptotic performance in a related game (e.g., Shisen-Sho-Cifar10 or Swap). These experiments are feasible within VLM-Gym — the games share grid-based visual structures and coordinate-based action formats — and their absence is a significant gap given the paper's framing of VLM-Gym as a multi-game platform.

All Four Games Are Grid-Based Tile-Matching Puzzles, Severely Limiting Generality Claims

The assumption or constraint. The paper's four games — 2048, Shisen-Sho, Shisen-Sho-Cifar10, and Swap — all belong to a narrow genre: grid-based matching puzzles where the agent views a 2D board, identifies tiles by their visual features, and selects coordinate pairs or directions as actions. The visual input is always a static 2D grid of discrete icons or numerals; the action space is always discrete and small relative to continuous control domains; the dynamics are fully deterministic given the action (except for random tile spawns in 2048); and the reward is always a binary +1/-1 signal for correct/incorrect actions. The paper's claims about emergent perception–reasoning patterns, bootstrapping dynamics, and the diagnostic framework (perception prior gap, inaccurate reward credit, sparse reward) are all derived from and demonstrated exclusively on this game genre.

The consequence. The diagnostic framework and training recipes may not generalize to game genres with fundamentally different structures. Consider several counterexamples:

• Continuous control games (e.g., Atari Pong, racing games): Actions are joystick movements or button combinations, not discrete coordinate selections. The "large action space → good credit assignment" logic does not apply in the same way.
• First-person 3D environments (e.g., Minecraft, Doom): The visual input is a perspective rendering, not a top-down grid. Tile localization patterns (enumerating coordinates) are not a natural representational strategy for perspective views.
• Games with temporal dynamics (e.g., platformers): The state changes continuously between actions, and perception must track moving objects, not static tile layouts.
• Games with partial observability (e.g., fog-of-war strategy games): The model cannot perceive the entire board, so the "localization pattern" (enumerating all tiles with coordinates) is impossible.

The bootstrapping hypothesis — that large action spaces structurally couple perception and reasoning because incorrect perception leads to invalid actions — is plausible for any domain with discrete choices where random guessing rarely succeeds. But the specific mechanism (localization patterns enabling enumeration-based reasoning) may be an artifact of the grid-matching genre. A reader interested in deploying this method for, say, a robotic manipulation task or a real-time strategy game would need to determine whether the bootstrapping dynamic would emerge in their domain, and the paper provides no evidence beyond the tile-matching domain to support such extrapolation.

What evidence exists in the paper. The paper's own limitation section (Section 4.4) acknowledges this implicitly: "VLM-Gym currently comprises a specific set of visual games... which, while offering varied perceptual and reasoning challenges, predominantly feature relatively straightforward rule sets. A significant avenue for future work involves expanding this suite to include games with more complex mechanics, deeper strategic requirements, and diverse genres." This is an honest acknowledgment, but the gap is larger than "straightforward rule sets" — the structural similarity across all four games (grids, tiles, discrete coordinate actions, binary rewards) means the findings may reflect properties of this narrow genre rather than general principles of VLM-RL interaction.

Mitigation status. The paper explicitly flags this as future work and does not claim generality beyond the tested games. The "compositional difficulty" feature of VLM-Gym (adjustable perceptual complexity, reasoning depth) is presented as a capability that could support broader generalization studies, but no such studies are conducted. A reader should treat the findings as demonstrated for grid-based matching puzzles only, pending replication on games with substantially different visual, action, and reward structures.

No Statistical Rigor: All Results Are Point Estimates Without Confidence Intervals or Significance Tests

The assumption or constraint. All quantitative results in the paper — Table 1's evaluation scores, Figure 3's reward curves, Figure 6's perception and reasoning accuracy curves — are reported as point estimates (averages over 10 evaluation runs) without any measure of variance. The paper does not report standard deviations, standard errors, confidence intervals, or the results of any statistical significance test comparing model performance. For the training curves (Figures 3, 5, 6, 7), only mean values are plotted; there are no shaded regions indicating variance across runs or seeds.

The consequence. This makes it impossible to assess whether the paper's key comparative claims are statistically reliable or within the range of run-to-run noise. Several specific comparisons are suspect without variance information:

• G1-7B vs. Claude-3.7-Sonnet-Thinking on 2048: G1 scores 1070, Claude scores 892 — a difference of 178 points. But 2048 has high inherent variance due to random tile spawns; a run where two "1024" tiles happen to appear adjacent can yield a large score spike unrelated to agent skill. With only 10 runs per model, the standard error of the mean could be large enough that the difference is not statistically significant. The paper provides no way to evaluate this.

• G0-7B vs. o1 on Shisen-Sho: G0 scores 12.8, o1 scores 3.9 — a substantial difference, but o1's evaluation is also based on only 10 runs. If o1's performance is bimodal (some runs succeed, most fail completely, producing high variance), the difference might be smaller in expectation than the point estimates suggest.

• Training curve comparisons (Figure 7): The G0 and G1 curves for Shisen-Sho appear to converge by step 500, but without error bands, it's unclear whether the apparent convergence is real or an artifact of plotting means. If both curves have high step-to-step variance, the final values might not be distinguishable.

The absence of variance reporting is particularly problematic given the small number of evaluation runs (10 per model) and the known stochasticity of the game environments (random tile spawns in 2048, random board layouts in Shisen-Sho). A practitioner deciding whether to adopt this method needs to know not just the mean performance but the reliability — does the trained model consistently outperform baselines, or does it occasionally collapse to near-random performance?

What evidence exists in the paper. The paper provides no variance information anywhere. The evaluation protocol is described (Section 3.4: "100 steps × 10" for 2048, "36 steps × 10" for Shisen-Sho, "1 step × 100" for Swap) but only means are reported. The training curves (Figures 3, 5, 6, 7) are plotted without error bands. This is a significant methodological weakness relative to standard practice in both RL and ML evaluation, where confidence intervals or at minimum standard deviations are expected.

Mitigation status. The paper does not address this limitation. The authors do not mention variance, report error measures, or discuss the statistical reliability of their comparisons. This is a correctable limitation — the raw data from the 10 evaluation runs and the training runs presumably exist and could be used to compute standard errors — but as presented, the quantitative claims should be interpreted as suggestive rather than statistically established.

The G1 vs. Teacher Comparison Confounds Architecture, Training Data, and Training Algorithm

The assumption or constraint. The paper's headline comparison — "G1 models consistently surpass their teacher across all games and outperform leading proprietary models like Claude-3.7-Sonnet-Thinking" (abstract) — compares a fine-tuned + RL-trained 7B parameter open-source model against a zero-shot proprietary model evaluated through an API. These two models differ along multiple dimensions simultaneously: (1) architecture (Qwen2.5-VL-7B vs. Claude-3.7-Sonnet-Thinking, whose architecture and parameter count are undisclosed), (2) training data (G1 has seen 1,000 environment-specific SFT examples + 320,000 environment interactions; Claude has seen none), (3) training algorithm (G1 uses SFT + GRPO; Claude presumably uses a different training procedure), and (4) inference procedure (G1 uses the structured <perception>...<answer> format enforced by the format reward; Claude generates free-form text in response to the same prompt).

The consequence. Because all four factors vary simultaneously, the comparison does not isolate which factor or combination of factors drives G1's superior performance. Several alternative hypotheses are consistent with the data:

• Environment-specific training alone explains the gap: If Claude-3.7-Sonnet-Thinking were given the same 1,000-example cold-start SFT and 500 steps of RL on VLM-Gym, it might dramatically outperform G1-7B, maintaining or widening its lead over the student. The zero-shot evaluation tells us about Claude's out-of-the-box game-playing ability, not about its capacity to learn from environment interaction.

• The structured output format is doing the work: G1 is explicitly trained (via the format reward) to produce perception in a structured, parseable format that separates visual description from reasoning from action. Claude, prompted zero-shot, may produce less-structured outputs that are harder to parse or that conflate perception and reasoning, leading to worse decisions. The format, not the model quality, could explain the gap.

• The teacher distillation is doing the work, not the RL: G1 undergoes SFT on Claude-distilled trajectories before RL, meaning it already starts from a position that incorporates Claude's reasoning patterns. The post-SFT starting points in Figure 6 show non-trivial game rewards (e.g., Swap at ~−0.8 rather than ~−1.0), but the paper does not report evaluation scores (in the Table 1 format) for the SFT-only checkpoint. Without this ablation, we cannot determine how much of G1's advantage over Claude comes from the SFT phase (distilling Claude's knowledge into a smaller model that can then be further optimized) vs. the RL phase (which improves beyond what the teacher provided).

What evidence exists in the paper. The paper does not include an SFT-only evaluation in Table 1, nor does it include Claude fine-tuned on VLM-Gym as a baseline. The training curves in Figure 6 show game reward at step 0 (post-SFT, pre-RL) for the training environment, but these are not comparable to the Table 1 evaluation scores (which use cumulative game scores under a different metric). The paper's claim that "our G1 models consistently surpass their teacher across all games" (abstract) is true as a statement about final evaluation scores, but the causal attribution — that the G1 training recipe produces a model superior to the teacher in some general sense — is not directly supported by the experimental design.

Mitigation status. This is partially addressed by the paper's framing: the abstract does not claim that G1's architecture is better than Claude's, only that the trained G1 model outperforms the untrained Claude model. Section 4.3 ("SFT+RL Jointly Helps the Base Model Outperform the Supervisor") acknowledges the distinction between SFT and RL contributions but does not provide the ablation that would separate them. The paper characterizes the result as "SFT+RL jointly helps," which is accurate but leaves the relative contribution of each component unquantified. A natural experiment would evaluate: (a) SFT-only G1, (b) SFT-only G1 with more data (behavior cloning to convergence), and (c) Claude given the same RL training — but none of these are reported.

The Revision/Correctness Problem in Sequential Decision-Making Is Not Addressed

The assumption or constraint. All training and evaluation in the paper uses a single-step RL formulation: at each timestep, the model receives an observation, outputs an action, and receives a reward for that single action. The GRPO algorithm optimizes per-step action selection assuming each step is an independent decision problem. This is a deliberate simplification — the paper does not model multi-step planning, does not use trajectory-level credit assignment (e.g., discounting future rewards, Monte Carlo returns), and does not address the problem of error propagation across steps. If the model makes a suboptimal but valid move at step $t$ that makes the game harder at step $t+1$ (e.g., in 2048, moving left when right would have set up a better board), the single-step reward provides no signal about this mistake until the downstream consequences manifest many steps later.

The consequence. The paper's training objective optimizes for immediate action correctness, not for strategic gameplay that maximizes cumulative reward over an episode. This is particularly visible in the 2048 results: G0 fails because short-term rewards are noisy (random actions sometimes produce merges), but even G1, which achieves strong performance (1070, Table 1), may be learning a myopic policy that selects merges greedily without planning ahead for tile placement. The paper does not report episode-length metrics (e.g., average number of steps before game-over in 2048, completion rate in Shisen-Sho) that would distinguish a myopic tile-merger from a strategic player. In the Shisen-Sho and Swap games, the fixed evaluation length (36 steps for Shisen-Sho, 1 step for Swap) further obscures multi-step effects — 36 Shisen-Sho steps may not be enough to complete a board, and the single-step Swap evaluation completely ignores the sequential nature of the game (each swap changes the board configuration for the next step).

This limitation has implications for the bootstrapping hypothesis as well. The paper argues that perception and reasoning co-evolve because accurate perception enables correct action selection. But "correct" here means "immediately rewarded" — a model that learns to perceive tiles accurately enough to make short-term matches may never develop the deeper strategic reasoning (e.g., planning several swaps ahead to create cascading matches) that distinguishes expert human play. The current training signal may saturate at a level of perceptual and tactical competence that falls short of genuine strategic mastery.

What evidence exists in the paper. The evaluation protocol itself reveals the simplification: 2048 is evaluated over 100 steps (a single episode), Shisen-Sho over only 36 steps, and Swap over single steps. There is no discount factor, no value function, and no multi-step return estimation mentioned anywhere in the paper. The GRPO advantage is computed from single-step rewards only (Equation 6). The paper's discussion of "sparse rewards" in Swap (Section 4.1) refers to the rarity of positive rewards at the single-step level, not to the temporal credit assignment problem where an action's consequences are only visible after several steps.

Mitigation status. The paper acknowledges a related limitation in Section 4.4: "Future research could focus on developing effective RL strategies or reward-shaping mechanisms to effectively train VLM agents in scenarios with extended multi-turn interactions where feedback is infrequent, thereby better addressing the complexities of long-horizon decision-making and sparse reward tasks." This is a weaker version of the limitation described here — it frames the issue as "sparse rewards" in specific games like Swap, rather than as a fundamental property of the single-step RL formulation. The paper does not discuss the possibility that even in games with dense per-step rewards (like 2048), a myopic policy may be suboptimal for long-horizon performance. Extending the method to use trajectory-level rewards, value functions, or Monte Carlo returns would be a natural next step, but the current results should be understood as optimizing immediate action quality rather than strategic depth.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new RL algorithm, a new model architecture, or a new theoretical framework. Its contribution is more subtle and, in some ways, more practically significant: it provides the first systematic evidence that perception and reasoning in VLMs are not independent modules to be optimized separately, but are causally coupled abilities that co-evolve under a single verifiable outcome signal. This reframes the conversation around VLM training from a static, separable-capabilities view (perception accuracy + reasoning accuracy = total capability) to a dynamic, interdependent one where improving one ability creates pressure for the other to improve, and where the quality of the reward signal — not the presence of explicit supervision for each capability — determines whether this virtuous cycle engages.

The magnitude of this shift is not paradigm-level — the paper does not propose that all VLM training should be RL-driven, nor does it claim to replace supervised multimodal pretraining. Rather, it is a diagnostic reframing with direct methodological consequences. Before this work, a researcher observing that their VLM performs poorly on an interactive visual task might ask: "Should I improve the perception module or the reasoning module?" After this work, the more productive question becomes: "Is the reward structure of this environment capable of discriminating between good and bad perception–reasoning chains, or do I need a cold start to bootstrap the coupling?" The paper's three-part diagnostic framework — perception prior gap, inaccurate reward credit assignment, sparse reward — provides a concrete checklist for answering this question, grounded in the observed successes (Shisen-Sho) and failures (2048, Swap) of the G0 experiments.

The paper also reconciles a latent tension in the VLM-for-games literature. On one hand, works like PCA-Bench (Chen et al., 2024) and MMEvalPro (Huang et al., 2025) emphasize that perception and reasoning must be evaluated separately — that many apparent reasoning failures are actually perception failures in disguise. On the other hand, the RLVR literature (DeepSeek-R1, R1-V, MMEureka) demonstrates that outcome-only rewards can drive complex reasoning improvements without any decomposition into sub-capabilities. These perspectives appear contradictory: if perception errors masquerade as reasoning errors, how can a single outcome reward improve both simultaneously? The bootstrapping hypothesis resolves this: the single outcome reward works precisely because perception and reasoning are coupled in environments where bad perception reliably produces bad actions. The outcome signal does not need to decompose into perception and reasoning components — the structure of the environment does that decomposition automatically through the action–reward mapping. This resolution shifts research attention from the question "How do we design rewards for each sub-capability?" to "How do we design environments where the outcome reward naturally discriminates good perception–reasoning chains from bad ones?"

Several research directions become more attractive in light of this work, and several become less so:

More attractive directions:

• Environment design for VLM-RL. If the action space size and reward density of an environment determine whether bootstrapping occurs, then designing environments with the "right" properties — large action spaces that penalize sloppy perception, dense enough rewards to provide a learning signal but not so dense that random actions succeed — becomes a first-class research problem. VLM-Gym's compositional difficulty feature is a prototype of this idea.
• Cold-start recipes that separate perception from reasoning. The G1 pipeline's key insight — give the teacher ground-truth perception from the environment, have it produce only reasoning, then train the student on both — suggests a general pattern for scalable VLM training in simulated domains: use the simulator's privileged state for perception supervision, use a strong LLM for reasoning supervision, and use RL to weld them together.
• Analyzing VLM training dynamics through the lens of coupled learning. The paper's tracking of perception accuracy and reasoning accuracy as separate metrics during RL (Figure 6) provides a template for future VLM-RL studies. Rather than reporting only final task performance, researchers can and should report how each sub-capability evolves, to understand whether improvements are driven by better perception, better reasoning, or both.

Less attractive directions:

• Manual chain-of-thought curation as the default approach for VLM game-playing. The paper explicitly argues this faces "scalability limitations" (Section 1) and demonstrates an alternative (G0) that achieves strong performance with zero human annotation on amenable games. While manual CoT may remain necessary for the hardest games (like Swap, where G1's cold start was essential), the default assumption should shift toward trying pure RL first, then adding teacher distillation only if the diagnostic framework predicts failure.
• Scaling model size as the primary solution to the knowing-doing gap. The comparison between Qwen2.5-VL-7B and Qwen2.5-VL-72B in Table 1 shows that a 10× increase in parameters yields minimal improvement on interactive game tasks (2.6 vs. 1.9 on Shisen-Sho, 0.8 vs. 0.4 on Shisen-Sho-Cifar10). This suggests that the knowing-doing gap is not primarily a capacity problem — at least not at these scales — and that investment in training methodology (RL, cold-start recipes) yields higher returns than investment in parameter count for this class of tasks.

Follow-Up Research This Work Enables

1. Multi-game co-training with VLM-Gym to test whether bootstrapping transfers across games. The paper trains separate models per game but never tests whether skills transfer between structurally similar games in VLM-Gym. A natural next experiment would co-train a single model on Shisen-Sho and Shisen-Sho-Cifar10 simultaneously, measuring whether: (a) the CIFAR variant learns faster (perception transfer from shape-based tiles to image-based tiles), (b) the shape variant reaches a higher asymptote (reasoning transfer in the other direction), or (c) negative transfer occurs (the different visual domains interfere). This would directly test whether the perception–reasoning coupling learned in one game generalizes to another game with shared rules but different visual complexity — a question the paper's single-game design cannot address. A strong follow-up would also test transfer to a held-out game (e.g., train on 2048 + Shisen-Sho, evaluate on Swap) to determine whether any general "grid reasoning" skill emerges.

2. Systematic variation of action space size within a single game to isolate the credit assignment mechanism. The paper argues that Shisen-Sho's large action space (~2000 possible pairs) is what prevents incorrect perception–reasoning chains from being rewarded, enabling bootstrapping. This hypothesis is supported by comparing Shisen-Sho (large action space, bootstrapping observed) against 2048 (tiny action space, bootstrapping fails), but these games differ along many dimensions beyond action space size. A direct test would create Shisen-Sho variants with systematically reduced action spaces — e.g., by shrinking the board from 8×8 to 6×6 to 4×4, which reduces the number of possible tile pairs exponentially — and measure the point at which bootstrapping (measured by localization pattern emergence, Figure 5) breaks down. The prediction: below some threshold action space size, the G0 reward curve should flatten (like 2048) because random guessing starts succeeding often enough to contaminate the credit assignment signal. This would convert the paper's post-hoc observation into a quantitative relationship.

3. Behavior cloning baseline to quantify how much RL adds beyond teacher distillation. The paper reports that G1-7B outperforms its teacher (Claude-3.7-Sonnet-Thinking) but does not isolate whether this is due to: (a) the SFT phase capturing the teacher's reasoning and distilling it into a smaller model, (b) the RL phase improving beyond what the teacher demonstrated, or (c) a combination where SFT provides a necessary foundation that RL then refines. A straightforward ablation would train on increasing amounts of teacher-distilled SFT data (100, 500, 1000, 5000, 10000 examples) without any RL, and measure where performance saturates. If SFT-only with sufficient data approaches G1's performance, then RL is adding little beyond distillation. If SFT-only saturates well below G1 (as the Swap curves in Figure 7 suggest — SFT-only starts at game reward ~-0.8), then RL is providing a genuine capability gain beyond what the teacher can provide through demonstrations. This would also test the paper's implicit claim that "SFT+RL Jointly Helps the Base Model Outperform the Supervisor" (Section 4.3 heading).

4. Extending the single-step RL formulation to trajectory-level credit assignment for long-horizon games. All training in this paper uses single-step rewards and a myopic policy. For 2048 — a game where strategic planning (setting up future merges by managing tile placement) is essential for high scores — this is a significant limitation. A strong follow-up would replace the single-step GRPO formulation with a trajectory-level variant: sample full episodes (not just single actions), compute Monte Carlo returns or use a learned value function as baseline, and optimize for cumulative episode reward. The hypothesis is that trajectory-level training would produce substantially higher 2048 scores than G1's 1070, because the model would learn to sacrifice short-term merges for long-term board positioning. This experiment would also clarify whether the 2048 results in Table 1 represent a ceiling imposed by the myopic objective or a ceiling imposed by the model's capacity — a question the paper leaves open.

5. Testing the bootstrapping hypothesis on a non-grid, non-puzzle game genre. All four games in VLM-Gym are grid-based matching puzzles with static, fully observable boards. The paper's bootstrapping hypothesis is stated in general terms — "perception and reasoning abilities mutually bootstrap each other throughout the RL training process" (abstract) — but has only been demonstrated in this narrow genre. A critical stress-test would implement a game from a different genre in VLM-Gym: for example, a simple platformer (perception requires tracking moving sprites, action space includes timing-dependent jumps), a first-person maze (partial observability, continuous turning), or a real-time puzzle (temporal dynamics, reaction speed matters). If the bootstrapping pattern (localization-like patterns emerging before performance improvement) appears in these structurally different games, the hypothesis gains substantial generality. If it fails, the hypothesis would need to be scoped to fully-observable discrete-grid environments — still interesting, but much narrower than the paper's framing suggests.

6. Direct causal intervention to verify the mutual direction of bootstrapping. The paper's evidence for bootstrapping is temporal (perception patterns precede reward improvement, Figure 5) and correlational (perception accuracy co-evolves with game reward, Figure 6 for Shisen-Sho-Cifar10), but lacks a direct intervention establishing causality in the "reasoning drives perception" direction. A clean experiment: take a G0 model mid-training on Shisen-Sho (say, at step 200, when localization patterns have started emerging but game reward is still low), freeze its reasoning head (the layers that map perception representations to action logits), and continue training only the visual encoder with the same GRPO objective. If perception continues to improve even with frozen reasoning, the "reasoning drives perception" claim would be weakened — perception would be improving independently, perhaps through self-supervised learning from exposure to more game images, rather than through pressure from the reasoning module. If perception improvement stalls immediately, the claim would be strengthened. Conversely, freeze the visual encoder and continue training only the reasoning head — this would test the "perception enables reasoning" direction. These experiments are feasible within the paper's existing infrastructure and would transform the bootstrapping claim from an observed pattern into a demonstrated mechanism.

Practical Applications and Downstream Use Cases

1. Training VLM agents for grid-based puzzle games and UI automation tasks without human annotation. The G0 result — taking Qwen2.5-VL-7B from a Shisen-Sho score of 1.9 to 12.8 using pure RL with zero human labels — demonstrates that for tasks with the right structural properties (large discrete action space, binary verifiable rewards, fully observable grid-based state), a weak VLM can bootstrap itself to competence entirely through environment interaction. This has direct application to domains like automated game testing (where a VLM agent plays through levels to find bugs), educational game tutors (where a VLM learns to solve puzzles to demonstrate solutions), and grid-based UI automation (where actions are discrete UI element selections and correctness is verifiable by checking whether the next screen matches expectations). The key practical insight is that for these domains, the investment should go into building a high-fidelity simulator with verifiable rewards and parallel action evaluation support, not into collecting human demonstration data. The 4–10× effective compute multiplier (G0-7B outperforming the 72B base model by 3.9× on Shisen-Sho and 10× on Shisen-Sho-Cifar10) provides a rough ROI estimate for simulator construction vs. human annotation.

2. Scalable teacher-student distillation pipelines for VLM decision-making where a strong teacher and programmable simulator are both available. The G1 pipeline — ground-truth perception from simulator + reasoning from teacher → SFT on student → RL refinement — has an architecture that generalizes beyond games to any domain with a programmable simulator that can provide privileged state information. Consider a robotic manipulation task: a physics simulator knows the exact positions and orientations of all objects, which a VLM agent cannot perceive perfectly from camera images but can be surfaced as "ground-truth perception" to a teacher model (e.g., a large VLM or a human operator). The teacher produces reasoning and action plans conditioned on accurate state information; these are distilled into a smaller VLM via SFT; and the smaller VLM then undergoes RL in the simulator (or on the real robot) to learn to perceive the state from raw pixels while retaining the teacher's decision-making patterns. The paper's finding that SFT+RL jointly outperforms the teacher (G1-7B > Claude-3.7-Sonnet-Thinking on all four games) suggests that this pipeline can produce a student that not only matches the teacher's quality but exceeds it through environment-specific RL refinement. The Swap result (G0-7B: 0.05, G1-7B: 0.78) is particularly instructive: when the base model cannot learn at all from pure RL, the perception-enhanced cold start transforms the task from impossible to tractable, with the student ultimately achieving an 81% improvement over the teacher.

3. Lightweight, per-task VLM fine-tuning for edge deployment in interactive visual applications. The paper demonstrates that a 7B model, after task-specific RL training, can outperform much larger models (72B, proprietary) and even its own teacher on the trained tasks. This has direct implications for deployment scenarios where a large general-purpose VLM is too expensive to run at scale: instead of querying Claude-3.7-Sonnet-Thinking or GPT-4o through an API for every game step (incurring latency, cost, and privacy concerns), an organization can distill task-specific capabilities into a small model using the G1 recipe and deploy it locally. The training cost (1,000 SFT examples + 500 RL steps, totaling a few hundred thousand environment interactions) is a one-time investment, after which inference runs on commodity hardware at a fraction of the API cost. This is particularly relevant for applications like mobile game AI opponents, interactive educational software, and assistive technology for visually impaired users playing puzzle games, where low latency and offline operation are critical requirements that API-based models cannot satisfy.

4. Diagnostic framework for predicting whether RL will work on a new interactive visual task. The paper's three-part categorization — perception prior gap, inaccurate reward credit assignment, sparse reward — provides a practical pre-training checklist for practitioners considering VLM-RL for a new domain. The diagnostic procedure suggested by the paper's results would be: (1) test the base model's zero-shot perception accuracy on a sample of task states (if near-zero, expect a perception prior gap — cold start likely needed); (2) measure whether random actions receive positive rewards at a non-trivial rate (if so, expect inaccurate reward credit — G0 will likely degenerate as it did on 2048); (3) measure the frequency of positive rewards during initial random exploration (if near-zero, expect sparse rewards — G0 will fail to learn, as on Swap). A task that passes all three checks (adequate base perception, large action space, non-zero early reward frequency) is a candidate for pure RL (G0). A task that fails one or more checks needs a cold start (G1-style SFT on teacher trajectories). This diagnostic framework, while not experimentally validated as a predictive tool, is directly actionable based on the paper's results and requires only access to the environment and the base model — no additional infrastructure.

When to Prefer This Method

The paper does not explicitly position G0 and G1 against a named alternative method (e.g., "prefer G0 over behavior cloning when X") with quantified tradeoffs. However, the experimental results implicitly define a decision boundary based on game properties. This boundary can be inferred from the paper's diagnostic framework and the G0 vs. G1 comparison in Figure 7:

• Prefer pure RL (G0) when: The game has a large discrete action space (like Shisen-Sho's ~2000 tile pairs), the base model's visual perception is at least minimally functional (it can roughly distinguish tile shapes and colors), and the reward signal is dense enough that positive rewards occur with non-trivial frequency during random exploration. Under these conditions, the G0 recipe provides strong performance (Shisen-Sho: 12.8 vs. base 1.9) with zero annotation cost and no dependence on a teacher model. The action space size is the critical discriminator — it must be large enough that random actions rarely succeed, so the reward signal cleanly separates correct from incorrect perception–reasoning chains.

• Prefer perception-enhanced cold start + RL (G1) when: The game has a small action space (like 2048's 4 directions), where even random policies receive positive rewards regularly and pure RL degenerates (G0-2048: "the model crashed, completely ignored the screenshot image"); OR the game has extremely sparse rewards (like Swap), where the base model rarely or never receives positive rewards and pure RL produces no learning signal (G0-Swap: flat at -1.0); OR the base model's perception is so poor that it cannot begin to engage with the task (perception prior gap, as partially seen in Shisen-Sho-Cifar10 where G1 substantially outperforms G0). The G1 recipe requires a programmable environment that can provide ground-truth perception (to prompt the teacher) and a strong teacher model that can generate reasoning conditioned on that perception, but these are one-time costs that produce a student model capable of eventually surpassing the teacher.

• Prefer scaling model size over RL post-training when: The game requires capabilities fundamentally absent from the base model — the paper's difficulty bin 5 analogy from the compute-optimal scaling paper is instructive here. If the base model's zero-shot perception accuracy is zero and no amount of environment interaction provides a learning signal (Swap with G0), and if no teacher model or programmable environment is available for a cold start, then scaling to a larger base model that does have the requisite capabilities may be the only viable path. However, the paper's comparison of Qwen2.5-VL-7B vs. 72B (Table 1) suggests that scale alone provides minimal improvement on interactive tasks in this domain — a 10× parameter increase yielded only a 0.7 point improvement on Shisen-Sho (1.9 to 2.6) and no improvement on 2048 (246 to 243). This implies that the "scale instead of train" escape hatch is narrow for the game-playing domain, and that the G0-vs.-G1 decision is the more practically relevant choice for most scenarios within this task family.