Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents

ArXiv: 2510.23691

🎯 Pitch

A single game-playing agent, Game-TARS, matches the generality of fresh human players on unseen 3D web games by simply predicting native keyboard and mouse actions. It doubles the prior state-of-the-art on Minecraft tasks and even beats frontier models like GPT-5 at FPS benchmarks, proving that scaling simple device-level control across hundreds of games creates a surprising degree of transferable, generalist intelligence.

---

1. Executive Summary

This paper introduces Game-TARS, a generalist game agent trained with a unified, scalable action space anchored to native keyboard–mouse inputs rather than environment-specific APIs or GUI elements. The system is built on Seed-VL and Qwen2.5-VL backbones, continually pre-trained on over 500B tokens spanning ~20k hours of gameplay across ~500 games, then post-trained with cross-domain agent trajectories. Key mechanisms include a decaying continual loss (exponentially down-weighting loss for repetitive actions to reduce causal confusion — e.g., decaying weight by γ^(k−1) for k consecutive identical actions), native Sparse ReAct pretraining via Thinking Aloud (interleaving reasoning text only at critical decision points, collected through synchronized screen/keyboard/audio recording and aligned via visual-anchor cursor tracking), and Sparse Thinking via Reinforcement Fine-Tuning (rejection sampling that retains reasoning steps only when the predicted action matches the ground truth). On open-world Minecraft tasks, Game-TARS achieves approximately 2× the success rate of the previous state-of-the-art (72.0% on embodied tasks vs. 42.1% for UI-TARS-1.5), while on unseen FPS benchmarks it outperforms GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet, establishing that device-level action representations combined with cross-domain pre-training produce transferable sensorimotor skills that generalize to entirely unseen environments — though the unified action space requires substantially more data to match GUI-based approaches in low-data regimes.

2. Context and Motivation

The Core Problem: Action Spaces Are the Bottleneck for Generalist Game Agents

The fundamental question this paper tackles is: what is the right action representation for building a single agent that can play hundreds of different video games without per-game customization? This is not merely an engineering convenience question — it strikes at whether generalist artificial agents are architecturally feasible at all under current paradigms.

The action space — the set of possible outputs an agent can produce to interact with its environment — determines everything about an agent's architecture. It shapes what training data can be collected, how models are trained, what kinds of generalization are possible, and whether an agent trained on one set of games can transfer any skill to another. Despite this central role, the paper argues that the field has been systematically under-thinking the action space problem, treating it as an implementation detail rather than a first-class architectural decision.

Consider the state of game AI in 2025. We have agents that achieve superhuman performance in specific games — AlphaStar in StarCraft II (Vinyals et al., 2019), agents mastering Atari games from pixels (Mnih et al., 2015), VPT learning Minecraft behaviors from YouTube videos (Baker et al., 2022). Each of these is a triumph of domain-specific engineering. Each relies on a tightly coupled action space: AlphaStar uses StarCraft II's internal API, VPT uses Minecraft's specific keybinding-to-action mapping, Atari agents use pre-defined discrete action sets (e.g., "FIRE," "RIGHT," "LEFT-FIRE"). These action spaces are defined by human engineers for each environment. When you want the agent to play a new game, you redefine the action space from scratch, redesign the training pipeline, and recollect training data tailored to that new action space.

This tight coupling creates three concrete problems that the paper identifies:

1. No cross-game data aggregation. If every game requires its own action space, you cannot pool trajectory data from multiple games into a single training run. Each game's data is siloed. This directly contradicts the bitter lesson of deep learning (Sutton, 2019): methods that leverage large-scale computation and data consistently outperform methods that rely on domain-specific human knowledge. By enforcing per-game action space design, the field has been systematically preventing game agents from benefiting from the same scaling laws that have driven progress in language and vision.

2. No transfer of sensorimotor skills. When an action space is defined in terms of high-level game concepts (e.g., "craft_item(iron_pickaxe)"), the agent learns to map observations to API calls. It does not learn the universal sensorimotor primitives — how to move a cursor to a target, how to time a click, how to coordinate keyboard and mouse for fluid movement — that transfer across all games. A human who learns to aim in one FPS game brings that skill to every other FPS game. API-based agents cannot do this because there is no shared representation of low-level motor actions.

3. Brittleness to environment changes. When an action space is defined in terms of GUI elements or API calls, any change to the interface — a redesign of the crafting menu, a change in key bindings, a different OS — breaks the agent. The paper is explicit about this in Section 2.1: "Significant discrepancies in GUI element structures and interaction modalities across Windows, macOS, and Linux, or even between different software applications, prevent the realization of a truly universal action space." An agent that operates at the keyboard-mouse level is immune to these changes because the input primitives remain constant regardless of the application layer above them.

These problems are not merely theoretical. The paper's own scaling experiments (Section 4.3.1, Figure 10) provide empirical evidence: when you use GUI-based action spaces (visual grounding — clicking on detected UI elements), agents achieve reasonable performance with modest training data, but the performance ceiling is low because data cannot be aggregated across games. The paper calls this the "GUI-based ceiling." The unified device-level action space starts lower (because pre-trained VLMs have no native understanding of keyboard-mouse inputs) but scales upward as more cross-game data is added, eventually overtaking GUI-based approaches.

Why This Problem Matters Beyond Academic Benchmarks

The practical stakes are high for several reasons that extend well beyond achieving higher scores on game benchmarks:

Games as a proxy for all computer interaction. The paper's thesis — clearly stated in the title and throughout — is that a generalist game agent is a stepping stone to a generalist computer-use agent. Every desktop application, every web interface, every productivity tool is ultimately controlled through the same keyboard-mouse primitives. If you can build an agent that learns to play 500 diverse games through these primitives, you have built an agent that has the sensorimotor foundation to operate any software. The paper explicitly tests this hypothesis by incorporating GUI agent trajectories, code agent trajectories, and research task trajectories into post-training (Section 3.5), then evaluating on non-game environments including the MiniWorld 3D simulator. The results — competitive performance on unseen simulators without any domain-specific training — suggest the transfer is real.

The scalability imperative. The paper frames its argument in terms of the "Bitter Lesson" (Sutton, 2019): general methods that leverage computation and scale consistently outperform domain-specific approaches in the long run. The quote from Section 4.3.1 is worth reproducing in full:

"This observation echoes the Bitter Lesson in AI: general and simple representations, when combined with scaling in data and compute, yield stable and long-term improvements."

The implication is that the entire direction of building per-game specialist agents — however impressive their individual results — is on the wrong side of history. The field needs action representations that scale rather than action representations that are convenient for a single environment. This paper is making a bet on a particular scaling path (unified keyboard-mouse primitives + massive cross-game pre-training) and providing the first large-scale empirical evidence that the bet pays off.

The data collection bottleneck. Per-game action spaces require per-game data annotation. Every new game requires human annotators to play it using whatever action format the agent expects, or engineers to write API wrappers that translate between agent actions and game commands. For 500 games, this is a massive recurring cost. The unified action space, by contrast, enables a fundamentally more efficient data collection pipeline: record humans playing games normally (keyboard, mouse, screen capture), synchronize the streams, and the raw recording is the training data. No per-game engineering is needed. The Think-Aloud pipeline described in Section 2.2 (and discussed further in the Technical Approach section) makes this concrete: annotators play games while verbalizing their thoughts, and the resulting audio-visual-motor stream is automatically processed into training sequences. This pipeline scales across games because it assumes nothing about the game's internal structure.

Where Existing Approaches Fall Short

The paper positions itself against three broad classes of prior work, each with distinct failure modes:

1. Specialist RL Agents (AlphaStar, DQN, VPT). These agents achieve remarkable performance within their target domains but provide zero transfer to new domains. VPT (Baker et al., 2022) is an especially relevant comparison because it shares Game-TARS's philosophy of using native human inputs (keyboard and mouse) and pre-training on video data. However, VPT's pre-training was confined entirely to Minecraft. It learned a Minecraft-specific behavioral prior that did not transfer because (a) the training data was only Minecraft, and (b) there was no effort to build an architecture or training protocol that would support cross-game data ingestion. Game-TARS can be seen as "VPT scaled across 500 games," with the critical addition of a ReAct reasoning component that VPT lacked entirely. The paper's Table 3 shows VPT achieving 6.0% ± 11.4% success rate on embodied Minecraft tasks — essentially non-functional on the evaluation benchmark — while Game-TARS achieves 72.0%, demonstrating that the unified action space only becomes powerful when coupled with cross-game scale.

2. API-Based and GUI-Based Agents (Voyager, UI-TARS, Cradle). This class of agents represents the dominant paradigm in 2024–2025. Voyager (Wang et al., 2023a) achieved impressive Minecraft performance by generating JavaScript code that called high-level Minecraft APIs — effectively using an LLM as a planner that sequences pre-defined function calls. The paper identifies three specific failures of this approach:

• Semantic gap: "learning to sequence functions rather than acquiring sensorimotor skills" (Section 5.1). The agent understands what actions do at a conceptual level but has no experience with the physical execution. This matters when execution is messy — when a crafting menu opens but the cursor is in the wrong position, or when an enemy interrupts a multi-step plan. API agents assume perfect execution; keyboard-mouse agents learn to deal with imperfect execution because they've experienced it millions of times in training.

• Scalability ceiling: Each new game requires defining a new set of API functions. The paper's Figure 10 (right panel) shows this empirically: GUI-based action spaces achieve good performance with small data but plateau, while the unified action space starts lower but continues improving as more cross-game data is added. At 200B additional cross-game tokens, the unified space overtakes the GUI space, achieving 65% vs. ~42% success rate in Minecraft.

• Non-transferability: API-based agents cannot play games for which no API wrapper exists. The paper's evaluation on unseen web games (Figure 7) and VizDoom (Figure 8) would simply not be possible with an API-based agent — these environments don't expose clean APIs. The fact that Game-TARS can be evaluated on these benchmarks at all is a direct consequence of its action space design.

UI-TARS (Qin et al., 2025) and UI-TARS-1.5 (Seed, 2025b) are especially relevant comparisons because they represent state-of-the-art GUI-based agents that operate by detecting and interacting with on-screen UI elements. Table 3 shows UI-TARS-1.5 achieving 42.1% success rate on embodied Minecraft tasks — the previous SOTA that Game-TARS doubles. The paper acknowledges that GUI-based approaches initially outperform unified keyboard-mouse approaches when training data is limited, but argues this is a temporary advantage that reverses at scale. The key insight is that GUI element detection (visual grounding) is itself a learned capability that comes "for free" in pre-trained VLMs, while keyboard-mouse control requires learning from scratch — hence the initial performance gap. But GUI representations cannot be unified across games because different games have different UI elements, different layouts, different interaction patterns.

3. Generalist Agent Architectures (Gato, SIMA). These works share Game-TARS's ambition of a single model for multiple environments, but the paper argues they stopped short on the action space problem. Gato (Reed et al., 2022) was landmark in demonstrating that a single Transformer could be trained across domains (Atari games, simulated robotics, image captioning, dialogue) by serializing everything into tokens. But Gato's action space was still domain-specific: it used different action tokenization schemes for different environments (discrete actions for Atari, continuous actions for robotics). This meant adding a new environment still required designing a new action tokenization. The paper positions Game-TARS as solving the problem Gato identified but didn't fully address: "a truly universal interface."

SIMA (Raad et al., 2024) is the closest prior work in spirit, sharing the "pixels in, keyboard/mouse out" philosophy. However, the paper is careful to note SIMA's limitations: "initial SIMA agents were evaluated primarily on short-horizon tasks and limited in-domain games, underscoring the learning challenge" (Section 5.2). SIMA demonstrated the feasibility of the paradigm but at a much smaller scale — fewer games, shorter trajectories, less cross-domain transfer. Game-TARS can be seen as scaling SIMA's core idea to ~500 games with 500B+ training tokens, and adding the sparse thinking mechanism that SIMA lacked.

4. Off-the-Shelf VLMs as Game Agents (GPT-5, Gemini-2.5-Pro, Claude-4-Sonnet). The paper's Figures 1, 7, 8, and 9 establish that general-purpose vision-language models, despite their impressive capabilities, are substantially worse at game playing than a model specifically pre-trained on game data. On VizDoom (Figure 8), Game-TARS-mini achieves roughly 5–15× higher episode rewards than GPT-5, Claude-4-Sonnet, and Gemini-2.5-Pro across all four maps. On web games (Figure 7), Game-TARS-mini is competitive with fresh human players and substantially outperforms GPT-5 in Temple Run and Airplane Flying.

The paper's explanation for this gap — implicit but clear from the architecture — is that general VLMs lack sensorimotor experience. They can reason about game screenshots but have no learned priors about action timing, mouse movement scaling, or the physical consequences of keyboard inputs. They haven't experienced the causal relationship between "press W" and "character moves forward" millions of times. Game-TARS has internalized these relationships through pre-training on raw keyboard-mouse trajectories, giving it a sensorimotor foundation that general VLMs cannot acquire through vision-language pre-training alone. This finding is practically significant because it suggests that even as general VLMs improve, there will remain a gap for tasks requiring fluid, real-time motor control — and that gap must be filled by domain-specific pre-training on interaction data.

How This Paper Positions Itself

The paper's positioning is simultaneously ambitious and precise. The ambition is clear from the framing: this is not a paper about building a better Minecraft agent or a better FPS agent. It is a paper about demonstrating that a particular architectural choice — device-level unified action spaces — enables a scaling pathway toward generalist computer-use agents. The current results (Minecraft, web games, VizDoom, MiniWorld) are presented not as endpoints but as existence proofs: evidence that the scaling trend is real, that cross-game transfer happens, and that the approach produces behaviors (corner-peeking in VizDoom, tracking fox movement in Minecraft over long horizons, adapting to arbitrary key rebindings) that would be impossible under specialist paradigms.

The precise part is the paper's honesty about limitations and tradeoffs. The unified action space does not work well with small data (Figure 10 — the GUI curve starts higher). It requires engineering effort for data collection (the Think-Aloud pipeline, visual-anchor cursor tracking, timestamp alignment). It demands substantial computation (500B tokens of pre-training). The paper doesn't claim the approach is universally better — it claims it is scalably better, which is a different and more specific claim. The scaling experiments in Section 4.3.1 are designed to test exactly this claim by showing how performance evolves as data volume and diversity increase along four dimensions: in-domain game data, cross-game data, multimodal general data, and cross-domain agent trajectories.

The paper also positions itself within a broader narrative about the future of AI agents. The post-training with cross-source agentic trajectories (Section 3.5) — code agents, GUI agents, MCP research agents — is explicitly framed as part of "cultivating a more comprehensive generalist agent" that goes beyond gaming. The vision is of a layered architecture where game pre-training provides the sensorimotor foundation (fast, reactive, physically grounded), and cross-domain post-training adds higher-level cognitive capabilities (structured reasoning, multi-step planning, information integration). This is a coherent architectural thesis: physical skills and cognitive skills are learned in sequence, with the former providing the substrate for the latter. The paper doesn't claim to have fully realized this vision — the post-training is preliminary and the evaluation is gaming-focused — but it has established the pathway and provided initial evidence that cross-domain data improves game performance, which is a necessary (though not sufficient) condition for this architecture to work.

A final positioning point concerns the relationship between reasoning and action. The paper introduces Sparse Thinking — the idea that explicit reasoning should only occur at critical decision points, not at every timestep — as a key efficiency mechanism. This is positioned against two straw men: (a) pure reactive agents that never reason (like VPT), which succeed on simple tasks but fail on complex multi-step planning, and (b) agents that reason at every step, which the paper shows actually degrade performance in fast-paced environments due to hallucinated reasoning loops and computational overhead. The Sparse Thinking mechanism (detailed further in the Technical Approach section) is presented as a cognitively plausible middle ground that mirrors human decision-making: deliberate planning before a complex crafting sequence, reflexive execution during the sequence itself. The ablation in Table 4 quantifies this tradeoff: in Minecraft, sparse thinking achieves 63% vs. 55% (no-thinking) and 45% (greedy thinking); in Doom, greedy thinking drops to 7.2 reward vs. 13.2 (no-thinking) and 11.5 (sparse). The optimal strategy is task-dependent, and the agent must learn to modulate its own reasoning depth — which Game-TARS does through the sparse thinking training process described in Section 3.3.

In summary, this paper addresses a structural bottleneck in the path toward generalist agents: the action space fragmentation problem. It provides a concrete architectural solution (unified keyboard-mouse primitives), a scalable data collection and training pipeline (Think-Aloud, decaying loss, sparse ReAct), and extensive empirical evidence that the solution works across diverse game environments including entirely unseen ones. The contribution is not a single algorithm or model but a paradigm shift in action representation that the paper argues is necessary for the field to benefit from the same scaling laws that have transformed language and vision.

3. Technical Approach

3.1 Reader Orientation

Game-TARS is a continually pre-trained vision-language model that outputs keyboard and mouse commands — it takes in a stream of screen images and a text instruction, and at each timestep produces both an internal reasoning trace (in natural language) and a physical action (keyPress, mouseMove, mouseClick) that the computer executes. The system solves the action-space fragmentation problem — the fact that every previous game agent needed a custom-designed set of output commands per game — by collapsing all possible computer interactions into three device-level primitives that are identical across every application, every game, and every operating system, then training a single model on massive cross-game data (~500 games, 500B tokens) so that sensorimotor skills learned in one game transfer to others.

3.2 Big-Picture Architecture (Diagram in Words)

The Game-TARS system has five major functional components, arranged in two training stages:

1. Data Collection Pipeline (Think-Aloud Recording) — captures synchronized screen video, keyboard/mouse input signals, and audio narration from human players verbalizing their thoughts while playing games. An ASR model transcribes the audio; a visual-anchor alignment system (tracking the on-screen mouse cursor) corrects timestamp mismatches to restore causal ordering; an LLM refines and consolidates the transcribed thoughts. This produces (observation, reasoning, action) triples at irregular intervals — reasoning only at critical decision points, not every frame.

2. Continual Pre-training Stage — takes a pre-trained vision-language model (Seed-VL-1.5 or Qwen2.5-VL-7B-Instruct) and trains it autoregressively on the collected trajectory data plus general multimodal data, producing a model that predicts both reasoning text and device-level actions from visual observations. This stage uses a novel decaying continual loss that exponentially down-weights the loss contribution of repeated identical actions to prevent the model from learning to simply copy the previous action.

3. Post-Training Stage — fine-tunes the pre-trained model on curated, high-quality data with several targeted strategies: (a) Automatic Action Space Augmentation, which randomly replaces key bindings in trajectories to force the model to read the System Prompt rather than memorizing key meanings; (b) Rejection Fine-Tuning for Sparse Thinking, which filters reasoning steps to keep only those that causally lead to correct actions; (c) Cross-Source Agentic Trajectories, which mixes in code agent, GUI agent, and research agent data to build higher-level cognitive capabilities on top of the game-trained sensorimotor foundation.

4. Memory Architecture (Dual-Tier) — a two-level memory system used consistently during both training and inference. Short-term contextual memory holds the most recent ~32k tokens of interleaved observations, thoughts, and actions in the model's context window. Long-term summary memory compresses older history by retaining only the "sparse thought" text summaries and discarding the large visual tokens, enabling the agent to maintain context across 2000+ interaction steps.

5. Inference-Time Execution Loop — at each timestep, the model receives the current visual observation and the accumulated memory, produces a reasoning step (only at sparse intervals determined by the model's learned policy), and outputs one or more keyboard/mouse actions. The environment executes these actions, returns a new observation frame, and the cycle repeats.

Information flows as follows: human players → Think-Aloud recording → synchronized (o_t, r_t, a_t) triples → autoregressive pre-training with decaying loss → post-training with rejection-filtered reasoning and action augmentation → deployment as a ReAct loop with dual-tier memory.

3.3 Roadmap for the Deep Dive

• First, the unified action space definition (Section 2.1) — the three primitive actions, why they are chosen, and what makes them "unified" — because this is the architectural foundation that everything else depends on.
• Second, the Think-Aloud data collection pipeline (Section 2.2) — how raw human gameplay is converted into causally-aligned training sequences with sparse reasoning — since data quality determines pre-training quality.
• Third, the decaying continual loss function (Section 2.3) — the mathematical form, the motivation (causal confusion from action repetition), and the empirical validation — because this is the key optimization innovation that makes pre-training work.
• Fourth, the post-training strategies (Sections 3.1–3.5) — instruction following, multimodal prompts, sparse thinking via rejection fine-tuning, long episodic memory compression, and cross-source trajectory mixing — since these transform the pre-trained foundation into a deployable agent.
• Fifth, the sparse thinking mechanism specifically (Section 3.3) — how the model learns to decide when to reason and when to act reflexively, and the rejection sampling procedure that enforces causal thought-action chains — because this is the paper's solution to the reasoning-vs-latency tradeoff.
• Sixth, the memory architecture (Section 3.4) — the two-tier design and the 128k-token context window training — since long-horizon tasks require information beyond the immediate visual frame.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical scaling paper whose core idea is that a unified device-level action space (keyboard and mouse primitives), when combined with sufficiently large-scale cross-game pre-training and a sparse reasoning mechanism, produces a generalist agent that transfers sensorimotor skills to entirely unseen environments.

---

The Unified Device-Level Action Space: Definition and Rationale

What it is. The agent's output space consists of exactly three primitive action types, each parameterized by simple numerical or string arguments:

• mouseMove(dx, dy): Move the mouse cursor by dx pixels horizontally and dy pixels vertically, relative to its current position. dx and dy are integers that can be positive (right/down) or negative (left/up).
• mouseClick(button): Press and release a mouse button. The button parameter takes values from {left, right, middle}.
• keyPress(keys): Press and release one or more keyboard keys simultaneously. The keys parameter is a string that can be a single key (e.g., keyPress(w)) or a combination (e.g., keyPress(ctrl, c)).

Two supplementary actions complete the set: Think (explicit reasoning without taking an action), and No-Op (doing nothing at a timestep, representing waiting or idle periods). Table 1 in the paper enumerates the full taxonomy.

Why these three primitives. The design principle is universality through minimalism. Every computer interaction, regardless of application, OS, or game engine, ultimately reduces to sequences of these three physical operations. Clicking a button in a web browser is mouseMove(to button coordinates) + mouseClick(left). Crafting an item in Minecraft is a sequence of keyPress(e) (open inventory), mouseMove (to crafting grid), mouseClick(left) (place ingredients), mouseMove (to result slot), mouseClick(left) (take result). Shooting an enemy in an FPS is mouseMove (aim) + mouseClick(left) (fire). The paper argues that this reduction to physical primitives is not just theoretically elegant but practically enabling because it means:

1. Training data can be collected from any game without engineering. A human plays the game normally; the recording system captures keyboard events, mouse events, and screen frames. The raw recording IS the training data format, modulo alignment and reasoning annotation. No action-space definition, API wrapper, or GUI element detector is needed per game.

2. The action space has identical semantics across all environments. keyPress(w) means "the W key goes down and up" regardless of whether the game maps W to "walk forward," "accelerate," "jump," or any other function. The agent learns the environment-specific mapping from observation-action-outcome sequences, exactly as a human would. This is what the paper calls the Human-Native Interaction paradigm: the agent's interface to the computer is identical to a human's interface.

3. Actions are inherently time-aligned. Each action primitive corresponds to a discrete physical input at a specific moment. There is no need to model action durations separately because the agent's action frequency (typically 50–100ms per action, as stated in Section 2.2) is high enough that continuous behaviors (holding a key down, sweeping the mouse) decompose naturally into sequences of discrete primitives. The agent autonomously learns the timing and rhythm of its operations through exposure to human demonstrations.

Key design decision: relative mouse movement, not absolute. The mouseMove action takes relative displacements (dx, dy), not absolute screen coordinates. This is a subtle but critical choice. Absolute coordinates would be specific to the screen resolution, window position, and OS configuration — a 1920×1080 fullscreen game has different absolute coordinates than the same game running in a 1280×720 window. By using relative displacements, the action space is resolution-independent. The agent must learn to adjust its movement magnitude based on visual feedback (how far the cursor actually moved on screen) rather than relying on fixed coordinate transforms. The paper addresses this explicitly in Section 3.2 under "Adaptive Calibration of Continuous Actions," where random scaling factors are applied to mouse movements during training to force the agent to infer the correct movement-to-screen mapping from context.

Key design decision: no "hold key" or "release key" actions. The action space has keyPress (press and release) but no separate keyDown and keyUp events. This simplifies the action space but requires the agent to emit repeated keyPress actions when a key must be held (e.g., holding W to walk forward continuously). The paper's data collection pipeline handles this by recording at 50–100ms intervals, so a held key appears as a sequence of repeated keyPress actions across consecutive frames. The decaying loss (Section 2.3) is specifically designed to handle the training challenges that arise from these repetitive action sequences.

Supplementary actions: Think and No-Op. The Think action is a special token that signals the model should produce reasoning text without emitting a physical action. This is how the Sparse Thinking mechanism controls reasoning density: the model learns to emit a Think token at critical decision points (producing a paragraph of reasoning followed by an action) and to skip the Think token during routine execution (producing only actions). The No-Op action represents periods where the human demonstrator was idle — not pressing any keys, not moving the mouse. Including No-Ops in the training data is important because waiting is a legitimate behavior (e.g., waiting for an enemy to appear, waiting for a crafting operation to complete). The paper explicitly contrasts this with VPT (Baker et al., 2022), which "remove all no-ops from the trajectory; however, this disrupts the complete action distribution, as waiting is necessary in some scenarios" (Section 2.3).

---

The Think-Aloud Data Collection Pipeline

Why offline ReAct synthesis fails. Prior methods for creating ReAct training data typically take existing action-only trajectories and use an LLM to retroactively insert reasoning steps — "given this sequence of observations and actions, what was the agent thinking?" The paper identifies two critical failures of this approach (Section 2.2):

1. Retrospective reconstruction of real-time thought processes is nearly impossible. A human looking at a recorded action sequence cannot reliably infer what the original player was thinking at each moment — especially the uncertainty, the false starts, the corrections, and the moment-by-moment situational assessment that drove each decision. Offline-synthesized reasoning tends to be post-hoc rationalization that sounds plausible but is causally inaccurate.

2. Forcing reasoning at every timestep creates data that doesn't match inference-time behavior. When the action frequency is 50–100ms (10–20 actions per second), a human's cognitive state does not change meaningfully between consecutive frames. Generating a full reasoning paragraph for every single action creates training data where reasoning is decoupled from action — the model sees reasoning text associated with routine, low-entropy actions (like holding W to walk) that require no deliberation. This decoupling weakens the learned association between thought and action.

The Think-Aloud protocol. The paper's solution, inspired by Deitke et al. (2024), is to capture reasoning during gameplay rather than synthesizing it afterward. Human annotators play games while continuously verbalizing their thought process through a microphone. The instructions to annotators specify "at least three key moments per minute" where they explicitly narrate their reasoning — planning, situational assessment, error detection, strategy shifts. The rest of the time, they play silently, and the recording captures only the raw actions.

The recording system captures three parallel streams: (1) screen frames at regular intervals (as observations), (2) low-level mouse and keyboard input signals (as actions), and (3) audio from the microphone (as raw reasoning). These three streams are timestamped independently but recorded on the same machine.

Audio processing. The raw audio stream is transcribed to text using an ASR model (the paper cites Radford et al., 2023, which is Whisper). The transcribed text is then "refined using a Large Language Model (LLM) to remove noise and enhance logical consistency" (Section 2.2). This LLM post-processing step serves multiple purposes: it removes filler words and false starts ("um, I should... wait, no, I need to..."), it consolidates fragmented thoughts into coherent paragraphs, and it ensures the reasoning text follows a consistent format suitable for training. The paper does not specify which LLM is used for this refinement step.

The temporal alignment problem — why timestamps aren't enough. This is one of the most technically interesting parts of the pipeline. The paper describes a fundamental engineering challenge: "due to minor latencies in the recording system, relying solely on timestamps causes action signals to lag behind their corresponding screen frames" (Section 2.2). In other words, when you record screen, keyboard, and mouse simultaneously, the hardware and software pipeline introduces small but systematic delays — the keyboard event might be recorded 20–50ms after the frame where the key was actually pressed, because of USB polling rates, OS event queuing, or recorder thread scheduling.

If you naively align by timestamp, you get data where the action appears AFTER the frame where it logically occurred — for example, frame 100 shows the character starting to move, but the keyPress(w) action is timestamped at frame 102. This is a causal violation: the observation reflects the effect of the action before the action is recorded. The paper identifies a severe consequence of training on such misaligned data: "it can lead the agent model to degenerate into an Inverse Dynamics Model (IDM) — where the model learns to infer actions from state differences rather than deciding based on the current state" (Section 2.2).

To understand why this matters: an inverse dynamics model learns P(a_t | o_t, o_{t+1}) — "given the current frame and the next frame, what action must have occurred?" This is the wrong objective for an agent. It learns to recognize what actions produce what visual changes, but it does NOT learn to decide what action to take based on the current situation. An IDM-trained agent would look at the current frame, see that it needs to move forward to reach a target, but be unable to produce the forward action because the model never learned to ask "what should I do now?" — it only learned "what must have happened between these two frames?"

Visual-anchor cursor alignment. The paper's solution is elegant: use the on-screen mouse cursor as a calibration target. The cursor is the only visual element whose position the recording system knows exactly — the mouse driver reports (x, y) coordinates that directly correspond to the cursor's pixel position on screen. The alignment procedure works as follows:

1. Run a visual grounding model (the paper cites UI-TARS-2, Wang et al., 2025a) on each recorded frame to detect the pixel position of the mouse cursor.
2. Compare the frame-to-frame changes in detected cursor position with the recorded mouse movement events (which report dx, dy values).
3. Compute the temporal offset that maximizes the correlation between visual cursor displacement and recorded mouse displacement — this is the systematic timestamp bias.
4. Shift all action timestamps by this offset to align them with their causally correct observation frames.

The paper states this method "completely corrects the timestamp bias and restores the causal chain." This is a critical quality-control step because it ensures that the training data has the correct causal structure: observation_t → thought_t → action_t → observation_{t+1}, not the corrupted observation_t → observation_{t+1} → action_t.

Sparse reasoning construction. After alignment, the pipeline constructs training sequences where reasoning steps appear only at a subset of timesteps. The formal definition (Equation 3) is:

$\tau = {(r_0, a_0, o_0), \ldots, (a_n, o_n), \ldots, (r_m, a_m, o_m), \ldots, (r_T, a_T, o_T)}$

where r_t denotes a reasoning step, a_t denotes an action, and o_t denotes an observation. Some timesteps m contain the complete (r, a, o) triple, while other timesteps n contain only (a, o) pairs with no reasoning. The notation τ represents a full trajectory of length T, and the ellipses \ldots indicate that reasoning steps are sparsely distributed throughout the sequence.

What this equation encodes: It specifies that a trajectory is a time-ordered sequence where each index t has an observation o_t and an action a_t, but only some indices also have a reasoning step r_t. The trajectory must contain reasoning at the first step (t=0, for initial planning) and the last step (t=T, for final summarization), but between these endpoints, reasoning appears at sparse, irregular intervals determined by when the human annotator chose to verbalize their thoughts.

Why this sparse format is necessary: If every timestep had reasoning, the model would learn to generate reasoning that is decoupled from action (producing text for routine frames where no deliberation occurred) and would waste inference-time computation generating unnecessary reasoning tokens. The sparse format forces the model to learn when reasoning is beneficial and when it can act directly from perception.

Thought timestamp relocation. The final processing step is a heuristic optimization: "intelligently adjusting the timestamp of a thought to align with the subsequent action to which it is most semantically relevant" (Section 2.2). Human verbalization has inherent latency — you think "I need to jump over that gap" and then, maybe 100–300ms later, you press the spacebar. The recorded thought timestamp reflects when you SAID the thought, not when you HAD the thought. The relocation strategy shifts the thought forward in the recorded timeline to be immediately before the action it motivates, strengthening the causal association between reasoning and action in the training data.

Scale of collected data. The data collection produced three types of trajectory data at different scales (from Section 4.1):

• Pure action game trajectories: 61B tokens across 50 open-world games. These are trajectories without any reasoning — just observation-action pairs — used for initial pre-training.
• Offline-synthesized ReAct trajectories: 45B tokens across 50 games. These use the retrospective LLM-based reasoning synthesis that the paper later acknowledges is inferior, but were included in the blended pre-training.
• Online human Think-Aloud ReAct trajectories: 127B tokens across 500 games. This is the high-quality data from the pipeline described above, with games "selected with a focus on reasoning" and "enhanced diversity."

Together, the game-specific data totals approximately 233B tokens. The remaining pre-training tokens come from general multimodal data (200B, including VQA, captioning, video understanding) and cross-domain agent trajectories (20B+).

---

The Decaying Continual Loss Function

The problem: action repetition dominates training data. In game trajectories, the most common action is to continue doing what you were already doing. When a player walks forward, they hold the W key for seconds at a time, producing dozens of consecutive identical keyPress(w) actions. When they craft items, they may spend hundreds of frames navigating menus with repetitive mouse movements. The paper states that "temporally persistent actions, where a_t = a_{t-1}, constitute the vast majority (over 90%) of the action sequence" (Section 2.3). The exact percentage varies by game type — an FPS has more action switches than a crafting game — but the dominance of repetition is universal.

Standard cross-entropy loss treats every action prediction equally. When 90% of the training examples are "do the same thing as last time," the model learns a degenerate strategy: predict the previous action regardless of the current observation. This strategy achieves 90% accuracy (matching the data distribution) but fails catastrophically when an action change IS needed — it never learns to recognize the visual cues that signal when to stop walking, when to switch weapons, or when to evade.

The paper's diagnosis: causal confusion. The paper explicitly invokes the causal confusion framework from De Haan et al. (2019): in imitation learning, when a policy observes both the true causal state (the current game frame) and a spurious correlate (the previous action), it will preferentially learn from the correlate because it's a simpler predictor. The model ends up as a sophisticated "action copier" with no real understanding of when actions should change.

The decaying loss mechanism. The solution is to down-weight the loss contribution from repeated actions, making the model pay disproportionately more attention to action transitions (where a_t ≠ a_{t-1}). The formal definition is:

$\mathcal{L} = -\sum_{t=1}^{T} \omega_t \log P(a_t \mid H_t)$

$\omega_t = \gamma^{k_t - 1}$

where T is the trajectory length (total number of timesteps), ω_t is the per-timestep loss weight (a scalar between 0 and 1), H_t is the full history of all preceding observations and actions up to timestep t, P(a_t | H_t) is the model's predicted probability of the ground-truth action, k_t is the number of consecutive timesteps including t where the action has remained unchanged (i.e., how long the current action "streak" has persisted), γ ∈ (0, 1) is a decay factor hyperparameter (set to γ = 1/2 in all experiments), and the exponent k_t - 1 ensures that the first occurrence of any NEW action gets weight 1 (since γ^0 = 1).

What this computes: For each timestep, the standard cross-entropy log P(a_t | H_t) is multiplied by a weight ω_t that decays exponentially with the length of the current action repetition streak. The first time an action appears (e.g., switching from walking to jumping), k_t = 1, so ω_t = γ^0 = 1 — full weight. The second consecutive identical action gets ω_t = γ^1 = 1/2. The third gets ω_t = γ^2 = 1/4. The eighth gets ω_t = γ^7 = 1/128 — essentially zero weight. The sum over t from 1 to T produces the total weighted loss for the trajectory, which is then differentiated for gradient computation.

In operational English: the model is trained to predict every action, but repeated actions contribute exponentially less to the gradient. Action transitions — the moments when the agent decides to do something different — receive full learning signal. Repetitive continuations receive progressively vanishing signal. The model can still learn to continue actions (the prediction is still made, just with lower weight), but its optimization focus shifts toward recognizing when and why actions should change.

Why this exponential form specifically: The paper considered but rejected several alternatives:

• Removing all repeated actions (VPT approach): This would destroy information about action persistence — the model wouldn't learn that walking forward requires continuous keyPress(w) events. Some repetition structure is important for learning timing and rhythm.

• Binary weighting (1 for transitions, 0 for continuations): This would completely eliminate learning on continuation actions, making the model unable to predict action durations or learn continuous behaviors like holding W to walk.

• Linear decay instead of exponential: Linear decay would still give substantial weight to long repetition streaks (e.g., the 100th consecutive W press would get weight 1/100, which is small but cumulative across many such streaks). The exponential decay provides a cleaner separation — after ~10 consecutive repeats, the weight is effectively zero (1/1024), so the model allocates essentially no capacity to learning these frames.

The hyperparameter γ = 1/2 means the half-life of repetition weight is one step — each consecutive repeat halves the contribution. This is an aggressive decay that strongly prioritizes action transitions. The paper's ablation experiments (Table 5, Section 4.3.3) validate this choice: the decaying loss dramatically improves "non-repetitive accuracy" (action prediction accuracy on frames where the action differs from the previous frame) from 12% to 39%, even though overall accuracy drops from 59% to 47%. The drop in overall accuracy is expected and benign — it reflects the model no longer "cheating" by copying the previous action. The gain in non-repetitive accuracy is the meaningful metric because it measures genuine decision-making.

The implicit design philosophy. The decaying loss is not just a trick to handle imbalanced data — it encodes a hypothesis about what makes good game-playing behavior. The hypothesis is that the difficult, high-entropy decisions are the action transitions, and that a model that masters these transition points will produce fluid, adaptive behavior even if it occasionally fumbles routine continuations (which are easy to recover from). This aligns with how humans think about skill: the mark of expertise is not holding W to walk in a straight line, but knowing when to stop walking, when to dodge, and when to change strategy.

Empirical validation (Table 5, Section 4.3.3). The paper reports ablation results comparing training with and without the decaying loss. Key findings:

• Online success rates: Combat tasks improve from 14.54% to 22.22% (a ~53% relative improvement). Embodied tasks improve from 44.90% to 46.15% (modest). GUI tasks improve from 37.75% to 42.26%. Combat sees the largest gain because combat requires frequent, rapid action changes (shoot, dodge, reload, switch weapons) — exactly the transitions that the decaying loss emphasizes.

• Trajectory efficiency: Agents trained with decaying loss produce substantially shorter trajectories (fewer average steps to completion), indicating they are "less likely to become stuck or generate redundant action loops."

• Behavioral diversity: Diversity (measured as the percentage of unique action sequences across runs) increases from 60–80% to 90–100%, suggesting the decaying loss prevents the model from collapsing to a small set of high-frequency action patterns.

• Offline prediction: The tradeoff between overall accuracy (59% → 47%) and non-repetitive accuracy (12% → 39%) confirms that the standard loss exploits dataset bias by learning to copy the previous action, while the decaying loss forces genuine state-dependent prediction.

---

Automatic Action Space Augmentation for Instruction Following

The prior distribution problem. After pre-training on data from hundreds of games, the model has learned strong priors about key mappings. In most games, W means "move forward," Space means "jump," E means "interact," and so on. These priors are useful for generalization (they encode the conventions of game design) but create a problem: the model may ignore explicit instructions in the System Prompt and instead rely on its memorized key mappings.

The paper gives a concrete example: "Even if the System Prompt specifies the UpArrow as the jump key, the model might still output Space to perform a jump out of habit" (Section 3.1). This is behavioral inertia — the model minimizes loss by producing the most common action from its pre-training distribution rather than conditioning on the instruction.

A second, related problem is action space violation: the model might output actions that don't exist in the current environment. If the pre-training data never used the F1–F12 keys, the model may have extremely low probability for these keys even when the System Prompt says they're valid.

The augmentation mechanism. The solution is deceptively simple: during post-training, randomly rewrite the key mappings in the training data while preserving the semantic meaning. For a trajectory where the human pressed W to walk forward, the system might randomly replace W with X, and simultaneously update the System Prompt to say "X = move forward." The trajectory's action sequence now contains keyPress(x) instead of keyPress(w), and the prompt explicitly defines X as the forward key.

This forces the model into a specific learning mode: it cannot rely on memorized key meanings because those meanings have been deliberately scrambled. The only way to correctly predict the action sequence is to read the System Prompt, parse the current key mapping definitions, and use those definitions at each decision point. This is precisely the behavior needed for generalization — the model must learn to treat key mappings as variables that are resolved at inference time from the prompt context, not as constants that are baked into the weights.

Why this works at a mechanistic level. The neural network learns two things simultaneously: (1) what semantic actions are needed in a given situation (e.g., "I need to move forward to reach the tree"), and (2) what physical key to press given the current mapping (e.g., "the prompt says X is forward, so press X"). The augmentation ensures these two learning objectives are disentangled in training. The first (semantic reasoning) is stable across augmentations — the agent always needs to move toward the tree regardless of what key is bound to forward. The second (key translation) varies randomly, forcing the model to develop a generic "look up the key for the semantic action in the prompt" capability that works for any mapping.

Why standard fine-tuning without augmentation fails. Without augmentation, the model can learn a simpler strategy: memorize that "walk forward = W" and ignore the prompt entirely. This strategy works on the training distribution (where W IS forward) but fails on any environment with different mappings. The paper frames this in terms of the loss landscape: the un-augmented objective has a deep local minimum at "ignore prompt, use memorized mappings" that the model readily finds, while the augmented objective makes that minimum unreachable (because the memorized mapping is wrong half the time), forcing the model to climb toward the prompt-conditioning minimum instead.

Implementation detail: what is replaced. The paper specifies that "we preserve the semantics of actions in the original trajectory data but randomly replace their corresponding key bindings." This means the augmentation operates at the level of individual keys, not semantic actions. The trajectory's sequence of semantic actions (move forward, move forward, jump, interact, move forward) is preserved; only the physical keys bound to those semantics are randomly reassigned. The System Prompt is updated to reflect the new bindings.

The inverse-dynamics auxiliary task. As a complementary mechanism, the paper introduces an auxiliary training objective: given the current observation, the next observation, and the action space definition, predict what action must have occurred between them. Formally:

$a_t \sim P(\cdot \mid A, [o_i, a_i]_{i=1}^{t-1}, o_t, o_{t+1})$

where A is the action space definition provided in natural language (the System Prompt's key binding descriptions), [o_i, a_i]_{i=1}^{t-1} is the history of observations and actions up to the previous timestep, and (o_t, o_{t+1}) is the pair of consecutive observations.

What this task teaches. Unlike the main imitation objective (which asks "given current state, what action should I take?"), this auxiliary objective asks "given current state and next state, what action must have caused this transition?" This is exactly the inverse dynamics model (IDM) objective that the paper earlier warned against when it arises from data misalignment. But here, used deliberately as an auxiliary task, it serves a different purpose: it teaches the model to understand the causal effects of actions. By learning to recognize that keyPress(space) causes the character to rise (jump), and that mouseMove(100, 0) causes the view to pan right, the model builds a causal model of action effects that complements the forward decision-making policy.

The training data for this auxiliary task is constructed naturally from the same trajectories: for any consecutive pair of observations (o_t, o_{t+1}), the recorded action a_t is the target, and the model must predict it given the state transition. The final training objective combines the forward policy prediction and this inverse prediction, both conditioned on the system prompt instruction I:

$P(a_t \mid I, A, o_t)$

where the instruction I describes the task goal and A defines the available actions and their key bindings.

---

Sparse Thinking via Rejection Fine-Tuning

The problem with uncurated reasoning data. The Think-Aloud pipeline produces reasoning text that is temporally aligned with actions but not necessarily causally useful. A human might say "I need to be careful here" while pressing W to walk — the reasoning is vaguely related to the situation but does not directly cause or explain the specific action of walking forward. More problematically, some annotators paused their gameplay while verbalizing, resulting in reasoning text associated with No-Op actions. When the model is trained on such data, it learns that reasoning can be arbitrary commentary rather than action-driving deliberation.

Step 1: Identifying which steps need reasoning. The paper uses a necessity criterion: a step needs reasoning if the model cannot predict the correct action without it. The procedure:

1. Take the pre-trained Game-TARS model and strip its reasoning capability by prompting it to produce actions only (π_action — a variant that predicts a_t from (o_t, history) without any intermediate reasoning tokens).
2. Run this action-only model on the offline trajectory data.
3. For each timestep t, compare the model's predicted action with the ground-truth action a_t^gt.
4. If the prediction matches, the step does NOT need reasoning — a reactive policy suffices.
5. If the prediction fails, the step DOES need reasoning — the correct action requires deliberation beyond what the reactive policy can provide.

Formally, the set of reasoning-requiring timesteps S_r is:

$S_r = \{t \in \{1, \ldots, T\} \mid \mathbb{I}(\pi_{\text{action}}(a \mid o_t, (o_i, a_i)_{i=0}^{t-1}) = a_t^{\text{gt}}) = 0\}$

where \mathbb{I}(·) is the indicator function (returns 1 if the condition is true, 0 if false), π_action is the action-only policy, o_t is the current observation, (o_i, a_i)_{i=0}^{t-1} is the history, and a_t^gt is the ground-truth action from the human demonstration. The condition = a_t^gt checks whether the predicted action matches the ground truth, and the indicator returns 1 only when they match. S_r is therefore the set of all timesteps where the action-only model FAILED to predict the correct action — these are the steps that require reasoning augmentation.

What this filter accomplishes. It distinguishes between steps where behavior is routine (the model's learned reactive policy already produces the right action) and steps where deliberation is needed (the reactive policy fails, suggesting the human demonstrator was making a non-obvious decision). This is a computationally efficient way to identify decision boundaries in the demonstration data without requiring manual annotation of "important moments."

Step 2: Rejection sampling for causally valid reasoning. For each timestep in S_r, the procedure generates candidate reasoning and tests whether it leads to the correct action:

1. Take the full Game-TARS model (with reasoning capability) and generate a candidate thought r_t^cand conditioned on the observation o_t.
2. Feed this candidate thought back into the model and generate a predicted action a_t^pred based on both the observation AND the candidate thought.
3. Compare the predicted action with the ground-truth action a_t^gt.
4. Accept the candidate thought ONLY if the predicted action matches the ground truth.

Formally:

$\text{Accept } r_t^{\text{cand}} \text{ iff } \mathbb{I}(\pi(a_t \mid o_t, r_t^{\text{cand}}) = a_t^{\text{gt}}) = 1$

where π is the full model with reasoning, r_t^cand is the candidate thought, a_t^gt is the ground-truth action, and \mathbb{I} is the indicator. The acceptance condition is: the model, when given the candidate thought, produces exactly the ground-truth action.

What this filter accomplishes. This is the critical quality control step. It eliminates reasoning that, while perhaps logically sound, is not causally connected to the action that follows. For example, a candidate thought might say "I should explore the cave to find iron ore" but the ground-truth action is keyPress(w) (walk toward a tree). This thought might be broadly correct (the player's overall goal IS to find iron) but it doesn't explain WHY the next action is to walk toward a tree specifically. The rejection filter discards such thoughts, retaining only those that directly motivate the immediate action.

The paper frames this as enforcing a thought-action causal chain: "Through this method, we filter out a significant amount of abstract reasoning that is irrelevant to the actual behavior, retaining only those thought chains that directly lead to correct actions" (Section 3.3).

Post-processing after rejection sampling. The filtered data undergoes two additional refinement steps:

1. Thought consolidation: For consecutive identical actions with similar reasoning content, merge the thoughts into a single instance at the first occurrence. This reduces redundancy and prevents the model from learning to repeat the same reasoning across many frames.

2. LLM rewriting and density control: Use an LLM to rewrite the accepted reasoning texts for conciseness and clarity. Simultaneously, control the overall density of reasoning in the trajectory — the proportion of steps that contain reasoning — to match "a natural human cognitive rhythm." The paper doesn't specify the exact target density, but the implication from the Sparse Thinking design is that reasoning should appear at a small fraction of total steps (likely 5–20%, corresponding to the "at least three key moments per minute" from the data collection instructions).

Why reinforcement fine-tuning (RFT) rather than standard SFT. The paper uses the term "Rejection Fine-Tuning" to emphasize that the training data is generated by the model itself (self-generated reasoning) and filtered by an outcome-based criterion (does this reasoning lead to the correct action?). This is a form of reinforcement learning without explicit reward modeling: the model proposes candidate thoughts, and the environment (ground-truth action matching) provides binary feedback. Only successful proposals are used for further training. This creates an iterative improvement loop where the model's reasoning becomes progressively more action-relevant over successive rounds of generation and filtering.

The outcome: task-adaptive reasoning depth. The qualitative analysis in Section 4.3.2 reveals that the trained agent develops an implicit meta-cognition capability: it chooses when to reason and when to act reflexively based on task demands. In Minecraft crafting sequences, the agent "reasons carefully at the start of a task (to identify needed resources and steps) but executes subsequent mining or crafting actions without redundant re-planning." In 2048 (a combinatorial puzzle game), "it reasons at every move due to the inherent combinatorial structure." In VizDoom (a fast-paced FPS), "it minimizes reasoning to maintain responsiveness."

This emergent behavior is not explicitly programmed — it arises from the interaction between the sparse reasoning training data (which shows reasoning only at critical junctures) and the rejection fine-tuning (which filters out non-action-driving reasoning). The model learns that reasoning is a tool to be deployed when needed, not a requirement for every action.

---

The Dual-Tier Memory Architecture

The fundamental constraint: context window limits temporal horizon. Vision transformers produce large numbers of tokens per image — the paper cites image token counts in the billions across the full training corpus (Table 2: 208B image tokens in pre-training, 10B in post-training). At inference time, storing every observation frame in the model's context window would exhaust the available tokens quickly, limiting the agent to a short interaction memory (perhaps 30–60 seconds of gameplay). Tasks like crafting a complex item in Minecraft or tracking a fox across a forest (as shown in Figure 13) require remembering events from minutes ago.

Tier 1: Short-term contextual memory (the "working memory"). This tier is stored directly in the model's context window as an interleaved sequence of observations, thoughts, and actions in chronological order. It operates as a sliding-window queue: the most recent M steps are retained, and older steps are evicted. The standard context window is 32k tokens, which in the pre-training stage supports up to 80 image frames (Table 2: "Maximum Image Sequence Length 80" in pre-training). During post-training, this is extended to 300 frames with a 128k-token context window ("Maximum Image Sequence Length 300" and "Maximum Token Sequence Length 128,000" in post-training).

This memory contains the full, high-fidelity record of recent interactions — every pixel of every recent frame, every action taken, every thought produced. It is the agent's immediate awareness of what is currently happening.

Tier 2: Long-term summary memory (compressed history). When information is evicted from the short-term context window (because newer observations need the space), it is NOT discarded entirely. Instead, it is compressed and stored in a separate memory tier that is also maintained in the context, but at vastly lower token cost.

The compression mechanism leverages the sparse thinking output: each evicted step retains its thought text but discards its visual observation. Since thoughts are sparse (only present at a fraction of steps) and much shorter than encoded images, this achieves an extremely high compression ratio. The paper states this "allows the agent to retain a long-term task context of over 2000 interaction steps at an extremely low token cost" (Section 3.4).

The two tiers together form a structured memory hierarchy: the recent past is stored in full detail (visual + action + thought), while the distant past is stored only as a compressed narrative (thought summaries of key events). This mirrors human episodic memory, where recent events are remembered vividly and older events are remembered as semantic summaries.

Consistency across training and inference. The paper emphasizes that this memory architecture is "consistent during both training and inference" (Section 3.4). This is important because many agent architectures use different memory mechanisms at training time (where full trajectories are available) versus inference time (where memory must be accumulated online). If the memory representation differs between training and deployment, the model may learn to rely on information that is available during training but absent during inference, causing a train-test mismatch. By using the same two-tier sliding window during both phases, Game-TARS ensures that what the model learns to use during training is exactly what will be available during deployment.

Ultra-long context training. To ensure the model can effectively use the long-term memory, the training data includes "ultra-long sequences with context window lengths reaching up to 128k tokens" and trajectories "exceeding 240 steps" (longer than the 80-frame pre-training maximum and the 300-frame post-training maximum cited in Table 2 — suggesting these ultra-long sequences may be special curriculum examples rather than standard training instances). These sequences force the model to learn to retrieve and reason over information that was observed many steps ago, developing the capability to maintain coherent goals and plans over extended time horizons.

Memory gating via sparse thought. There is a subtle synergy between the sparse thinking mechanism and the memory architecture: only steps that contain reasoning contribute to the long-term summary memory (since the compression discards observations and keeps only thoughts). This means the agent implicitly controls what goes into long-term memory by choosing when to produce a thought. If a step is routine and produces no thought, it will be evicted without a trace when it leaves the short-term window. If a step is a critical decision point and produces a thought, that thought will persist in long-term memory and remain accessible for future retrieval. This creates a natural gating mechanism where only notable events are remembered — again mirroring human memory, where routine actions are forgotten and significant decisions are retained.

---

Cross-Source Agentic Trajectory Post-Training

The layered skill acquisition hypothesis. Section 3.5 articulates a specific theory of how generalist agent capabilities should be acquired: game pre-training provides the sensorimotor foundation (rapid perception-to-action mapping, fluid mouse/keyboard control, real-time responsiveness), while cross-domain post-training builds higher-level cognitive capabilities (structured reasoning, multi-step planning, information synthesis, self-reflection) on top of that foundation.

This is not presented as a proven theory but as a design rationale for the post-training data mixture. The idea is that gaming creates an ideal training environment for sensorimotor skills because games demand fast, accurate, physically grounded actions with immediate feedback, while non-gaming agent trajectories (code, GUI, research) provide richer examples of abstract reasoning and long-horizon planning that are harder to find in game data alone.

Three categories of cross-source data:

1. Code Agent Trajectories: Data from agents performing software engineering tasks — understanding a coding problem, decomposing it into subproblems, writing code, testing and debugging. The paper hypothesizes this strengthens "structured thinking and long-range logical reasoning abilities" (Section 3.5). The specific datasets are not enumerated in detail, but citations point to SWE-bench (Jimenez et al., 2023), SWE-agent (Yang et al., 2024), and related work.

2. GUI Agent Trajectories: Data from agents completing tasks in desktop and web applications — booking flights, managing files, filling forms. The paper hypothesizes this "strengthens the agent's ability to understand and execute multi-step instructions in non-gaming environments." This is particularly relevant because GUI tasks share the keyboard-mouse action space but operate in more structured, information-dense interfaces than games. Citations include UI-TARS (Qin et al., 2025), OSWorld (Xie et al., 2024), and BrowseComp (Wei et al., 2025).

3. MCP Agent Trajectories: Data from agents using the Model Context Protocol (Anthropic, 2024) to perform research tasks — information retrieval, document reading, content synthesis, summarization. The paper hypothesizes this enhances "the agent's ability to plan, reflect, and integrate knowledge within the ReAct loop." Citations include deep research systems (OpenAI, 2025b), RFT-based reasoning models (ByteDance Seed, 2025a), and tool-use RL systems (Feng et al., 2025).

Integration with game data. The cross-source data is co-trained with game data during the post-training phase, not trained sequentially. The paper hypothesizes a "synergistic learning effect: game data solidifies the foundational interaction skills, while cross-domain agentic data builds more abstract and general problem-solving capabilities on top of this foundation" (Section 3.5). This implies a specific causal model: the shared model weights enable bidirectional transfer — improvements in abstract reasoning from code trajectories benefit game performance, and improvements in real-time perception from game trajectories benefit GUI task performance.

The scaling experiments in Figure 10 (right panel) provide partial evidence: adding multimodal data and cross-domain agent trajectories to the training mix increases Minecraft success rate from 65% (game-only) to 72% (game + multimodal + cross-domain). This is a nontrivial gain (+7 percentage points) from data that has nothing to do with Minecraft, supporting the claim that general cognitive capabilities learned from diverse tasks transfer to game performance.

Limitation acknowledged. The paper does not claim to have fully realized the generalist computer-use agent vision. The post-training with cross-source data is described as part of the process of "cultivating a more comprehensive generalist agent" (Section 3.5), implying it is work in progress rather than a completed system. The evaluation remains gaming-focused, and there is no reported performance on pure GUI or code benchmarks that would directly validate the cross-domain transfer claims. This is flagged as future work implicitly by the framing of the paper as establishing a foundation rather than delivering a final product.

4. Key Insights and Innovations

Innovation 1: The Action Space as a First-Class Architectural Primitive, Not an Implementation Detail

The most fundamental conceptual contribution of this paper is elevating the action space from a per-environment engineering decision to the central architectural choice around which the entire training paradigm is organized. Before Game-TARS, the field treated action spaces as something you design after deciding on your model architecture, training objective, and data strategy. Voyager (Wang et al., 2023a) used high-level Minecraft APIs because that was the most convenient way to get strong results on Minecraft. VPT (Baker et al., 2022) used keyboard-mouse inputs but confined training to a single game because there was no framework for cross-game data aggregation. Gato (Reed et al., 2022) tokenized actions differently per domain because the Transformer architecture was flexible enough to handle heterogeneous outputs. In every case, the action space was subordinate to other concerns — chosen to fit the task, the available data, or the model architecture.

Game-TARS inverts this priority. The paper argues — and provides the first large-scale empirical evidence — that the action space is not merely a convenience or a constraint, but the primary determinant of scaling behavior. A unified device-level action space enables cross-game data aggregation, which enables the kind of scaling that has transformed language and vision but has been absent from game AI. A fragmented action space, however well-engineered per environment, imposes a scaling ceiling that no amount of model capacity or training data within a single game can overcome. This is not a claim about what works best today but about what works in the limit of scale — and the paper explicitly invokes the Bitter Lesson (Sutton, 2019) to frame this as a bet on general methods over domain-specific engineering.

What makes this insight distinctive is not the keyboard-mouse action space itself — SIMA (Raad et al., 2024) had the same basic idea, and VPT used it within Minecraft — but the paper's systematic demonstration that the scaling properties of different action spaces are qualitatively different. Figure 10 (right panel) shows this with unusual clarity: GUI-based action spaces produce better performance at small data scales (because pre-trained VLMs already understand visual grounding), but the unified device-level space overtakes them as cross-game data accumulates, and then continues improving with multimodal and cross-domain data that the GUI space cannot incorporate. This is a scaling law for action representations — the first of its kind in the agent literature — and it implies that the field's historical preference for high-level, semantically rich action spaces may have been systematically limiting progress toward generalist agents.

The paper also provides a clean diagnostic for distinguishing "convenient" from "scalable" action spaces: can data from a new game be added to the training mix without per-game engineering? For GUI-based spaces, the answer is no — each new game requires a different set of UI elements, different grounding models, different interaction patterns. For device-level spaces, the answer is yes — every interaction reduces to the same three primitives. This diagnostic is simple enough to be operationalizable by other researchers and rigorous enough to expose fundamental differences between approaches that might otherwise appear architecturally similar.

Innovation 2: Causal Confusion as the Central Learning Pathology in Imitation-Based Agents, and the Decaying Loss as a General-Purpose Remedy

The paper identifies a specific, previously underappreciated failure mode in imitation learning for interactive agents and provides a mathematically simple, empirically validated solution. The failure mode is causal confusion (De Haan et al., 2019): when action repetition dominates the training distribution (over 90% of steps), the model learns to predict the previous action rather than conditioning on the current observation, because "copy the last action" is a simpler, lower-loss strategy than genuine state-dependent decision-making. The solution is the decaying loss (Section 2.3), which exponentially down-weights the loss contribution of consecutive identical actions via the weight ω_t = γ^(k_t - 1).

Prior work recognized that action repetition was problematic but addressed it with crude heuristics. VPT removed all no-ops from trajectories entirely — which the paper correctly identifies as destroying information about action persistence and timing. Other approaches used class-balanced sampling or simple re-weighting schemes that didn't account for the temporal structure of repetition. The decaying loss is distinctive because it is temporally aware — it distinguishes between the first occurrence of an action (full weight), the second (half weight), the eighth (essentially zero weight) — and because the exponential form creates a clean separation between transitions (which receive full learning signal) and continuations (which receive vanishing signal) without completely eliminating the latter from training.

What elevates this from a useful trick to a genuine insight is the paper's experimental decomposition of where the benefit comes from. Table 5 reveals that the decaying loss actually reduces overall action prediction accuracy (from 59% to 47%) while dramatically improving non-repetitive accuracy (from 12% to 39%). This is a textbook example of Goodhart's Law in machine learning: the standard loss metric rewards models that exploit dataset bias (copying the previous action), while the re-weighted metric rewards models that learn genuine causal relationships. The paper makes this diagnosis explicit — "models trained without decaying loss tend to exploit dataset bias by repeatedly copying the previous action, 'hacking' the accuracy metric without genuinely improving decision quality" — and provides converging evidence from online task success rates (combat improves from 14.5% to 22.2%), trajectory efficiency (fewer steps to completion), and behavioral diversity (60–80% → 90–100%).

The broader implication is that causal confusion is likely pervasive in imitation-learned agents across domains — any setting where actions exhibit temporal persistence will suffer from the same pathology — and the decaying loss provides a general, hyperparameter-light remedy. The choice of γ = 1/2 (half-life of one step) is aggressive but empirically validated, suggesting that when action repetition dominates, strong interventions are necessary.

Innovation 3: Sparse Thinking as a Learned Meta-Cognitive Capability, Not a Fixed Reasoning Schedule

The paper introduces Sparse Thinking — the idea that agents should reason explicitly only at critical decision points rather than at every timestep — and provides a training methodology (Rejection Fine-Tuning, Section 3.3) that causes the agent to learn when to reason rather than following a hard-coded schedule. This is a more fundamental contribution than it might first appear because it resolves a tension that has plagued the ReAct agent literature: reasoning is beneficial for complex decisions but harmful for fast, reactive behaviors, and no fixed reasoning density is optimal across all tasks or all moments within a single task.

Prior work on reasoning-augmented agents largely fell into two camps. One camp (Wei et al., 2022; Yao et al., 2022) treated reasoning as something you add at every step — chain-of-thought before every action — which works for benchmark reasoning tasks but introduces unacceptable latency and hallucination risk in real-time environments. The other camp (Baker et al., 2022 with VPT) treated reasoning as unnecessary — pure reactive policies trained on observation-action pairs — which works for simple behaviors but fails on tasks requiring multi-step planning. The paper's ablation in Table 4 quantifies this tension precisely: in Minecraft, greedy thinking (reason at every step) achieves only 45% success rate versus 55% for no-thinking and 63% for sparse thinking; in Doom, greedy thinking drops to 7.2 reward versus 13.2 for no-thinking and 11.5 for sparse thinking. The optimal reasoning density is task-dependent, and forcing reasoning everywhere is worse than never reasoning at all in fast-paced environments.

What distinguishes Sparse Thinking from a simple "sometimes reason, sometimes don't" heuristic is that the agent learns the reasoning gating policy through its training data rather than through an explicit meta-controller. The Think-Aloud data collection captures human reasoning at natural frequencies — annotators verbalize at "key moments" but play silently most of the time — and the Rejection Fine-Tuning filters out reasoning that doesn't causally drive actions. The resulting model internalizes an implicit meta-cognition: it produces reasoning when the situation demands deliberation (planning a crafting sequence in Minecraft, evaluating a combinatorial move in 2048) and suppresses reasoning when speed matters (dodging projectiles in Doom, executing a practiced movement sequence). The paper's qualitative analysis confirms this emergent behavior: "the agent develops an implicit meta-cognition mechanism: it chooses when to reason and when to act reflexively."

This is a conceptual advance over prior sparse reasoning approaches because it treats reasoning density as a learned policy output rather than an architectural hyperparameter. The model learns to modulate its own compute budget at inference time — spending reasoning tokens only when they are likely to improve action quality — which is essentially a form of adaptive computation that emerges from the training distribution rather than being explicitly optimized. The connection to human cognition (deliberate System 2 reasoning at decision points, automatic System 1 execution during routine behavior) is noted but not over-claimed; the paper presents the mechanism as engineering rather than cognitive modeling.

Innovation 4: Empirical Demonstration That Cross-Game Sensorimotor Transfer Is Real and Quantifiable

A persistent question in the generalist agent literature — raised by Gato, SIMA, and related work — is whether training on diverse environments actually produces transferable skills or merely produces a model that has memorized environment-specific behaviors that happen to share a common output format. Game-TARS provides the strongest evidence to date that the transfer is genuine, and it does so through a specific experimental design: evaluating on games and simulators that were completely absent from the training data.

The evidence comes from three sources. First, the VizDoom results (Figure 8): Game-TARS-mini outperforms GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet by large margins (5–15× on episode rewards) on a classic FPS benchmark that was not in the pre-training distribution. General VLMs have never seen Doom-specific data either, but they lack the sensorimotor priors that Game-TARS acquired from its 500-game pre-training — priors about how mouse movements translate to view changes, how to time shots, how to strafe and dodge. The transfer is from general FPS-like gameplay (present in the 500-game corpus) to a specific FPS (absent from the corpus).

Second, the web game results (Figure 7): Game-TARS-mini achieves near-human or super-human performance on four Poki games (racing, jumping, endless running, flying) that it has never seen. The agent adapts to novel visual styles, novel control schemes, and novel game mechanics within a single episode, suggesting that the pre-training has produced generalizable visuomotor policies rather than game-specific recognition patterns.

Third, the MiniWorld results (Figure 9): a minimalist 3D simulator requiring navigation and object interaction — a domain with a visual style and task structure different from any of the pre-training games — yet the agent achieves robust performance across nearly all maps.

What makes this evidence compelling rather than merely suggestive is the scaling trend in Figure 10 (right panel). Minecraft performance improves from 42% to 65% when 200B cross-game tokens are added to the training data, and further to 72% when multimodal and cross-domain agent trajectories are included. These gains come from data that has nothing to do with Minecraft — they are pure transfer effects. If the model were merely memorizing environment-specific behaviors, adding data from other games would either have no effect or would cause catastrophic interference. Instead, it produces steady, monotonic improvement, consistent with the hypothesis that the model is learning general sensorimotor primitives that compose into game-specific behaviors.

The paper does not claim to have fully characterized what transfers or how — this remains a limitation. But the combination of zero-shot evaluation on multiple unseen environments plus a clear scaling trend with cross-game data volume establishes that cross-game sensorimotor transfer is not only possible but is the primary driver of performance gains at scale. This finding validates the central bet of the paper: that a unified action space + massive cross-game pre-training is a viable path to generalist agents.

Innovation 5: The Human-Native Data Collection Paradigm as an Alternative to Synthetic Data Generation

A less obvious but practically significant innovation is the Think-Aloud data collection pipeline (Section 2.2) and its implications for how agent training data should be produced at scale. The dominant paradigm in 2024–2025 for creating agent training data has been retrospective synthesis: take action-only trajectories (or even just task descriptions) and use a capable LLM to generate plausible reasoning text post-hoc (e.g., UI-TARS's offline ReAct synthesis). The Game-TARS paper identifies a specific failure mode of this approach — retrospective reasoning is causally inaccurate because it cannot reconstruct the annotator's real-time uncertainty, false starts, and situational awareness — and proposes an alternative: capture reasoning during task execution via think-aloud protocols, then computationally align, transcribe, and refine the resulting multimodal streams.

This is not a new idea in psychology (think-aloud protocols have been used in cognitive science for decades) or in AI data collection (Deitke et al., 2024 used similar methods for vision-language data). But the paper's contribution is engineering the pipeline for agent-specific challenges — particularly the causal alignment problem and the causal confusion it causes. The visual-anchor cursor alignment (Section 2.2) solves a real, pernicious problem: timestamp misalignment between screen recording and input logging causes actions to appear after their effects, which trains inverse dynamics models rather than forward policies. The paper provides both a diagnosis (the IDM degeneration problem) and a concrete solution (cursor tracking as a calibration signal), making the pipeline reproducible by other groups.

More broadly, the Think-Aloud paradigm represents a philosophical stance on what makes good training data for agents. Synthetic reasoning — however fluent — is disconnected from the physical, temporal, and causal structure of real interaction. It captures what an observer might infer about the player's thoughts but not the thoughts that actually drove behavior. Native reasoning — captured during task execution — preserves the temporal dynamics of deliberation, the coupling between thought and action, and the sparse, irregular distribution of cognitive effort that characterizes skilled performance. The paper's results suggest this difference matters: the model trained on 127B tokens of native Think-Aloud data (plus other data sources) substantially outperforms models trained only on offline-synthesized reasoning, though the paper does not provide a clean ablation isolating Think-Aloud from other data components. The empirical case is therefore suggestive rather than definitive, but the conceptual argument — that action-driving reasoning must be collected in situ, not synthesized retrospectively — is a substantive contribution to the agent data methodology literature.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation spans four distinct environment categories: (1) Minecraft using tasks from the Minedojo (Fan et al., 2022) and MCU (Lin et al., 2023) benchmarks — over 800 diverse tasks with manually curated initial environments, each completable within 600 steps and paired with a unique natural language instruction; (2) Unseen web games from the Poki online platform — four games spanning racing, jumping, endless running, and 3D flying, selected specifically because they were absent from the training distribution; (3) VizDoom (Wydmuch et al., 2018) — a semi-realistic FPS benchmark using four standard maps (Battle-1, Battle-2, Defend the Line, Defend the Center); and (4) MiniWorld (Chevalier-Boisvert et al., 2023) — a minimalist 3D simulator evaluating navigation, object interaction, and physical reasoning. The absence of any of these environments from the pre-training corpus is explicitly verified: "We carefully selected a set of representative benchmarks to assess the agent's adaptability and performance... these benchmarks encompass a variety of game types, task complexities, and interaction modalities, ensuring a comprehensive evaluation of the agent's generalization ability in new, unseen environments" (Section 4.2.1). For Minecraft specifically, the paper states that "we further controlled the random seed to guarantee that the agent had not been exposed to these environments during training."

• Base model(s). Three variants of Game-TARS are trained: Game-TARS-MoE-mini (initialized from Seed-VL-1.5 (Guo et al., 2025), with a 632M-parameter vision encoder and a Mixture-of-Experts LLM with 2.5B active parameters), Game-TARS-MoE-Large (a larger MoE variant, scale unspecified beyond "larger version"), and Game-TARS-Dense (initialized from Qwen2.5-VL-7B-Instruct (Bai et al., 2025), trained with VeOmni strategies (Ma et al., 2025)). The dense model is "trained exclusively on game datasets, without incorporating datasets from other agents trajectories or general visual question-answering" (Table 3 footnote), serving as a controlled comparison to isolate the effect of cross-domain data. The MoE-mini is the primary evaluation model; the MoE-Large appears only in the model description and its results are not separately reported in any evaluation table or figure. The choice of Seed-VL and Qwen2.5-VL backbones reflects the paper's strategy of starting from strong vision-language foundations and adapting them to agent-specific interaction through continual pre-training.

• Metrics. Evaluation metrics are task-specific and environment-dependent. For Minecraft, the primary metrics are Average Success Rate (ASR) and Average Completion Steps, measured across at least three runs per task to reduce variance; tasks are further categorized into Embodied (3D exploration and object interaction), GUI (2D crafting and interface manipulation), and Combat (survival and fighting) sub-groups. For web games, metrics are game-specific: checkpoint time in seconds for Race (lower is better), levels completed within 20 seconds for Jump-Only, score achieved within 20 seconds for Temple Run, and cumulative survival score for Airplane Flying. For VizDoom, the metric is average episode reward (cumulative over an episode). For MiniWorld, the metric is task completion success, reported per map. All metrics are computed from the agent's final selected action sequence, with no post-hoc filtering or oracle selection.

• Baselines. The paper compares against multiple categories of prior work. For Minecraft (Table 3): Policy-based agents include VPT (Baker et al., 2022), STEVE-1 (Lifshitz et al., 2023), and Dreamer V3 (Hafner et al., 2023). VLM-based agents include JARVIS-VLA (Li et al., 2025b), OmniJARVIS (Wang et al., 2024b), OpenHA (Wang et al., 2025d), and UI-TARS-1.5 (Seed, 2025b). Despite UI-TARS-1.5 being published by an overlapping set of authors, the paper treats it as an independent baseline and does not train it on the Game-TARS data mixture. For VizDoom (Figure 8): baselines include GPT-5 (OpenAI, 2025a), Claude-4-Sonnet (Anthropic, 2025), and Gemini-2.5-Pro (Comanici et al., 2025) — all evaluated as zero-shot game-playing agents with identical system prompts specifying game rules and action spaces. For web games (Figure 7): GPT-5 serves as the sole model baseline, with fresh human players (at least four individuals with no prior experience in the selected games) providing a human performance reference. For MiniWorld (Figure 9): GPT-5 is again the sole model baseline.

• Generation budget / compute accounting. The paper does not use a standardized "generation budget" in the sense of counting sampled solutions (as is common in LLM agent evaluations). Instead, evaluation is conducted with fixed-step budgets: Minecraft tasks are limited to 600 steps per episode; web games are evaluated within 20-second windows (Jump-Only, Temple Run) or until failure (Airplane Flying, Race); VizDoom and MiniWorld use standard episode lengths. The primary computational resource accounting is implicit in the training data scale — 500B+ pre-training tokens, 20B post-training tokens — rather than in inference-time compute allocation. For fair comparison with general VLMs (GPT-5, Claude, Gemini), "both models were provided with identical system prompts, including basic game rules and action space definitions" (Section 4.2.3). There is no attempt to equalize total FLOPs or inference compute between Game-TARS and these baselines — the comparison is zero-shot capability given identical prompting, not compute-matched performance.

• Cross-validation / statistical protocol. For Minecraft evaluations, "each task was evaluated at least three times to reduce randomness in results" (Section 4.2.2). The 800+ tasks span "distinct initial environments" with controlled random seeds to prevent overlap with training data. For web games and VizDoom, "we conducted at least five runs per game and reported the best-of-5 (Bo5) results" for both GPT-5 and Game-TARS (Section 4.2.3). Human baselines report "their best scores" from at least four participants. The paper does not report confidence intervals, standard errors, or statistical significance tests for any comparison. For the offline action prediction analysis in Section 4.3.3, the evaluation uses "1,000 trajectories containing more than 100k steps" — a substantial sample, but statistical reliability metrics are not provided. There is no cross-validation in strategy or hyperparameter selection for the main evaluation; all models are evaluated once at the end of training with the reported configurations.

---

Main Quantitative Results

Evaluation on Open-World Minecraft (Table 3)

The headline result is that Game-TARS-MoE-mini achieves approximately 2× the success rate of the previous state-of-the-art across all task categories. On embodied tasks, Game-TARS-MoE-mini reaches 72.0% ± 21.5% success rate versus 42.1% ± 20.4% for UI-TARS-1.5 (the strongest prior baseline) and 30.1% ± 13.9% for OpenHA — a 1.7× improvement over the SOTA. On GUI tasks, it achieves 55.4% ± 17.6% versus 36.7% ± 17.2% for UI-TARS-1.5 (1.5× improvement). On combat tasks, it achieves 66.1% ± 25.2% versus 31.9% ± 13.7% for OpenHA (2.1× improvement) and 31.0% ± 16.4% for UI-TARS-1.5.

The comparison with policy-based agents is even starker: VPT achieves 6.0% ± 11.4% on embodied tasks, STEVE-1 achieves 8.0% ± 17.0%, and Dreamer V3 achieves 3.0% ± 10.2%. These are essentially non-functional on the MCU benchmark despite being trained specifically on Minecraft. The paper does not explicitly discuss why these specialist Minecraft agents perform so poorly, but the likely explanation is that MCU tasks require instruction-following and multi-step reasoning that pure behavioral cloning (VPT) or text-conditioned video models (STEVE-1) were not designed to handle — these models learned Minecraft behaviors but not the ability to map diverse natural language instructions to action sequences.

The Game-TARS-Dense variant (trained exclusively on game data, no cross-domain agent or general VQA data) achieves substantially lower performance: 50.4% on embodied, 39.1% on GUI, 38.1% on combat tasks. The gap between Dense (game-only) and MoE-mini (game + multimodal + cross-domain) quantifies the contribution of non-game data: +21.6 percentage points on embodied, +16.3 on GUI, +28.0 on combat. This is a direct measure of cross-domain transfer — data from captioning, VQA, code agents, and research agents substantially improves Minecraft gameplay despite having no Minecraft content.

A second comparison evident in Table 3 is completion step efficiency. Game-TARS-MoE-mini completes embodied tasks in an average of 188 steps versus 290 steps for UI-TARS-1.5 and 305 steps for JARVIS-VLA — a 35% reduction in steps-to-completion. For combat tasks, it requires 271 steps versus 346 for UI-TARS-1.5 (22% reduction). For GUI tasks, the pattern is less pronounced (288 vs. 320 steps, 10% reduction), possibly because GUI interactions have inherent latency (menu navigation, crafting animations) that place a floor on minimum completion time regardless of decision quality.

Task category analysis. The paper groups the 800+ tasks into three categories with qualitatively different demands. GUI tasks (2D interface manipulation) show the smallest relative advantage for Game-TARS over prior VLM-based agents (1.5×) — these tasks primarily test visual grounding and menu navigation, capabilities that GUI-specialized agents like UI-TARS-1.5 are explicitly designed for. Embodied tasks (3D exploration, object interaction) show a larger advantage (1.7×) — these require the sensorimotor skills that the unified action space and game pre-training are specifically designed to develop. Combat tasks show the largest advantage (2.1× over the best VLM baseline) — these demand rapid decision-making, adaptive movement, and real-time responsiveness that general VLMs lack entirely and that the game pre-training provides.

Comparison with the previous state of the art in context. The paper claims "approximately 2x performance improvement in Minecraft compared to previous state-of-the-art expert models" (Section 1, citing OpenHA, Wang et al., 2025d). The actual comparison to OpenHA shows: 72.0% vs. 30.1% on embodied tasks (2.4×), 55.4% vs. 32.5% on GUI tasks (1.7×), and 66.1% vs. 31.9% on combat tasks (2.1×). So the "2×" claim is accurate for the overall pattern but the multiplier varies by task type, with embodied and combat showing larger gains than GUI tasks.

---

Evaluation on Unseen Web 3D Games (Figure 7)

The headline result is that Game-TARS-mini approaches or exceeds fresh human performance on unseen web games and substantially outperforms GPT-5. Specific numbers, read from Figure 7:

• Race (checkpoint time, lower is better): Game-TARS-mini achieves approximately 9.3 seconds to reach the first checkpoint. The fresh human baseline achieves approximately 8.5 seconds (best score). GPT-5 achieves approximately 15+ seconds (substantially slower). Game-TARS is competitive with humans and roughly 1.6× faster than GPT-5.

• Jump-Only (levels completed in 20 seconds, higher is better): Game-TARS-mini achieves approximately 8 levels. The human baseline achieves approximately 6 levels (best). GPT-5 achieves approximately 3 levels. Game-TARS exceeds human performance by ~2 levels and achieves ~2.7× more levels than GPT-5.

• Temple Run (score within 20 seconds, higher is better): Game-TARS-mini achieves a score of approximately 800. The human baseline achieves approximately 1000 (best). GPT-5 achieves approximately 200. Game-TARS is at roughly 80% of human performance and 4× GPT-5's score.

• Airplane Flying (cumulative survival score, higher is better): Game-TARS-mini achieves approximately 120. The human baseline achieves approximately 60 (best). GPT-5 achieves approximately 15. Game-TARS doubles human performance and achieves 8× GPT-5's score.

A pattern emerges across these results: Game-TARS-mini outperforms fresh humans on two of four games (Jump-Only by ~33%, Airplane Flying by ~100%) and is competitive on a third (Race), while underperforming humans only on Temple Run (80% of human best). GPT-5, by contrast, is substantially below human performance on all four games and is outperformed by Game-TARS by margins ranging from 1.6× (Race) to 8× (Airplane Flying).

The Airplane Flying result is particularly notable: humans achieve a survival score of ~60, while Game-TARS achieves ~120 despite never having seen this specific game during training. The paper does not provide a mechanistic explanation for superhuman performance, but the implication is that the agent's pre-training on diverse game types has produced visuomotor policies that are more consistent and less error-prone than human novices on this specific task.

Important caveat on evaluation methodology. The paper uses best-of-5 (Bo5) for both GPT-5 and Game-TARS, but reports best score for human baselines (from "at least four individuals with no prior experience"). This is not a symmetric comparison — Bo5 for models vs. single-best for humans favors the models, especially on tasks with high variance. The human baseline is also "fresh" (no prior experience), while Game-TARS has been pre-trained on 500+ games that may include games with similar mechanics to the Poki test games even if the specific games themselves are unseen. The claim of "near-human generalization" should be interpreted with this asymmetry in mind: Game-TARS approaches or exceeds the performance of novice humans who are seeing the game for the first time, not experienced players.

Inference latency and the sandbox environment. A critical methodological detail: "Due to the inference latency of the model, we packaged these web games into a sandbox and used time-pause to ensure causal interaction" (Section 4.2.3 footnote). This means the evaluation was NOT conducted in real-time — the game was paused between frames to accommodate the model's inference latency, then resumed after the action was produced. This is a significant difference from how humans play (in continuous real-time) and how GPT-5 was evaluated (presumably also with pausing, though this is not explicitly stated). The pausing removes time pressure from the evaluation, which may benefit Game-TARS more than GPT-5 (since Game-TARS's pre-training included real-time game data where latency was not an issue, creating a potential train-test mismatch in temporal dynamics). The paper does not discuss how this affects the ecological validity of the comparison.

---

Evaluation on FPS Benchmark VizDoom (Figure 8)

The headline result is that Game-TARS-mini significantly outperforms all general-purpose VLM baselines across all four VizDoom maps, often by large margins.

Specific episode rewards, read approximately from Figure 8:

The margins are substantial: on Battle-1, Game-TARS achieves roughly 7.5× the reward of GPT-5 and 30× the reward of Gemini-2.5-Pro. On Defend the Line, it achieves roughly 14× GPT-5's reward. The general VLMs are essentially non-functional as FPS agents — their episode rewards are in the single digits across all maps, consistent with agents that do little more than survive the first few seconds of each episode.

The paper supplements these quantitative results with a qualitative observation that deserves attention: "Beyond raw scores, qualitative analysis revealed that the agent exhibited advanced FPS behaviors, including effective corner-peeking, evasive maneuvers to dodge projectiles, and adaptive targeting strategies" (Section 4.2.3). These behaviors — corner-peeking (moving in and out of cover to take shots), projectile dodging (strafing to avoid incoming fire), adaptive targeting (adjusting aim based on enemy movement) — are emergent sensorimotor skills that were never explicitly programmed or rewarded. They emerged from pre-training on diverse FPS-like gameplay data (present in the 500-game training corpus) and transferred to the unseen Doom environment. The paper does not quantify these behaviors (e.g., "percentage of episodes where corner-peeking was observed"), so the claim remains qualitative, but it is consistent with the quantitative results — the reward gap between Game-TARS and baselines is too large to be explained by basic survival behaviors alone.

What this evaluation demonstrates and what it doesn't. The VizDoom results demonstrate that Game-TARS has acquired transferable FPS skills from its multi-game pre-training — skills that general VLMs, despite their broad knowledge, completely lack. However, the evaluation does NOT demonstrate that Game-TARS approaches human-level FPS performance. The paper does not provide a human baseline for VizDoom, so we cannot assess whether an episode reward of 60 on Battle-1 represents skilled play or merely competent survival. Published VizDoom competition results (Wydmuch et al., 2018) suggest that specialized RL agents achieve substantially higher rewards, but the paper does not make this comparison. The claim is specifically about relative to general VLMs, not absolute FPS capability.

---

Evaluation on MiniWorld Simulator (Figure 9)

The headline result is that Game-TARS-mini demonstrates robust generalization to a 3D simulator with no training exposure, substantially outperforming GPT-5 across nearly all tested maps. Specific per-map results are shown in Figure 9 as a bar chart; the paper states that "across nearly all tested maps, Game-TARS demonstrated robust performance, successfully navigating new environments and completing tasks." GPT-5 achieves lower or zero performance on most maps. The paper does not enumerate the specific maps or provide exact success rates in the text, relying entirely on Figure 9 for the quantitative results.

The significance of MiniWorld. MiniWorld is a visually minimalist environment (simple geometric shapes, flat colors) that looks nothing like the commercial games in Game-TARS's pre-training corpus (which includes adventure games, shooters, RPGs, and racing games with rich visual textures). Successful navigation in MiniWorld therefore cannot be explained by visual similarity to training data — the agent must be transferring abstract visuomotor skills (how to move toward a target, how to avoid obstacles, how to interact with objects) that generalize across visual domains. This is the strongest evidence in the paper for domain-invariant sensorimotor transfer, though the paper does not emphasize this point.

---

Scaling Experiments (Figure 10)

The paper presents two complementary scaling analyses that together form the empirical backbone of its central claim — that the unified action space enables sustained improvements with data scale that alternative action spaces cannot match.

Training-time scaling (Figure 10, right panel). The experiment measures Minecraft success rate as a function of training data composition and volume:

1. GUI-based action space (blue curve): With limited data, GUI agents achieve relatively strong performance (exact numbers not provided in the text, visible in Figure 10 — approximately 42% success rate at the data scale where the unified space achieves ~42%). However, "such action spaces require extensive preprocessing of raw trajectories and do not naturally transfer across heterogeneous games. As a result, their scalability and ceiling performance are inherently constrained" (Section 4.3.1). The GUI curve plateaus — additional cross-game data cannot be incorporated because GUI representations are game-specific.

2. Unified action space, in-domain game data only (orange curve, leftmost point): With only Minecraft-specific data, the unified space underperforms GUI-based approaches (approximately 42% success rate, vs. GUI's higher performance at equivalent data volume). The paper explains: "the proposed unified device-level action space is highly scalable but not natively supported by pre-trained VLMs. As a result, its performance lags behind GUI-based approaches when trained on small datasets."

3. Adding cross-game data (+200B tokens, orange curve, middle point): Minecraft success rate jumps from approximately 42% to 65%. The paper frames this as the pivotal finding: "Incorporating 200B additional tokens from other games led to a significant improvement in Minecraft performance, boosting the success rate from 42% to 65%. This observation echoes the Bitter Lesson in AI: general and simple representations, when combined with scaling in data and compute, yield stable and long-term improvements."

4. Adding multimodal + cross-domain agent data (orange curve, rightmost point): Success rate increases further to 72%. "Extending pre-training to include multimodal data (e.g., captioning, VQA, video understanding) and agent trajectories from other domains led to an additional improvement, pushing performance to 72%. This finding demonstrates that generalist agent models can benefit from heterogeneous sources of action–trajectory data, acquiring broadly transferable decision-making capabilities."

The progression 42% → 65% → 72% represents a 71% relative improvement from the in-domain-only baseline to the full data mixture. The paper attributes this improvement to two mechanisms: (a) cross-game transfer of sensorimotor primitives (the 42% → 65% jump), and (b) cognitive capability improvements from non-game reasoning data (the 65% → 72% jump).

Inference-time scaling (Figure 10, left panel). The experiment shows that "Game-TARS can improve the task success rates through exploration in inference-time steps." The left panel of Figure 10 plots success rate as a function of inference steps (likely meaning the number of interaction steps the agent is allowed to take per episode, though the paper does not specify the exact x-axis definition). The curve shows monotonically improving performance with more inference steps, suggesting that the agent benefits from extended interaction — it can explore, recover from mistakes, and refine its behavior over longer time horizons. This is consistent with the paper's earlier findings about the importance of the dual-tier memory and long-context training for enabling effective long-horizon behavior.

The significance of these scaling curves. The training-time scaling experiment is arguably the paper's single most important result because it operationalizes the central claim: that the unified action space is not just convenient but scalably better. The GUI curve reaches a ceiling; the unified curve continues climbing. The inference-time scaling experiment demonstrates that the model can make productive use of additional environment interaction — a necessary condition for deployment in real tasks where maximum step counts are a practical constraint.

However, the paper does not provide a rigorous scaling law analysis (no fitted power laws, no extrapolation to larger data volumes, no measurement of diminishing returns). The three data points on the unified action space curve (in-domain, +cross-game, +multimodal) are sufficient to establish a positive trend but insufficient to characterize the functional form of the scaling relationship. The claim that the trend will continue with further data is plausible but unverified.

---

Ablation Studies and Robustness Checks

Sparse Thinking vs. alternative reasoning strategies (Table 4): The paper ablates three reasoning modes — No-thinking (pure reactive), Greedy Thinking (reason at every step), and Sparse Thinking (the proposed method) — across three environments (Minecraft, Doom, and 2048) plus an efficiency metric (average tokens per step). On Minecraft, Sparse Thinking achieves 63% vs. 55% (No-thinking) and 45% (Greedy Thinking). On Doom, Greedy Thinking is actively harmful (7.2 reward vs. 13.2 for No-thinking and 11.5 for Sparse). On 2048 (a combinatorial puzzle game), Greedy Thinking achieves the best score (986 vs. 924 for Sparse and 445 for No-thinking), but at the cost of 68 tokens/step vs. 37 tokens/step for Sparse — an 84% increase in inference cost for a 6.7% improvement in score. The key finding is that no single reasoning strategy is optimal across all environments: Greedy Thinking excels on puzzles requiring constant deliberation, No-thinking excels on fast-paced reactive environments, and Sparse Thinking provides the best average performance with the lowest token cost. The qualitative observation that the agent "develops an implicit meta-cognition mechanism: it chooses when to reason and when to act reflexively" is supported by these numbers — the agent adapts its reasoning density to task demands without explicit task identification.

Decaying loss validation (Table 5): The paper compares training with and without the decaying continual loss on both online and offline metrics. Online results (Minecraft): Embodied tasks improve from 44.90% to 46.15% ASR (modest gain), with diversity increasing from 60% to 90% and steps remaining comparable (172 vs. 168). Combat tasks show the largest improvement — 14.54% to 22.22% ASR (a 53% relative gain), with diversity increasing from 50% to 60% and steps decreasing from 305 to 247 (19% reduction). GUI tasks improve from 37.75% to 42.26% ASR, with diversity increasing from 80% to 100% and steps decreasing from 244 to 164 (33% reduction). The offline analysis on 1,000 trajectories (>100k steps) reveals the mechanism: the decaying loss reduces overall action prediction accuracy from 59% to 47% but increases non-repetitive accuracy (accuracy on steps where the action differs from the previous one) from 12% to 39% — a +28 percentage point absolute improvement. This tradeoff is the key diagnostic: standard loss incentivizes the model to copy the previous action (achieving 59% overall accuracy through cheating), while decaying loss forces the model to learn genuine state-conditioned action prediction at the cost of lower accuracy on easy repetitive cases.

Unified vs. GUI action space scaling (Figure 10, right panel): This is not presented as a formal ablation with controlled variables but as a comparison across training paradigms. The GUI space achieves higher performance in low-data regimes but plateaus because "such action spaces require extensive preprocessing of raw trajectories and do not naturally transfer across heterogeneous games." The unified space starts lower but overtakes GUI-based approaches as cross-game data is added. This is the empirical justification for the paper's central architectural choice, though it is a comparison between paradigms rather than a controlled ablation within the Game-TARS system.

Multi-stage vs. single-stage continual pre-training (Section 4.1): The paper reports, without a dedicated figure or table, that a multi-stage training strategy (pure action trajectories → offline-synthesized ReAct → online Think-Aloud ReAct → general multimodal fine-tuning) had a "detrimental effect on the model's final performance" compared to the adopted single-stage blended approach. This is a negative result — the paper tried the intuitive sequential curriculum and found it underperformed simply mixing all data together. The paper does not quantify the performance difference or hypothesize why sequential training failed (catastrophic forgetting? distribution shift between stages? inability to learn complementary skills simultaneously?), making this a suggestive but incomplete finding.

Thinking pattern analysis beyond aggregate scores (Section 4.3.2 discussion): The paper provides qualitative trajectory analysis showing that "in Minecraft crafting sequences, the agent reasons carefully at the start of a task (to identify needed resources and steps) but executes subsequent mining or crafting actions without redundant re-planning. Similarly, in 2048, it reasons at every move due to the inherent combinatorial structure of the game, while in VizDoom it minimizes reasoning to maintain responsiveness." This is not a controlled ablation but an observational finding that supports the claim of learned meta-cognition — the agent modulates reasoning depth based on task structure, not a fixed schedule.

Cross-source data contribution (implicit in Table 3): The comparison between Game-TARS-Dense (game data only, 50.4% embodied success) and Game-TARS-MoE-mini (full data mixture, 72.0% embodied success) functions as an ablation of cross-domain and multimodal data, though the models differ in architecture (Dense vs. MoE) and initialization (Qwen2.5-VL vs. Seed-VL), confounding the comparison. The +21.6 percentage point difference cannot be cleanly attributed to data alone.

---

Critical Assessment

Does Game-TARS achieve "approximately 2× the success rate over the previous SOTA model on open-world Minecraft tasks"?

This claim is supported for the specific baselines and task categories reported, but the "2×" framing requires qualification. The comparison to OpenHA (Wang et al., 2025d), cited in the abstract as "the previous sota model," shows: 2.4× on embodied (72.0% vs. 30.1%), 1.7× on GUI (55.4% vs. 32.5%), and 2.1× on combat (66.1% vs. 31.9%). The 2× multiplier is approximately accurate as a rounded average. However, the paper's Table 3 also includes UI-TARS-1.5, which achieves 42.1% on embodied tasks — a 1.7× improvement rather than 2× over this stronger baseline. The choice of OpenHA as "the previous SOTA" is debatable when UI-TARS-1.5 (published by an overlapping author group, Released 2025) achieves higher performance on embodied and GUI tasks. The claim would be more precisely stated as "1.7–2.4× improvement depending on the task category and baseline."

More importantly, the 2× claim applies to aggregate success rates on the MCU benchmark, not to individual task-level performance. The MCU benchmark consists of 800+ diverse tasks; aggregate success rate can mask substantial variance in which tasks benefit and which don't. The paper's per-category breakdown (embodied/GUI/combat) provides some disaggregation, but within each category there are likely tasks where Game-TARS shows no improvement and tasks where improvement is far larger than 2×. The paper does not provide task-level distributions, standard deviations (only some are reported), or statistical tests for the comparison.

Does Game-TARS achieve "generality close to fresh humans in unseen web 3D games"?

This claim is partially supported but the comparison is asymmetric in ways that favor Game-TARS. On Jump-Only, Game-TARS exceeds human performance (8 vs. 6 levels). On Airplane Flying, Game-TARS doubles human performance (120 vs. 60 survival score). On Race, Game-TARS is competitive (9.3s vs. 8.5s). On Temple Run, Game-TARS underperforms humans (800 vs. 1000 score). The aggregate picture is indeed "close to fresh humans" — Game-TARS outperforms humans on 2/4 games, is competitive on 1/4, and underperforms on 1/4.

However, three methodological concerns weaken this claim:

1. Asymmetric evaluation: Game-TARS gets best-of-5 (best score from 5 runs), while humans report their single best score from "at least four individuals." A Bo5 evaluation gives the model 5 attempts to get lucky, while each human gets one attempt. For high-variance games, this materially advantages the model.

2. Time-pausing during inference: The evaluation sandbox "used time-pause to ensure causal interaction" due to model inference latency. Humans played in real-time without pausing. This removes time pressure from the model's evaluation — it can deliberate between actions without the game state advancing — while humans must react in real-time. This is a substantial advantage for the model, particularly in the racing and flying games where reaction speed matters.

3. Small sample of games (4) and humans (4+): Four games and four human participants is insufficient to establish "generality." The human baseline is also "fresh" (no prior experience with these specific games), while Game-TARS has 20k hours of pre-training on games that may include titles with similar mechanics. A more rigorous evaluation would use a larger set of unseen games, a larger human sample with defined expertise levels, and symmetric evaluation protocols.

Does Game-TARS "outperform GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet in FPS benchmarks"?

This claim is strongly supported by the VizDoom results (Figure 8). Game-TARS-mini achieves 5–30× higher episode rewards than these general VLMs across all four maps. The margin is large enough that even substantial measurement variance would not change the qualitative conclusion. The result is also intuitively plausible: FPS games require real-time visuomotor control that general VLMs are fundamentally not trained for, while Game-TARS has extensive FPS-like pre-training data.

However, the claim is narrower than it might appear. The comparison is against zero-shot deployment of these VLMs — they are given the game rules and action space via system prompt and asked to play, with no fine-tuning, no few-shot examples, and no adaptation. This is a fair comparison for assessing innate capability, but it is not a comparison against the best these models could do with game-specific adaptation (fine-tuning, in-context examples, specialized prompting). It is also not a comparison against specialized FPS agents (e.g., RL-trained VizDoom agents from the literature), which achieve substantially higher rewards. The claim should be understood as "Game-TARS outperforms general-purpose VLMs used as zero-shot FPS agents," not "Game-TARS is state-of-the-art at FPS games."

Do the scaling experiments confirm that the unified action space "sustains improvements when scaled to cross-game and multimodal data"?

This claim is supported by the training-time scaling experiment (Figure 10, right panel), which shows Minecraft success rate improving from 42% (in-domain only) to 65% (+cross-game) to 72% (+multimodal + cross-domain). The trend is clearly positive and the absolute gains are substantial.

However, the evidence has important limitations:

1. Three data points do not a scaling law make. The paper shows improvement at three data scales but does not fit a functional form, extrapolate to larger scales, or measure the rate of diminishing returns. We cannot determine from these data whether adding another 200B cross-game tokens would produce another +23 percentage point gain or a +2 percentage point gain. The claim that the unified action space "sustains improvements" is qualitative, not quantitative.

2. The data points are confounded by data type, not just data volume. The progression is: Minecraft-only data → Minecraft + cross-game data → Minecraft + cross-game + multimodal + cross-domain data. The improvement could be driven by data diversity (adding new types of data) rather than data volume (adding more data of the same type). The paper cannot distinguish between "more data helps" and "more diverse data helps" because volume and diversity increase simultaneously.

3. The GUI baseline is not a controlled comparison. The blue curve in Figure 10 represents GUI-based action spaces, but the paper does not specify which GUI agent, trained on what data, at what scale. It is a conceptual comparison point rather than a controlled experiment where the same model architecture is trained on the same data with different action spaces. The claim that GUI spaces "plateau" while unified spaces scale is therefore a cross-study comparison rather than an experimental finding within this paper.

Missing experiments that would strengthen the paper

1. Clean data scaling curve for a fixed action space with fixed data type. A controlled experiment that varies only the volume of cross-game data while holding data composition constant would isolate the effect of scale from the effect of diversity.

2. Evaluation on non-game computer-use tasks. The paper claims that game pre-training provides a sensorimotor foundation for general computer use (Section 3.5, Conclusion), but all evaluations are on games or game-like simulators. Evaluation on a standard GUI benchmark (e.g., OSWorld, Mind2Web) or code benchmark (e.g., SWE-bench) would directly test this claim.

3. Evaluation of cross-domain transfer in the opposite direction. If game data improves GUI/code performance, that would validate the layered skill acquisition hypothesis. If game data only improves game performance, the hypothesis that game pre-training is a general sensorimotor foundation is weakened.

4. Human baseline for VizDoom. Without knowing what human-level VizDoom performance looks like, the absolute capability level of Game-TARS on FPS tasks is uncertain.

5. Statistical significance and confidence intervals. The paper reports point estimates (72.0%, 9.3s, 60 reward) without error bars, confidence intervals, or significance tests. This is standard for large-scale ML systems papers but limits the interpretability of comparisons, especially for the web game results where sample sizes are small (5 runs per game).

6. Ablation isolating Think-Aloud data from offline-synthesized ReAct data. The pre-training mixture includes both 45B tokens of offline-synthesized ReAct trajectories and 127B tokens of online Think-Aloud data. The paper argues that Think-Aloud is superior (Section 2.2) but provides no ablation showing that the Think-Aloud component specifically improves performance over using only offline-synthesized reasoning.

Summary of evidential support

The paper's central empirical claims are reasonably well-supported for the specific environments and baselines tested, with the strongest evidence being the VizDoom results (large margin over general VLMs) and the training-time scaling trend (clear positive slope). The weakest evidence is the human-comparability claim for web games (asymmetric evaluation protocol) and the broad claim about general computer-use capability (no non-game evaluation). The paper's contribution is primarily in demonstrating feasibility — that a unified action space CAN scale across games — rather than in providing definitive scaling laws or rigorous head-to-head comparisons with all relevant baselines. This is appropriate for a paper introducing a new paradigm and training pipeline at unprecedented scale, but the quantitative claims should be interpreted as establishing existence proofs rather than precise effect sizes.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted For

The assumption or constraint: Game-TARS treats the unified action space as cost-free at deployment — it assumes that once the model is trained, it can be deployed on any game or application without per-environment engineering. This is true for the action space itself, but it ignores a hidden cost: the pre-training required to make the unified space work is extraordinarily expensive, and the paper does not provide any analysis of how this cost amortizes over deployments. The model requires 500B+ tokens of pre-training data, including 127B tokens of human-collected Think-Aloud data across 500 games. The Think-Aloud pipeline (Section 2.2) requires human annotators to play games while verbalizing thoughts, a specialized recording infrastructure that captures synchronized screen/keyboard/mouse/audio, ASR transcription, LLM-based text refinement, and visual-anchor cursor alignment. This pipeline is described in detail but its cost (in annotator hours, compute for processing, and engineering effort) is never quantified.

The consequence: A practitioner deciding whether to adopt the unified action space paradigm cannot perform a cost-benefit analysis. The paper's central argument is that GUI-based action spaces have a scalability ceiling — they cannot aggregate data across games, so their performance plateaus — while the unified space continues improving with more data (Figure 10, right panel). But this comparison ignores the cost of producing that data. If the Think-Aloud pipeline costs 10× more per hour of gameplay than simply recording keyboard-mouse actions (for pure action trajectories), and if the 127B tokens of Think-Aloud data were essential for achieving the reported performance, then the total cost of the approach may be prohibitive for organizations without ByteDance-scale annotation resources. The paper's own scaling experiment (Figure 10) shows that the unified space underperforms GUI-based approaches in low-data regimes — meaning you must invest heavily in data collection before the paradigm pays off. Without cost estimates, the threshold at which the unified space becomes cost-effective relative to building per-game GUI agents is unknown.

What evidence exists in the paper: The paper provides detailed data volume statistics (Section 4.1: 61B pure action tokens, 45B offline-synthesized ReAct tokens, 127B online Think-Aloud tokens, 200B general multimodal tokens, 20B cross-domain agent tokens) and states that the Think-Aloud data spans "500 games" with annotators instructed to provide "at least three key moments per minute" of reasoning (Section 2.2). The training compute is partially specified in Table 2 (16,000 pre-training steps, 2048 batch size, 500B+ tokens total) but is not translated into GPU-hours or dollar cost. The paper does not report annotator hours, total gameplay hours collected, or the cost of the ASR-LLM processing pipeline. The multi-stage vs. single-stage training negative result (Section 4.1) hints at the engineering effort involved — multiple training strategies were attempted and discarded — but this R&D cost is also unquantified.

Mitigation status: The paper does not address this limitation. It does not provide cost estimates, does not discuss the economic tradeoff between the unified and GUI paradigms, and does not suggest cheaper alternatives to the Think-Aloud pipeline. The implicit assumption is that the unified action space is the only path to generalist agents, so the cost is necessary — but this is a claim about the future, not an analysis of the present.

---

Hardest Problems Remain Unsolved — The Method Has a Hard Capability Ceiling

The assumption or constraint: Game-TARS assumes that the base vision-language model's capabilities, combined with cross-game pre-training, are sufficient to produce correct behavior on any task within the evaluated environments. The paper does not explicitly state this assumption, but it is implicit in the architecture: the model learns by imitating human demonstrations, and if a task requires capabilities that are not present in the demonstration data distribution, the model cannot acquire them. If no human demonstrator successfully completed a particular type of task during the Think-Aloud data collection, the model has no positive examples to learn from.

The consequence: The model's performance is bounded by the skill level of the human demonstrators and the coverage of the pre-training data. Tasks that require novel strategies, creative problem-solving, or behaviors significantly different from what humans demonstrated will be beyond the model's reach. More importantly, the model cannot exceed the competence ceiling of its training distribution in any qualitative sense — it can become more consistent and less error-prone than humans (as the Airplane Flying results suggest, Section 4.2.3, Figure 7), but it cannot discover strategies that no human ever demonstrated. This is a fundamental limitation of pure imitation learning: the policy can interpolate within the demonstration distribution but cannot extrapolate beyond it to novel behaviors.

The paper provides indirect evidence of this ceiling in the Minecraft evaluation (Table 3). Despite the massive improvement over prior work (72.0% embodied success rate), 28% of embodied tasks still fail. These failures are not analyzed — the paper does not categorize which tasks fail, whether they share common characteristics, or whether they represent a particular capability gap. If the 28% failure rate is concentrated in specific task types (e.g., tasks requiring spatial reasoning about complex structures, tasks requiring multi-step planning with conditional branches), then the model has a systematic capability ceiling that additional data from the same distribution is unlikely to overcome.

What evidence exists in the paper: The paper does not directly analyze failure modes or capability ceilings. Table 3 reports average success rates with standard deviations (e.g., 72.0% ± 21.5% for embodied tasks) — the wide standard deviations suggest substantial variance in per-task performance, but the paper does not disaggregate this variance. The standard deviation of ±21.5% on a 72.0% mean implies that some tasks are solved reliably (near 100%) while others are solved rarely or never (near 0%). Understanding which tasks fall into which regime would reveal the capability boundary, but this analysis is absent.

The scaling curve in Figure 10 (right panel) shows improvement from 42% → 65% → 72% as data is added, with no evidence of saturation — but this curve is at three data points, and the functional form (linear? logarithmic? sigmoidal?) is unknown. If the curve is logarithmic, substantial further gains may require exponentially more data, making progress beyond ~80% practically infeasible. If it is linear, continued scaling will eventually reach near-perfect performance.

Mitigation status: The paper does not address the capability ceiling. There is no failure analysis, no categorization of error types, no discussion of whether the remaining failures are due to insufficient data, insufficient model capacity, or fundamental limitations of imitation learning. The paper's framing emphasizes the positive results and the scalability trend without interrogating where the trend leads or whether it will encounter diminishing returns.

---

No Non-Game Evaluation to Support "General Computer-Use Agent" Claims

The assumption or constraint: The paper explicitly frames Game-TARS as a step toward general computer-use agents — the title includes "Generalist Multimodal Game Agents," the Introduction positions games as a platform toward AGI, Section 3.5 describes the goal of transforming the model "from a specialized game-player into a versatile general computer-user," and the Conclusion states that Game-TARS provides "a promising path toward generalist computer-use agents." This framing assumes that the sensorimotor skills learned from game pre-training transfer to non-game computer tasks — navigating file systems, filling web forms, using productivity software, writing code in an IDE.

The consequence: The paper provides zero evaluation of Game-TARS on any non-game computer-use task. All reported benchmarks — Minecraft, Poki web games, VizDoom, MiniWorld, and 2048 — are games or game-like environments. There are no results on standard GUI agent benchmarks (OSWorld, Mind2Web, ScreenSpot), code benchmarks (SWE-bench, HumanEval), web navigation benchmarks (WebArena, BrowseComp), or any productivity software task. The cross-source agentic trajectory post-training (Section 3.5) incorporates code, GUI, and research agent data, and the scaling experiment (Figure 10) shows that this data improves Minecraft performance from 65% to 72% — but the paper does not test the reverse direction (whether game pre-training improves performance on the code/GUI/research tasks themselves).

A practitioner interested in building a general computer-use agent cannot determine from this paper whether Game-TARS's game-trained skills actually transfer to productivity tasks, or whether the transfer is asymmetric (non-game data helps games, but game data does not help non-game tasks). The paper's architecture and training philosophy rest on the assumption that game pre-training provides a "sensorimotor foundation" for general computer use, but this assumption is untested.

What evidence exists in the paper: The evidence for cross-domain transfer is entirely one-directional. Table 3 shows that Game-TARS-MoE-mini (trained with cross-domain data) outperforms Game-TARS-Dense (trained on game data only) by 21.6 percentage points on embodied Minecraft tasks — so non-game data improves game performance. Figure 10 (right panel) confirms this with the 65% → 72% improvement from adding multimodal and cross-domain data. But there is no evidence in the opposite direction — no table showing that game pre-training improves performance on code generation, GUI navigation, or web tasks relative to a VLM baseline without game pre-training.

The paper's qualitative claims about emergent behaviors (corner-peeking in VizDoom, fox tracking in Minecraft, adaptive reasoning depth) are all game-specific. The dual-tier memory architecture (Section 3.4) and the sparse thinking mechanism (Section 3.3) are described in general terms but evaluated only on games.

Mitigation status: The paper does not address this gap. Section 3.5 presents cross-source training as a forward-looking design choice ("we aim to achieve a synergistic learning effect") rather than an evaluated capability. The Conclusion's claim about "generalist computer-use agents" should be read as a research vision, not an achieved result — but the paper's rhetoric often blurs this distinction. A more precise framing would state that Game-TARS demonstrates the feasibility of the unified action space for games and that evaluation on non-game computer tasks is planned future work, but the paper does not include this qualification.

---

The Evaluation Protocols Contain Asymmetries That Favor Game-TARS Over Baselines

The assumption or constraint: The paper's headline comparisons — against fresh humans on web games, against general VLMs on VizDoom — assume symmetric evaluation conditions. The paper states that "both models were provided with identical system prompts, including basic game rules and action space definitions" for GPT-5 comparisons (Section 4.2.3), implying fair comparison.

The consequence: Several evaluation design choices systematically advantage Game-TARS over the baselines, making the quantitative comparisons less rigorous than they appear:

1. Game-TARS gets best-of-5, humans get single best score (Figure 7). The web game evaluation reports best-of-5 results for both Game-TARS and GPT-5, but reports "their best scores" for human participants from "at least four individuals with no prior experience in the selected games" (Section 4.2.3). A Bo5 protocol gives the model 5 independent attempts and selects the maximum; humans effectively get one attempt. On high-variance tasks, this inflates the model's performance relative to humans. If the four human participants had each been given 5 attempts and their Bo5 scores averaged, the human baseline might be substantially higher, potentially changing the "near-human or superior performance across most games" conclusion (Section 4.2.3).

2. Time-pausing eliminates reaction-time pressure for the model but not for humans (Section 4.2.3 footnote). The web games were "packaged into a sandbox and used time-pause to ensure causal interaction" because of model inference latency. This means Game-TARS can deliberate between actions while the game state is frozen. Human participants played in real-time, where reaction speed directly affects performance — particularly in the racing game (where checkpoint time is the metric) and the flying game (where collision avoidance requires rapid adjustment). The paper does not quantify the model's inference latency or discuss how the time-pausing affects the validity of the comparison.

3. General VLMs are evaluated zero-shot with no game-specific adaptation. GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet are given a system prompt describing the game rules and action space, then asked to play (Sections 4.2.3, Figures 7–9). These models receive no few-shot examples, no in-context demonstrations, no fine-tuning, and no opportunity to observe the effects of their actions before the evaluation begins. Game-TARS, by contrast, has been pre-trained on 500+ games with 20k hours of gameplay data — it has extensive prior experience with game mechanics, visual styles, and control schemes that overlap with the test environments (even if the specific test games were not in the training set). This is a comparison between a game-specialized model and general-purpose models used in a zero-shot capacity, not a comparison between models given equivalent adaptation opportunities.

What evidence exists in the paper: The evaluation methodology is described in Sections 4.2.1–4.2.3. The Bo5 vs. single-best human comparison is stated explicitly but not flagged as a limitation. The time-pausing footnote appears in Section 4.2.3 but is not discussed as a caveat to the human-comparison claims. The zero-shot nature of the VLM baselines is inherent to the evaluation design — the paper presents the comparison as "can Game-TARS play unseen games better than general VLMs?" which is a valid question, but the asymmetry in prior experience (Game-TARS: 500 games, VLMs: zero games) means the comparison primarily demonstrates the value of domain-specific pre-training, not the superiority of the Game-TARS architecture over VLM architectures.

Mitigation status: The paper does not address these evaluation asymmetries. None of the three issues above are discussed in the Limitations section (the paper does not have a dedicated Limitations section). The claims about human-level performance and VLM-outperformance should be interpreted with these methodological caveats in mind, but the paper does not provide the reader with the information needed to make this adjustment.

---

The Think-Aloud Data Pipeline Has No Ablation, Leaving Its Contribution Unquantified

The assumption or constraint: The paper devotes substantial technical detail to the Think-Aloud data collection and processing pipeline (Section 2.2), presenting it as a key innovation over offline-synthesized ReAct trajectories. The pipeline involves human annotators verbalizing thoughts during gameplay, ASR transcription, LLM refinement, visual-anchor cursor alignment to correct timestamp biases, and thought timestamp relocation to strengthen thought-action causal links. The paper argues that offline-synthesized reasoning (where an LLM retroactively adds reasoning to recorded action sequences) "is exceptionally difficult and inefficient" because "accurately reconstructing the annotator's real-time thought process is nearly impossible" (Section 2.2).

The consequence: The paper provides no experiment that isolates the contribution of the Think-Aloud pipeline. The pre-training data mixture (Section 4.1) includes all of the following simultaneously: 61B pure action tokens, 45B offline-synthesized ReAct tokens, 127B online Think-Aloud tokens, 200B general multimodal tokens, and 20B cross-domain agent tokens. The model is trained on this blended mixture as a single stage (the multi-stage approach was tried and found detrimental, Section 4.1). There is no ablation where Think-Aloud data is replaced with an equivalent volume of offline-synthesized ReAct data while holding everything else constant.

This means we cannot determine from the paper's results whether the Think-Aloud pipeline — with all its engineering complexity (visual-anchor alignment, thought relocation, ASR-LLM refinement) — actually produces better training data than the simpler approach of having an LLM synthesize reasoning post-hoc from recorded action sequences. It is possible that the 127B Think-Aloud tokens contributed to performance primarily through data volume (adding more diverse game trajectories to the training mix) rather than through data quality (native reasoning being causally superior to synthetic reasoning). The paper's qualitative arguments about causal fidelity are plausible but empirically unverified.

What evidence exists in the paper: The Think-Aloud pipeline is described in Section 2.2 with extensive technical detail (the causal alignment problem, the IDM degeneration risk, the visual-anchor solution, the thought relocation strategy). The data volume breakdown in Section 4.1 shows 127B Think-Aloud tokens vs. 45B offline-synthesized tokens — so Think-Aloud data is 2.8× larger, making it impossible to attribute performance to quality vs. quantity. The paper's Sparse Thinking ablation (Table 4) compares reasoning strategies at inference time (No-thinking, Greedy Thinking, Sparse Thinking) but does not compare training data sources — all variants in Table 4 are trained on the same data mixture, so the ablation tests the reasoning strategy, not the data pipeline that produced the reasoning.

Mitigation status: The paper does not address this gap. There is no mention of the missing ablation, no discussion of whether the Think-Aloud pipeline's benefits (if any) justify its cost, and no suggestion that future work should isolate the contribution of native vs. synthetic reasoning data. The Think-Aloud pipeline is presented as an established contribution, but its empirical value relative to cheaper alternatives is unknown.

---

The Scaling Analysis Does Not Establish a Scaling Law — Only a Positive Trend

The assumption or constraint: The paper frames its central contribution around scalability: "the unified action space, when trained with less than 10B data, did not initially outperform GUI-based action agents in terms of success rate. However, the key advantage of the unified action space lies in its scalability" (Section 1). The scaling experiment (Figure 10, right panel) is presented as the primary evidence for this claim. The paper invokes the Bitter Lesson (Sutton, 2019) and draws an implicit parallel to the scaling laws literature (Hoffmann et al., 2022; Kaplan et al., 2020) by arguing that the unified space "sustains improvements when scaled to cross-game and multimodal data" (Abstract).

The consequence: The training-time scaling experiment (Figure 10, right panel) provides three data points on the unified action space curve: in-domain game data (~42%), +cross-game data (~65%), +multimodal + cross-domain data (~72%). Three points establish a positive trend but are insufficient to characterize a scaling law. We cannot determine from these data:

• The functional form of the scaling relationship. Is performance improving as a power law (like language model loss vs. compute), logarithmically (rapid initial gains then diminishing returns), or linearly? The three points could fit any of these functions equally well. If the relationship is logarithmic, the next 200B tokens might add only 2–3 percentage points, and reaching 90% success rate might require trillions of additional tokens — potentially infeasible. If it is a power law with a favorable exponent, continued scaling could produce substantial further gains with manageable additional data.

• Whether the improvement is driven by data volume or data diversity. Each data point on the curve also changes the composition of the training data (game-only → game + cross-game → game + cross-game + multimodal + cross-domain). The improvement could be due to adding more total tokens (volume), adding tokens from new domains (diversity), or both. A proper scaling law would hold data composition constant and vary only volume, or systematically vary composition at fixed volume to separate these effects.

• The ceiling of the scaling trend. Without a functional form, we cannot estimate where the curve saturates. The three points show no obvious saturation, but three points rarely do. The ~72% result (with full data) might be near the ceiling, or it might be far from it.

What evidence exists in the paper: Figure 10 (right panel) shows the three-point scaling curve for the unified action space and a separate curve for GUI-based action spaces (with fewer points, unlabeled with specific data volumes). The paper does not fit any functional form, report confidence intervals, or discuss the statistical uncertainty in these estimates. The qualitative interpretation — "the unified action space enables sustained performance gains through cross-game and cross-domain scaling" — is consistent with the data but not uniquely supported by it. The inference-time scaling experiment (Figure 10, left panel) is also presented without a functional fit.

Mitigation status: The paper does not address the limitations of its scaling analysis. It does not claim to have established a scaling law — it uses qualitative language ("sustains improvements," "stable and long-term improvements") — but the Bitter Lesson framing and the central argument about scalability create an expectation of quantitative scaling analysis that the paper does not fulfill. Future work that systematically varies data volume at fixed composition, fits power laws, and extrapolates to larger scales would be needed to validate the claim that the unified action space has fundamentally different scaling properties than alternative representations.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a paradigm shift in action representation for interactive agents — it argues, with substantial empirical evidence, that the action space is not an implementation detail to be designed per environment but the primary architectural choice determining whether an agent can benefit from scaling. The shift is from treating action spaces as middleware (an interface layer between the model and the environment, designed for convenience) to treating them as the architectural foundation (the primitive that determines what data can be aggregated, what skills can transfer, and whether scaling laws apply).

The magnitude of this shift is substantial but not revolutionary — it does not introduce new neural architectures or learning algorithms, but rather reorients the field's priorities around a design choice that was previously under-theorized. The paper's empirical demonstration that different action spaces have qualitatively different scaling properties (Figure 10, right panel: GUI spaces plateau while device-level spaces continue improving) changes the conversation from "what action space works best for my task?" to "what action space will work best as I scale?" This reframes action space design as a long-term investment decision rather than a per-project engineering convenience — analogous to how Hoffmann et al. (2022) reframed model size vs. data quantity as a joint optimization rather than independent choices.

The paper reconciles a tension that has been building in the agent literature. On one side, specialist agents (AlphaStar, VPT, Voyager) achieved impressive results through tight environment-specific integration — custom action spaces, API wrappers, per-game engineering. On the other side, generalist architectures (Gato, SIMA) demonstrated that multi-domain training is possible but fell short of specialist performance, partly because their action spaces were either heterogeneous (Gato's domain-specific tokenization) or insufficiently tested at scale (SIMA's limited game count). The implicit message from specialist work was "generality sacrifices performance"; the implicit message from generalist work was "generality is possible but unproven at scale." Game-TARS resolves this tension by showing that generality and performance are not in tension when the action space enables cross-domain data aggregation — the 2× improvement over specialist Minecraft agents (Table 3) while simultaneously generalizing to unseen FPS, web, and simulator environments demonstrates that the right action space makes generality a performance advantage, not a compromise.

The paper also redirects research attention away from search and planning over high-level action abstractions and toward learning sensorimotor primitives from raw interaction data. Prior Minecraft agents like Voyager (Wang et al., 2023a) and DEPS (Wang et al., 2023b) invested heavily in hierarchical planning — decomposing tasks into subtasks, generating code for each subtask, executing via APIs. Game-TARS shows that when the model has sufficient sensorimotor experience (500+ games, 20k hours), complex behaviors emerge from end-to-end visuomotor policies without explicit hierarchical planning. The corner-peeking and projectile-dodging behaviors observed in VizDoom (Section 4.2.3) were never explicitly programmed or rewarded — they emerged from the interaction between diverse training data and a unified action space. This suggests that the field may have been over-investing in planning architectures and under-investing in sensorimotor data scale, a finding that parallels the "Bitter Lesson" dynamic the paper explicitly invokes (Section 4.3.1).

A less obvious but potentially important shift concerns data collection methodology. The Think-Aloud pipeline (Section 2.2) demonstrates that high-quality agent training data can be collected by recording humans playing games normally — no specialized annotation interfaces, no action space definitions, no per-game engineering. This lowers the barrier to entry for multi-game data collection and makes it feasible for other groups to build on this paradigm. The paper's finding that a single-stage blended training approach outperforms a carefully staged curriculum (Section 4.1, negative result) further simplifies the training recipe: future practitioners can simply mix all available data and train, rather than designing complex multi-phase curricula.

However, the paper does not resolve the fundamental question of whether game-trained sensorimotor skills transfer to productivity software. The cross-source post-training (Section 3.5) and the scaling experiment showing that non-game data improves game performance (Figure 10, right panel: 65% → 72%) provide suggestive evidence for bidirectional transfer, but the absence of any non-game evaluation leaves the "general computer-use agent" vision as a hypothesis rather than a demonstrated capability. This is the most significant open question the paper raises: does the unified action space paradigm extend beyond games, or is there something special about games (dense reward signals, clear success criteria, bounded interaction patterns) that makes this approach work?

Follow-Up Research This Work Enables

Directly evaluate Game-TARS — or a similarly trained model — on standard GUI and code benchmarks to test the bidirectional transfer hypothesis. The paper demonstrates that cross-domain agent data (code, GUI, research) improves game performance by +7 percentage points (Figure 10: 65% → 72%). The reverse question — does game pre-training improve code/GUI task performance? — is untested but is the critical validation for the "general computer-use agent" claim. A strong follow-up would take the Game-TARS-MoE-mini checkpoint, fine-tune it on a standard GUI benchmark like OSWorld (Xie et al., 2024) or Mind2Web (Deng et al., 2023), and compare against: (a) the same base VLM (Seed-VL-1.5) fine-tuned on the same GUI data without game pre-training, and (b) a state-of-the-art GUI agent like UI-TARS-1.5. If game pre-training provides a sensorimotor foundation that accelerates GUI learning, the game-pre-trained model should achieve higher accuracy with the same fine-tuning budget or equivalent accuracy with less fine-tuning data. Conversely, if game pre-training provides no benefit (or negative transfer) to GUI tasks, the layered skill acquisition hypothesis is weakened, and the paper's contribution would be more accurately scoped to game agents specifically rather than general computer use.

Characterize the functional form of the cross-game scaling relationship with controlled data volume experiments. The paper's scaling analysis (Figure 10, right panel) has only three data points with confounded volume and diversity. A rigorous follow-up would fix data composition (e.g., pure game trajectory data from a fixed set of 100 games) and systematically vary the volume of that specific data type, measuring Minecraft success rate at 5–8 logarithmically spaced volume levels (e.g., 10B, 20B, 50B, 100B, 200B, 500B tokens). Fitting a power law S(N) = S_0 × N^α would provide the scaling exponent α and enable extrapolation to larger volumes — answering whether another 2× data would produce another +10 percentage points or another +2. This experiment is computationally expensive (requiring multiple full pre-training runs) but is the minimum necessary to move from "positive trend" to "scaling law." The paper's existing infrastructure (500B token training pipeline, MCU evaluation benchmark) makes this experiment directly executable by the authors.

Ablate the Think-Aloud pipeline against offline-synthesized ReAct data at matched data volume. The paper argues that native, in-situ reasoning (captured via Think-Aloud) is causally superior to retrospectively synthesized reasoning (Section 2.2), but provides no experiment isolating this contribution. A clean ablation would train two models: one on the full data mixture (including 127B Think-Aloud tokens), and one where the 127B Think-Aloud tokens are replaced with an equivalent volume of offline-synthesized ReAct data (using the same base trajectories, so only the reasoning source differs). If the Think-Aloud model substantially outperforms the synthetic-ReAct model on tasks requiring multi-step reasoning (Minecraft crafting, 2048 gameplay), the paper's causal fidelity argument is validated and the expensive Think-Aloud pipeline is justified. If performance is comparable, the community can adopt the cheaper offline synthesis approach without sacrificing quality. This experiment also tests a specific mechanistic hypothesis: that the visual-anchor cursor alignment (preventing inverse dynamics model degeneration) is load-bearing — if synthetic-ReAct data (which lacks this alignment) produces IDM-like failures, the alignment step is a necessary quality-control measure; if not, it may be unnecessary engineering.

Evaluate whether Sparse Thinking produces genuinely emergent meta-cognition or merely reproduces training-data reasoning density. The paper's qualitative claim that "the agent develops an implicit meta-cognition mechanism: it chooses when to reason and when to act reflexively" (Section 4.3.2) is supported by aggregate performance numbers (Table 4: Sparse Thinking outperforms both No-thinking and Greedy Thinking) and anecdotal trajectory observations. A stronger test would measure the correlation between the agent's reasoning decisions and task-optimal reasoning density across a range of environments with known optimal reasoning schedules. For each task, define the "reasoning necessity" at each timestep (e.g., via the action-only model's prediction error, as in Equation 7), then measure whether the agent's actual reasoning production correlates with this necessity score. If the agent reasons at high-necessity moments and suppresses reasoning at low-necessity moments, this is evidence for learned meta-cognition. If the agent reasons at a roughly constant rate regardless of necessity, the sparse thinking is simply a fixed pattern learned from training data statistics rather than adaptive behavior. A negative result here would not invalidate the sparse thinking approach (fixed sparse reasoning is still more efficient than greedy reasoning) but would clarify that the meta-cognitive framing overstates what has been learned.

Test whether the unified action space enables zero-shot transfer to completely novel input modalities — specifically, whether Game-TARS can control a computer given only a task description and no action demonstrations. The paper's evaluations all provide the agent with the game's action space definition via System Prompt (e.g., "W = move forward, Space = jump"). This assumes the action mapping is known at deployment. A more challenging test of generality: deploy Game-TARS on a novel application (e.g., a spreadsheet program, a CAD tool, a video editor) with only a natural language task description and NO key binding specification. The agent would need to explore the action space — trying keys, observing their effects, and inferring the mapping — before it can complete the task. This tests whether the model has learned a general action-effect inference capability (can it figure out what keys do by trying them?) or merely a mapping-following capability (can it execute actions given a pre-specified mapping?). The Automatic Action Space Augmentation training (Section 3.1) was designed to teach prompt-conditioned action selection, but it assumed the mapping was provided in the prompt — the exploration setting would test a more fundamental capability that the paper does not currently evaluate.

Stress-test the decaying loss on domains with different action repetition statistics to determine whether γ = 1/2 is universally appropriate or domain-dependent. The paper uses a fixed decay factor γ = 1/2 for all games, justified by the observation that action repetition dominates across game types. But the degree of repetition varies — an FPS has more frequent action switches than a crafting game, and a racing game has different repetition patterns than a puzzle game. A study that systematically varies γ across game types and measures both the non-repetitive accuracy improvement and any degradation in action persistence learning would establish whether the decaying loss needs to be tuned per domain or whether γ = 1/2 is a robust default. If the optimal γ varies substantially, future work should develop adaptive decay schemes that estimate the repetition rate from data statistics and adjust γ accordingly. If γ = 1/2 is near-optimal across diverse domains, the decaying loss becomes a reliable off-the-shelf component for imitation-learned agents — a finding with broad applicability beyond gaming.

Practical Applications and Downstream Use Cases

Game testing and quality assurance at scale. Game-TARS demonstrates the ability to play hundreds of diverse games without per-game engineering. This directly enables automated game testing — deploying the agent to play through new game builds, attempting to complete tasks, and flagging situations where it gets stuck, encounters bugs, or finds unreachable objectives. The agent's 72% success rate on embodied Minecraft tasks (Table 3) and near-human performance on unseen web games (Figure 7) suggest it can serve as a first-pass filter for game QA, handling routine testing while human testers focus on edge cases and creative exploration. The key advantage over scripted testing bots is generality: the same agent can test a racing game, an FPS, and a crafting game without per-game test script development. The 600-step evaluation budget (Section 4.2.2) translates to roughly 30–60 seconds of gameplay per task at 10–20 actions per second — sufficient for rapid regression testing across many game builds. The primary barrier to deployment is inference cost: the MoE-mini model's per-step latency and token consumption (37 tokens/step with Sparse Thinking, Table 4) must fit within the testing budget for the application to be cost-effective relative to human testers.

Training data generation for specialist game agents via distillation. The paper positions Game-TARS as a generalist foundation model, but a practical intermediate application is using it to generate high-quality training data for specialist agents. For a game where human demonstration data is scarce or expensive, Game-TARS can be deployed to play the game thousands of times, generating diverse trajectories that include both successful and unsuccessful attempts. These trajectories — annotated with the agent's sparse reasoning — can then be used to train a smaller, faster specialist model via behavioral cloning or RL fine-tuning. This is the "distillation" direction the paper hints at in the Conclusions. The scaling results (Figure 10, right panel) suggest that adding more diverse data improves performance, so using Game-TARS to augment limited human data with synthetic trajectories could accelerate specialist agent development. The 127B tokens of Think-Aloud data required ~20k annotator hours (based on the 20k game hours cited in Section 4.1); replacing some of this with model-generated trajectories could dramatically reduce the cost of training new game agents. The key risk is that model-generated data may contain systematic errors or limited behavioral diversity compared to human data, and the paper provides no analysis of how well Game-TARS trajectories capture the diversity of human play.

Accessibility agents for players with motor disabilities. Game-TARS's unified keyboard-mouse action space maps directly to the same input devices used by human players, but the agent operates these inputs through inference rather than physical movement. This creates an opportunity for assistive technology: a player with limited motor control could provide high-level instructions ("craft an iron pickaxe," "defeat that enemy") while Game-TARS executes the low-level keyboard and mouse actions needed to accomplish the goal. The agent's instruction-following capability (validated on the MCU benchmark with 800+ diverse natural language tasks, Table 3) and its ability to adapt to arbitrary key bindings (via Automatic Action Space Augmentation, Section 3.1) mean it can be configured to work with the player's preferred input setup. The 2000+ step long-term memory (Section 3.4) enables the agent to maintain context across extended gameplay sessions without the player needing to repeatedly re-specify goals. This application would require integration with real-time input routing software (to inject the agent's keyPress and mouseMove outputs into the game) and careful latency management — the paper's web game evaluation used time-pausing (Section 4.2.3 footnote), suggesting inference latency is currently too high for real-time assistive use, but model optimization or hardware acceleration could close this gap.

Research platform for studying generalization in visuomotor policies. Game-TARS provides a unique testbed for studying what transfers across environments and what doesn't. Because the model is trained on 500+ games with a unified action space, researchers can systematically probe which game features enable or prevent transfer: Does pre-training on 2D platformers transfer to 3D platformers? Does FPS experience transfer to third-person shooters? Does experience with realistic graphics transfer to minimalist geometric environments (as the MiniWorld results in Figure 9 suggest)? The paper's evaluation on unseen environments (Poki web games, VizDoom, MiniWorld) demonstrates that transfer occurs, but the underlying mechanisms — shared visual features? shared motor patterns? shared causal structure? — are uncharacterized. A research program that trains Game-TARS variants on controlled subsets of games and measures transfer to systematically varied test environments could reveal the conditions under which visuomotor skills generalize, providing both scientific insight into sensorimotor learning and practical guidance for data collection (which games provide the most transferable skills?). This platform is particularly valuable because it operates at a scale (500+ games, 500B tokens) where transfer effects are large enough to measure reliably, avoiding the noise and instability that plague smaller-scale transfer studies.