Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

🎯 Pitch

An AI agent trained solely on Genshin Impact can suddenly complete a 100-minute mission in a completely different game, Wuthering Waves, without any fine-tuning. This zero-shot transfer from a single 3D world challenges the assumption that broad generalization requires broad training data.

1. Executive Summary

This paper introduces Lumine, an open recipe and prototype model for building generalist agents capable of completing hours-long missions in real time within 3D open-world video games, using Genshin Impact as the primary testbed built on a Qwen2-VL-7B-Base model. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner through three training stages—large-scale behavioral pre-training on raw gameplay (1731 hours of image–action pairs), instruction-following fine-tuning for language grounding (200 hours), and reasoning fine-tuning with a hybrid thinking strategy (adaptively entering a thinking mode to produce inner-monologue reasoning only when necessary, rather than at every step)—paired with a sliding-window context management system that preserves up to 20 recent frames as short-term memory and reasoning steps as long-term memory. Lumine completes the full five-hour Mondstadt main storyline with human-level efficiency (56 minutes for Act I vs. 78 minutes for fresh human players), achieves over 80% success across diverse short-horizon benchmark tasks spanning collection, combat, NPC interaction, and puzzle-solving, and demonstrates strong zero-shot cross-game generalization—completing 100-minute missions in Wuthering Waves and the five-hour first chapter of Honkai: Star Rail without any fine-tuning—establishing that visuomotor and reasoning competencies acquired from a single game environment can transfer to entirely distinct worlds and interaction dynamics only when the underlying control primitives (3D navigation, 2D GUI manipulation) share structural commonalities.

2. Context and Motivation

The Core Problem: We Don't Know How to Build Generalist Agents in Open-Ended 3D Worlds

The fundamental question this paper tackles is deceptively simple: how do you build an autonomous agent that can perceive, reason, and act at human-level across diverse, long-horizon tasks in rich 3D open-world environments? This matters because the field of AI has a long history of building agents that master specific games—Atari, Go, Dota II, StarCraft II—but these systems operate under fundamentally constrained conditions that make them brittle and non-transferable. The paper's first section explicitly frames this gap:

"Despite these achievements, such agents remain confined to closed environments, optimizing a single, well-defined objective with explicitly shaped rewards via large-scale reinforcement learning in a perceive-then-act paradigm. This specialization yields mastery but brittle intelligence with limited abstraction, weak transfer, and poor adaptability to the ambiguity and diversity inherent in open-world scenarios."

This is not just a matter of academic curiosity. Commercial video games—particularly AAA titles like Genshin Impact—represent the kind of rich, dynamic, multi-modal environments that agents must eventually master if they are to operate in real-world settings. These games feature realistic physics, hundreds of hours of content, diverse gameplay mechanics that must be composed and sequenced, and a natural progression from simple to complex challenges that mirrors human learning curricula. Yet they remain largely unexplored by the research community because they are closed-source, lack API access, and present genuine engineering challenges around perception, real-time control, and long-horizon planning.

Six Interlocking Challenges

The paper systematically identifies six core challenges that together define why building agents for these environments is so difficult (Section 1). Each is individually significant, but their interlocking nature makes the problem qualitatively harder than any one in isolation:

1. Scalable Environments. The environment itself must be rich enough to challenge agents while remaining standardized and reproducible. Prior research environments like the Arcade Learning Environment (Bellemare et al., 2013), DeepMind Lab (Beattie et al., 2016), or AI2-THOR (Kolve et al., 2017) provide controlled settings but lack the compositional complexity and scale of commercial games. The paper argues that Genshin Impact's combination of open-world exploration, real-time combat, puzzle-solving, NPC dialogue, character progression, and crafting systems—all integrated into a seamless world that takes weeks to fully explore—makes it an ideal testbed that prior environments cannot match.

2. Multimodal Perception. Agents must fuse and interpret heterogeneous sensory streams simultaneously. In Genshin Impact, this means processing 3D embodied vision (overworld navigation, enemy detection), 2D GUI elements (menus, dialogue boxes, inventory screens, the minimap), and textual information (quest descriptions, NPC names, tutorial prompts). This is qualitatively different from prior work in both robotics—where vision-language-action (VLA) models typically handle a single visual modality with limited textual overlay—and GUI agents—which focus almost exclusively on 2D interfaces while ignoring 3D navigation. The paper notes that commercial games saturate both axes simultaneously, demanding a unified perception system that no prior agent has demonstrated.

3. High-Level Planning. Long-horizon planning in open worlds differs fundamentally from planning in closed environments. There is no single reward function to optimize; objectives shift dynamically as the storyline progresses, and the agent must balance competing priorities—following the main quest versus exploring side content, fighting versus evading, collecting versus progressing. The paper emphasizes that this requires "self-motivated, long-horizon plans that adapt as environmental dynamics evolve, incorporating reflection and self-revision to refine strategies." Prior prompt-based agents (e.g., Voyager, Cradle) achieve some of this through explicit reasoning, but they operate at extremely low frequencies and cannot handle real-time constraints.

4. Low-Level Control. Abstract intentions must be grounded into precise keyboard and mouse actions operating at 30 Hz. This is far more demanding than typical GUI automation, where agents can "teleport" the cursor to absolute positions and issue coarse-grained commands like "click" or "type." In 3D games, mouse movement controls the camera through relative displacement—the trajectory matters, not just the endpoint. Keyboard inputs involve complex patterns: key-down and key-up events, held keys (sprinting, charging), rapid sequential combinations (skill chains), and simultaneous multi-key actions. The paper observes that prior approaches model these in "code-like formats, resulting in inefficiencies," and no prior system has achieved both the semantic expressiveness and the temporal precision needed for real-time 3D control.

5. Memory. Open-world missions span hours and involve navigation across vast terrains where the agent must remember objectives, recognize previously visited locations, and maintain context for consistent decision-making. The paper identifies this as requiring both short-term memory (maintaining awareness of immediate surroundings and recent actions) and long-term memory (retaining mission objectives, learned strategies, and environmental knowledge). Prior VLA models in robotics operate reactively—consuming only the current observation and producing a single action—which fundamentally limits temporal coherence. Prompt-based agents maintain longer context but do so inefficiently through periodic summarization, incurring substantial latency.

6. Real-Time Inference. Vision-language models are computationally expensive. A 7B-parameter model must process high-resolution images, generate reasoning tokens, and produce executable actions—all within the 200ms control cycle dictated by a 5 Hz perception rate. The paper notes that "it usually takes GUI agents several seconds to produce a single executable action in an autoregressive manner," making them entirely unsuitable for fast-paced game environments where missing a timing window means mission failure. This latency challenge has prevented VLM-based agents from being deployed in any real-time interactive setting prior to Lumine.

Where Prior Approaches Fall Short

The paper positions itself against three broad families of prior work, each of which addresses some but not all of these challenges:

Traditional RL Agents (System 1 Only). The lineage from DQN (Mnih et al., 2015) through AlphaStar (Vinyals et al., 2019) and OpenAI Five (Berner et al., 2019) to Gran Turismo (Wurman et al., 2022) represents a paradigm where agents learn policies from scratch via reinforcement learning in a "perceive-then-act" fashion—what the paper calls "system 1 style." These agents achieve superhuman performance but exhibit two critical limitations that make them unsuitable for open-world generalization. First, they optimize a single, well-defined reward function and cannot flexibly switch objectives based on natural language instructions or changing mission contexts. Second, they learn within environments that provide API access to internal states, meaning they never have to handle raw pixel input, partial observability, or the ambiguity of real visual scenes. Table 1 in the paper makes this explicit: DQN operates in closed environments on ~5-minute tasks with no multimodal understanding, no instruction following, no reasoning, and API-based interaction. The skill is real but the intelligence is narrow.

Prompt-Based LLM/VLM Agents (System 2 Dominant). The rise of large language and vision-language models enabled a different approach: use the model's pre-trained knowledge and reasoning capabilities to plan and decide, interfacing with the environment through hand-crafted skill APIs. Voyager (Wang et al., 2023) demonstrated this in Minecraft, achieving ~20-minute autonomous exploration through stepwise reasoning. Cradle (Tan et al., 2024) pushed further, completing ~1-hour missions in Red Dead Redemption 2 using keyboard-and-mouse control. These systems exhibit task composition, reflection, and contextual reasoning over extended periods—capabilities that traditional RL agents entirely lack. However, they suffer from three fundamental weaknesses:

They cannot generate precise low-level actions. They rely on predefined skill primitives (e.g., turn(degree), move_forward(duration), attack()) that abstract away the continuous control problem. This works for high-level planning but fails in situations requiring fine-grained manipulation—precise mouse aiming, timing-sensitive dodge rolls, or navigating through narrow gaps.
They are too slow for real-time interaction. The paper notes that API-based baseline models (GPT-5, Gemini 2.5 Pro, Grok4) "can take as long as 30 seconds to return" a single action, requiring the game to be paused during inference (Section 7). This eliminates the possibility of handling time-sensitive combat, platforming, or any scenario requiring reactions faster than several seconds.
They cannot recognize fine-grained visual patterns. Prompt-based agents rely on the VLM's general visual understanding, which is often insufficient for detecting small game-specific items (e.g., a specific flower among dense foliage), reading UI elements at small font sizes, or tracking rapid visual changes during combat. The paper's own baseline evaluation (Figure 10) shows these models achieving near-zero success on simple collection and combat tasks that Lumine handles at >80%.

The paper's Table 1 comparison is instructive: Cradle achieves a 1-hour task horizon with multimodal understanding and instruction following, but only with stepwise reasoning (not continuous), no real-time inference (game must pause), and code-format actions that are verbose and inefficient. Voyager achieves only 20-minute horizons and cannot handle mouse-keyboard input at all.

Data-Driven VLA Models (System 1 + Language, But Short Horizon). The most recent paradigm—exemplified by RT-2 (Zitkovich et al., 2023), OpenVLA (Kim et al., 2024), π₀ (Black et al., 2024), and CombatVLA (Chen et al., 2025)—fine-tunes VLMs on action data to create models that directly map observations to actions while following language instructions. This approach successfully grounds control in language and achieves robust performance on diverse short-horizon tasks. However, existing VLA models are fundamentally limited in task horizon. The paper notes that even the most advanced robotic VLAs handle tasks lasting "only a few seconds to minutes." SIMA (Raad et al., 2024), the closest prior work to Lumine in using human-like interfaces across multiple games, achieves only 10-second task horizons with no reasoning capability. JAVIS-VLA (Li et al., 2025) extends to game environments but also remains at the ~10-second level.

The critical limitation these models share is the absence of reasoning and memory. They operate in "a purely reactive, single-step manner that consumes only the current observation" (Section 2). This works for short tasks where the goal is continuously visible and no planning is needed—"pick up the red block" in robotics, "click the submit button" in GUI tasks—but collapses on long-horizon missions where objectives change, progress requires remembering previous steps, and the agent must actively decide what to do next rather than simply following an immediate instruction.

A Missing Training Recipe. Beyond these capability gaps, the paper identifies a methodological problem: prior data-driven agents "typically rely on pretraining with high-quality instruction-following or reasoning datasets, annotated by human labelers, but this leaves open the risk for continued scaling." Human annotation is expensive—the paper itself spent considerable resources annotating only 165 hours of instruction data and 15 hours of reasoning data, a tiny fraction of the 2424 hours of raw gameplay collected. If building generalist agents requires human-labeled data for every new game or domain, the approach cannot scale. The paper explicitly frames this as the motivation for its pre-training stage: "our approach focuses on exposing models to diverse in-game dynamics by primarily training on image-action pairs without additional labels," arguing that raw behavioral data—abundant and cheap to collect—can teach fundamental visuomotor competence that subsequent language grounding then channels toward specific objectives.

How Lumine Positions Itself

The paper explicitly frames Lumine as a unified recipe that addresses all six challenges simultaneously rather than treating them as separable problems (Section 1, Section 2 closing paragraph). Its positioning can be understood through three design choices that differentiate it from each prior paradigm:

Against traditional RL agents: Lumine replaces the "optimize a single reward" paradigm with language-grounded, instruction-following control that can flexibly switch between objectives. It uses a perception-reason-action loop that incorporates explicit reasoning rather than purely reactive policies. It operates through the standard human interface (raw pixels → keyboard/mouse) rather than requiring API access to internal game states.

Against prompt-based agents: Lumine generates actions directly rather than invoking predefined skill functions, enabling fine-grained control at 30 Hz with precise mouse movements and key combinations. It achieves real-time inference through a comprehensive optimization stack (25.3× speedup), eliminating the need to pause the game. It learns domain-specific visual recognition through large-scale behavioral pre-training rather than relying solely on general VLM capabilities. And critically, it uses a hybrid thinking strategy—generating explicit reasoning only at key decision points while operating reactively the rest of the time—which the paper argues "avoids redundant computation and latency without compromising decision quality" (Section 1). This is a middle ground between the "always reason" paradigm of prompt-based agents and the "never reason" paradigm of traditional VLA models.

Against existing VLA models: Lumine extends the task horizon from seconds to hours by incorporating the reasoning and memory mechanisms that VLA models lack. The paper's three-stage training curriculum explicitly builds this: pre-training teaches action primitives (like existing VLAs), instruction-following grounds them in language (like existing VLAs), but the third reasoning stage adds the explicit planning and reflection capabilities that enable long-horizon autonomy. The sliding-window context management (20 frames of short-term memory + reasoning steps as long-term memory) provides temporal coherence without requiring specialized memory architectures.

Perhaps most significantly, Lumine is positioned as an open recipe rather than a one-off model. The paper emphasizes that its contributions span "the entire lifecycle of developing generalist agents in 3D open world environments, from environment selection, data collection and preprocessing to interaction frequency, model design, training procedures, and inference optimization" (Section 9). This framing is deliberate: the goal is not just to build one agent for Genshin Impact, but to establish a methodology that can be applied to any commercial game or interactive 3D environment, demonstrated through the zero-shot transfer results to Wuthering Waves and Honkai: Star Rail.

3. Technical Approach

3.1 Reader Orientation

Lumine is a 7-billion-parameter vision-language model that has been trained through a three-stage curriculum to function as an autonomous game-playing agent — it takes raw pixels from a game screen as input and autoregressively produces keyboard and mouse actions as output, optionally interleaving explicit reasoning ("inner monologue") at key decision points. The system solves the problem of building a generalist agent that can complete hours-long missions in real time within 3D open-world games by decomposing the challenge into three sequential capabilities: (1) learning fundamental visuomotor control from large-scale unlabeled human gameplay, (2) grounding those motor skills in natural language so the agent can follow instructions, and (3) teaching the agent to generate its own goal-directed reasoning so it can plan and adapt autonomously over long horizons — all while operating at the 30 Hz interaction frequency needed for real-time 3D gameplay.

The "shape" of the solution is a single end-to-end model that unifies perception, reasoning, and action within a single autoregressive generation loop, trained on data that becomes progressively more semantically rich at each stage (raw actions → instruction-action pairs → reasoning-action trajectories), with a hybrid thinking mechanism that lets it skip explicit reasoning when current plans are still valid and only "think" when circumstances change.

3.2 Big-Picture Architecture

The Lumine system has five major components organized around the model and its data:

The Vision-Language Backbone (Qwen2-VL-7B-Base) — a pre-trained 7B-parameter model that processes sequences of raw pixel images and generates text. It serves as the foundation for all three training stages and is never modified architecturally; all capabilities are imparted through data and training alone.
The Observation Encoder (ViT + Projector) — the vision transformer and projection layers inherited from Qwen2-VL that convert 1280×720 (720p) game screenshots into visual tokens consumed by the language model. This component operates at 5 Hz (one frame every 200 ms) and is frozen during the final reasoning stage.
The Action Decoder — the language model head that autoregressively generates action strings in a compact custom format. Each action string specifies mouse displacement ΔX ΔY ΔZ followed by six 33ms chunks of key presses (yielding 30 Hz control from 5 Hz model outputs), all represented as semantically meaningful tokens that the base VLM can naturally interpret (e.g., "Shift W" for dashing, "F" for interaction).
The Hybrid Thinking Controller — a learned decision mechanism (not a separate module, but a behavior acquired during the third training stage) that determines whether to emit <|thought_start|>...<|thought_end|> tokens before the action string. When the current situation continues from the previous plan, the model emits only actions; when context changes (new objectives, unexpected obstacles, task completion), it generates explicit reasoning.
The Sliding-Window Context Manager — an inference-time mechanism that maintains up to 20 recent image–action pairs in a first-in-first-out queue as short-term memory, plus the most recent reasoning step as long-term guidance. When new reasoning is generated, the entire context is flushed and re-accumulated from that point onward.

Information flows as follows: the game client captures a screenshot → the server receives it, preprocesses it to 720p, and passes it through the ViT → visual tokens enter the language model along with the current context window (system prompt + previous reasoning + up to 20 image–action pairs) → the model autoregressively decides whether to generate reasoning tokens, then generates an action string → the action string is streamed back, with each complete 33ms chunk executed immediately upon generation via keyboard/mouse events → the cycle repeats at 200ms intervals.

3.3 Roadmap for the Deep Dive

First, the observation and action spaces, because all downstream training and inference depend on the precise definitions of what the model sees and what it outputs — the 5 Hz perception rate, the 720p resolution, the six-chunk action format with 33ms granularity, and the tokenization scheme for keyboard and mouse inputs.
Second, the three-stage training curriculum, explained sequentially — pre-training (raw image→action, 1731 hours, teaching visuomotor primitives), instruction-following (instruction+image→action, 200 hours, grounding control in language), and reasoning (image+memory→thought+action, 15 hours, enabling autonomous planning) — because each stage builds on the previous one and the design choices at each stage depend on what was learned earlier.
Third, the data curation pipeline, which is the engineering backbone of the entire recipe — how 2424 hours of raw human gameplay are collected, processed, filtered, and progressively annotated to produce the three training datasets, including the label-then-augment strategy for instruction data and the keyframe-based reasoning annotation protocol.
Fourth, the hybrid thinking mechanism, since it is the core innovation enabling real-time autonomy — how the model learns to decide when to reason, what reasoning looks like (the three-part structure: previous step summary, current situation analysis, next move planning), and why this matters for latency.
Fifth, the real-time inference optimization stack, which achieves the 25.3× speedup that makes the whole system practically deployable — covering communication streaming, KV-cache management with StreamingLLM, speculative decoding, tensor parallelism, quantization, and CUDA graph optimization.
Sixth, the context management strategy, which is the "memory" component that enables long-horizon coherence — the 20-frame sliding window, the FIFO eviction policy, the context flushing on new reasoning, and how this implements both short-term and long-term memory without specialized architectures.

3.4 Detailed, Sentence-based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a generalist game-playing agent can be built by training a single VLM through three sequential stages — behavioral pre-training on raw gameplay, instruction-following fine-tuning, and reasoning fine-tuning — without modifying the model architecture or vocabulary, and that this approach produces transferable visuomotor and reasoning competencies that generalize to unseen games.

Observation Space: What the Model Sees

Lumine perceives the game world through raw pixel screenshots captured at 1280×720 resolution (720p). This resolution is explicitly chosen to "balance UI text legibility and computational efficiency" (Section 4) — it is high enough that in-game text, dialogue options, quest markers, and small inventory icons remain readable by the vision encoder, but low enough that the ViT can process each frame within the strict latency budget.

Frame rate. The model processes one frame every 200 milliseconds, yielding a perception frequency of 5 Hz. The paper justifies this rate by reference to human visual reaction time: "roughly 200–250 ms" (citing Woods et al., 2015). This means Lumine perceives the world at roughly human speed — it does not have superhuman reflexes, and it cannot react to events that unfold faster than 200ms (e.g., frame-perfect dodges in Souls-like combat). This is a deliberate design choice that aligns the agent's temporal capabilities with human demonstrators, ensuring that the training data (collected from humans with the same ~200ms reaction time) is on-distribution.

What is NOT in the observation. Lumine does not receive any privileged game state information — no internal coordinates, no health values as numbers, no enemy AI states, no inventory contents as structured data. Everything it knows about the game world must be inferred from pixels. This includes reading its own health bar (a visual element that can be transparent, partially occluded, or rapidly changing during combat), parsing quest text from dialogue boxes, and recognizing enemies by their visual appearance. The paper also notes that "internal states and reward signals are usually inaccessible" in commercial games (Section 3), making reinforcement learning from environment rewards infeasible — a constraint that motivates the behavioral cloning approach.

Temporal asynchrony. An important practical detail: the visual input Lumine receives is always approximately 200ms old by the time it reaches the model, due to network transmission and preprocessing latency. The paper acknowledges this "asynchrony issue" (Section 6.2) and notes that it "can introduce side effects in highly time-sensitive scenarios." Models trained without history (single-frame input) are particularly vulnerable to this — they observe the pre-execution state of the previous action and incorrectly believe the action hasn't taken effect, leading to overshooting.

Action Space: Keyboard and Mouse Modeling

Lumine's action space is a custom textual representation that encodes both mouse movement and keyboard/mouse button states into a compact string format. The design is motivated by three constraints that prior approaches fail to satisfy simultaneously:

The actions must be semantically interpretable by a VLM — the base Qwen2-VL model has been pre-trained on natural language, so representing actions as natural-language tokens (e.g., "W" for forward movement, "Shift" for sprinting, "F" for interaction) allows it to leverage its existing knowledge of what these keys do in game contexts, rather than learning entirely de novo action mappings.
The actions must be compact enough for real-time generation — prior approaches that represent actions as code-like function calls (e.g., move_forward(duration=0.2), turn(angle=45)) are verbose and require many tokens per action, making them too slow for high-frequency control.
The actions must capture fine-grained temporal dynamics — the distinction between tapping a key (rapid press-release), holding a key (sustained press across multiple chunks), and releasing a key is essential for gameplay (sprinting vs. walking, charging a skill vs. quick-casting) but is lost in most prior action representations.

Action format. Each action string is enclosed by <|action_start|> and <|action_end|> tokens. Internally, it consists of:

Mouse displacement: ΔX ΔY ΔZ where ΔX and ΔY are relative pixel displacements as integers in the range (-1000, 1000), and ΔZ is the scroll wheel amount as an integer in [-5, 5]. The mouse values are discretized: during data processing, raw mouse movements are quantized to units of 5 pixels along the X-axis and 4 pixels along the Y-axis, reducing the effective vocabulary size and simplifying learning. The scroll values represent the number of scroll steps, with positive values meaning scroll up.
Six key press chunks: Following the mouse displacement, a semicolon-separated list of six action chunks, each corresponding to a 33ms interval. An action chunk K_t specifies zero to four keys (including keyboard keys and mouse buttons) that are simultaneously pressed during that 33ms window. Keys not listed in the chunk are implicitly released at the start of that interval. Keys that appear in consecutive chunks maintain their key-down state across chunk boundaries without being re-pressed.

The 30 Hz interaction frequency. The paper's key insight for achieving 30 Hz control from a 5 Hz model is action chunking (borrowed from the robotics literature, specifically Zhao et al., 2023). At each 200ms step, the model predicts six consecutive 33ms action chunks. The model outputs tokens at 5 Hz, but the game client executes actions at 30 Hz by interpolating the six chunks within each 200ms window. This means Lumine's effective interaction frequency matches the frame rate of most games (30 or 60 fps), even though its decision frequency is only 5 Hz.

Chunk execution via streaming. A critical inference optimization makes this work in practice: the system uses "a streaming output mechanism for timely action execution" (Section 6.2). As soon as a complete chunk is generated — signaled by a terminating semicolon — it is immediately transmitted and executed, without waiting for the entire six-chunk sequence to finish. This dramatically relaxes the timing constraints: each chunk must be generated within 33ms, but the full sequence can take up to 200ms. The paper reports that "the delay before producing the first action chunk is approximately 110 ms, well below the 200 ms threshold" (Section 6.2, Table 4), and even the worst-case chunk (four keys + semicolon) generates in only 12 ms.

Key tokenization. Each keyboard key and its preceding space are represented as a single token to minimize decoding steps. For example, " Shift" is one token, " W" is one token. Keys that would normally require multiple tokens — function keys F1–F12 and numeric keys 0–9 — are remapped to single-token forms: F1 becomes "One", F2 becomes "Two", 1 becomes "one", 2 becomes "two", etc. Mouse buttons are mapped to "LB" (left), "RB" (right), "MB" (middle). The full mapping is provided in Appendix Table 6. This tokenization is a practical engineering choice that significantly reduces the number of autoregressive decoding steps per action, directly translating to lower latency.

Why not absolute mouse positioning? The paper explicitly argues against the absolute positioning approach used by most GUI agents (where the model outputs screen coordinates and the cursor "teleports" there). In 3D games, mouse movement controls the camera through relative displacement — the game engine accumulates relative movements over time to determine camera rotation. The trajectory between two points matters (e.g., drawing gestures, smooth tracking of a moving target), not just the endpoint. Additionally, Genshin Impact "periodically re-centers the mouse cursor to the middle of the screen during overworld gameplay without the player's awareness" (Appendix A), making absolute positions meaningless. The paper's data processing system detects whether the current frame is an overworld scene (3D) or a GUI interface (2D) and applies either relative movement summation (for 3D scenes) or absolute position differencing (for GUI scenes) accordingly.

Example action. The paper provides a concrete example: "92 0 0 ; Shift W ; Shift W ; Shift W ; F W ; F W ; F". This represents: first, turn the camera right by 92 pixel units (mouse displacement 92 right, 0 down, 0 scroll); then, in the first three 33ms chunks, hold Shift and W simultaneously (sprinting forward); in the next two chunks, hold F and W (moving forward while pressing the interaction key); in the final chunk, only F (interact while stopped). This sequence captures a common gameplay pattern: sprint toward an object, then interact with it upon arrival.

Three-Stage Training Curriculum: Overview

The paper employs a multi-stage training procedure where "each stage builds on the previous one and the design choices at each stage depend on what was learned earlier" (Section 5). The core insight is that the capabilities needed for autonomous gameplay — visuomotor control, language grounding, and goal-directed reasoning — are best acquired sequentially rather than jointly, because each capability depends on the prior one being already established. Jointly training on all data would require the model to simultaneously learn low-level motor skills and high-level planning, which the paper's empirical results suggest is less effective.

The three stages, in order, are:

Pre-training (Stage I): Learn to act from raw observations across diverse scenarios. The model sees image→action pairs (no language, no reasoning) and learns fundamental visuomotor primitives — walking, turning, interacting with objects, opening menus, basic combat. Output: Lumine-Base.
Instruction Following (Stage II): Ground the learned motor skills in natural language. The model sees instruction+image→action triplets and learns to execute specific, language-described objectives (e.g., "collect the red flower ahead," "talk to the NPC on the left"). Output: Lumine-Instruct.
Reasoning (Stage III): Enable autonomous planning and long-horizon decision-making. The model sees image+memory→thought+action sequences and learns to generate its own internal goals and plans, deciding when to reason explicitly and when to act reactively. Output: Lumine-Thinking.

Each stage uses the model from the previous stage as its initialization, creating a curriculum where capabilities accumulate rather than being learned from scratch each time. The ViT backbone is kept frozen during the final reasoning stage to prevent catastrophic forgetting of visual features.

History vs. non-history variants. The paper trains and evaluates models in both "non-history" (single-frame input, each sample is one image→action pair) and "history" (multi-turn input, each sample is 20 interleaved image–action pairs) settings. The history models consistently outperform their non-history counterparts, particularly on tasks requiring temporal coherence (navigation, GUI manipulation, tracking moving targets), but are computationally more expensive to train (e.g., pre-training takes 19,008 GPU-hours for history vs. 5,376 for non-history, a 3.5× increase).

Pre-Training: Learning Visuomotor Primitives from Raw Gameplay

Design philosophy. The pre-training stage is the most distinctive element of Lumine's recipe. Rather than starting with expensive human-annotated instruction data (as prior VLA models do), the paper argues that "raw video-action data are naturally abundant and straightforward to collect" (Section 5.1). Human annotation is "both costly and difficult to scale," while "automatic labeling using VLMs remains unreliable due to their limited long-horizon understanding and insufficient domain knowledge." Moreover, open-world exploration inherently contains "seemingly irregular behaviors" — players wandering aimlessly, making suboptimal decisions, recovering from mistakes — that "capture valuable corner cases that significantly enhance the model's robustness and generalization."

Data volume and filtering. The raw dataset consists of 2,424 hours of human gameplay from 70 annotators. This is filtered to 1,731 hours by applying two rule-based operations: (a) removing segments where the player is idle (no meaningful actions), which constituted the vast majority of removed data — the paper states "95% of idle actions" are removed; and (b) removing clips "dominated by camera jitter" (rapid, small, purposeless camera movements). The filtering is intentionally minimal — the paper explicitly wants to preserve diverse, sometimes noisy behaviors because they improve robustness.

Data format. In the non-history setting, each training sample is a single image paired with its corresponding action string. In the history setting, each sample is 20 interleaved image–action pairs structured as a multi-turn dialogue (the format that Qwen2-VL was pre-trained on). The history setting provides temporal context — the model sees not just what the current frame looks like, but what actions it just took and what the game looked like before those actions — which is essential for learning state-dependent behaviors like "if I just pressed W and the screen didn't change, I'm probably against a wall and should turn."

Knowledge preservation. A critical detail: to prevent catastrophic forgetting of the base model's general visual and language capabilities during this action-centric training, the paper mixes in "an approximately 20% mixture of multimodal web data to retain general knowledge." This is approximately 10.8 billion tokens of web data mixed with 37.1 billion tokens of Genshin Impact pre-training data (Figure 4). Without this mixture, the model would likely overfit to the visual distribution of Genshin Impact and lose its ability to understand diverse scenes, which would harm transfer to new games and degrade instruction-following in the next stage. The paper also reserves a small amount of instruction-following data during pre-training for evaluation purposes (to track whether the base model is developing capabilities that will support the next stage).

Training configurations. The pre-training stage uses an initial learning rate of 2e-5 for the LLM and 7e-6 for the ViT, with a constant learning rate schedule, gradient norm clipping at 1.0, the AdamW optimizer with β1 = 0.9, β2 = 0.95, ε = 1e-8, a warm-up ratio of 0.05, batch packing length of 32,768 tokens, batch size of 128, and 1 training epoch for non-history (yielding the best performance; further epochs overfit) vs. 3 epochs for history (where multiple epochs continue to improve due to the richer temporal signal). Training uses 64 H100 GPUs, taking 3.5 days (5,376 GPU-hours) for non-history and 12.4 days (19,008 GPU-hours) for history.

Emergent capabilities. The paper's scaling analysis (Section 8.1, Figure 9b) provides a fascinating window into what the pre-training stage actually learns, as a function of data volume. The capabilities emerge in a predictable order that reflects their difficulty:

Immediate interaction (< 100 hours): Object interaction emerges first. The model learns to press F when near interactable objects (picking flowers, opening chests, talking to NPCs). This is the simplest behavior because the visual cue (a prompt "Press F to interact" appearing on screen) is directly predictive of the required action.
Extended interaction (~1,000 hours): Basic combat and GUI manipulation become reliable. The model learns to switch characters to chain skill combos, execute ranged attacks against elevated targets, and handle common GUI events (moving the cursor to select dialogue options, clicking the close button, choosing Confirm/Cancel buttons). These require more complex temporal reasoning — GUI navigation involves multi-step sequences (hover, then click), and combat involves recognizing what character is currently active and which enemies are on screen.
Game mechanics (> 1,800 hours): The model begins to recognize game-specific puzzle elements (elemental monuments, torch-lighting puzzles, pressure plates) and exhibit contextually appropriate reactions. However, the paper notes these remain challenging because "their sparse occurrence in raw gameplay, coupled with the wide variety present in Genshin Impact, makes these mechanics significantly harder for agents to master."
Navigation (starting ~1,000 hours, solidifying by > 1,800 hours): Robust navigation takes the most data. The model develops "robust road sense: they tend to follow the in-game roads to proceed, avoid obstacles such as trees and walls, halt at cliff edges, use the wind glider to prevent fall damage, and follow quest markers efficiently." This is the most complex behavior because it requires integrating multiple sub-skills: camera control to orient toward the quest marker, sustained forward movement, reactive obstacle avoidance, and periodic checks of the minimap or quest marker to maintain bearing.

The paper notes that "this reliable navigation ability establishes a solid foundation for longer-horizon tasks" — without it, the instruction-following and reasoning stages would have nothing to build on, because the agent needs to be able to physically reach objectives before it can plan which objectives to pursue.

Instruction-Following Stage: Grounding Control in Language

Design motivation. The pre-training stage produces a model that can act competently but has no mechanism for directing its behavior toward specific goals. It tends to follow its learned behavioral priors — for example, "instinctively interact with nearby objects, NPCs, and enemies, yet struggles with situations that require breaking its current behavioral inertia, such as navigating to specific target locations" (Section 5.2). Natural language instructions bridge this gap by providing explicit goal specifications that can override local behavioral priors.

The label-then-augment strategy. The core challenge for instruction-following data is scale: human annotation is expensive, but automatic labeling using VLMs is unreliable because they lack domain knowledge about game mechanics and objectives. The paper's solution is a clever two-phase approach:

Phase 1: Human annotation for classifier training. Human annotators label 165 hours of gameplay, identifying the start and end timestamps of 38 predefined activity categories within 20-second clips. The categories are organized hierarchically (Appendix B): Game Scene (Overworld, Domain, GUI Interface) → Game Content (Exploration, Collection, Combat, Puzzle, etc.) → Fine-grained activity (Following a Seelie, Activating Elemental Monuments, Opening a vine-wrapped chest, etc.). This annotation is used to fine-tune a Qwen2-VL-2B classifier that processes five consecutive frames and outputs a category label. The classifier achieves 83.04% accuracy (Table 7) — sufficient for downstream use because the subsequent GPT-4.1 verification step catches misclassifications.

Phase 2: Automatic labeling and instruction generation. The trained classifier is applied to all 2,424 hours of raw gameplay, assigning a category label to every frame. The paper then identifies "transition points between adjacent gameplay segments that are assigned with different labels by the classifier, typically indicating a shift in task context or objective." Around each transition point, a 20-frame (4-second) snippet is extracted and sent to GPT-4.1, which generates context-aware instructions based on the category labels. GPT-4.1 serves dual roles: it generates diverse natural language instructions and acts as a verifier, discarding samples when it detects inconsistencies between the visual content and the classifier-assigned label.

This approach transfers the generalization capability of large VLMs (GPT-4.1's ability to describe scenes in natural language) into the action domain without requiring the VLM to understand the action space itself. The result is 200 hours of high-quality instruction-following data — a 1.2× augmentation over the 165 hours of human annotation, achieved by using the classifier to find task boundaries and GPT-4.1 to describe them.

Data format. In the non-history setting, each training trajectory is decomposed into single-step samples, where each sample includes an instruction, an image, and the corresponding action. In the history setting, training is performed on the entire trajectory sequence (up to 20 frames), preserving temporal coherence. The instructions are generated in Chinese (the language of the game client and annotators), and the system prompts (Appendix E) provide detailed formatting rules.

Training configurations. The instruction-following stage uses a learning rate of 2e-5 for the LLM and 7e-6 for the ViT, with a cosine learning rate schedule, gradient norm clipping at 1.0, the same AdamW optimizer as pre-training, batch packing length of 32,768 tokens, batch size of 128, 2 training epochs (for non-history) and 3 epochs (for history). Training uses 32 H100 GPUs, taking 1.3 days (960 GPU-hours) for non-history and 2.2 days (1,664 GPU-hours) for history. The data is mixed with multimodal web data in the same 20% ratio as pre-training, and a minimal portion of action-only data is retained to preserve behavioral diversity.

What instruction-following achieves. The paper demonstrates that Lumine-Instruct achieves over 80% success across all four benchmark categories on simple tasks (Figure 10), representing a 61% performance gain over the base model. The largest improvement occurs in combat tasks, where the instruct model doubles the success rate of the base model by ensuring the agent actively seeks and opens treasure chests after battles (a behavior the base model often neglects, instead just moving forward aimlessly). This validates the paper's claim that language grounding provides goal-directedness that purely behavioral pre-training lacks.

The importance of pre-training. An ablation in Figure 11 shows that training only on instruction-following data (without pre-training) yields substantially worse performance, particularly on hard tasks. The paper notes an exception: the model without pre-training performs slightly better on NPC Interaction tasks, suggesting that "the pre-training data introduces a notable bias, where many trajectories involve players merely passing by NPCs without engaging in interaction." This is a revealing detail — it shows that behavioral pre-training on raw data picks up human biases (like ignoring NPCs unless there's a reason to interact) that must be explicitly corrected by instruction-following data.

Reasoning Stage: Enabling Autonomous Long-Horizon Planning

Design motivation. The instruction-following stage produces a model that can execute specific, human-provided instructions but cannot autonomously decide what to do next. For long-horizon missions — where objectives change dynamically, the environment evolves, and the agent must adapt its plans — explicit reasoning is necessary. The paper's approach is to train the model to generate inner monologue: first-person thoughts that articulate the rationale behind actions, enabling the model to maintain goal-directed behavior without external guidance.

Data curation. The reasoning dataset is the most carefully curated and smallest of the three stages, consisting of only 15 hours of gameplay, containing 15,000 reasoning traces with an average interval of 3.2 seconds between consecutive thoughts. The paper selects Act I of Genshin Impact's Mondstadt main storyline (Prologue: Act I — The Outlander Who Caught the Wind, approximately one hour of gameplay) as the data source, sampling 27 gameplay videos where human players engaged in this specific mission.

Annotation protocol. Annotators are provided with consecutive 10-second clips and instructed to identify key decision points at the frame level, then write first-person thoughts using a three-part structure (Appendix C):

Previous Step Summary: A concise reflection on what just happened and why it leads to a new line of thought (e.g., "I just defeated the slimes and the chest appeared").
Current Situation Analysis: An immediate interpretation of the current scene, focusing on essential elements like UI hints, mission description, mechanisms, NPCs, or enemies (e.g., "the quest marker is pointing northeast toward Mondstadt").
Next Move Planning: The player's adjusted plan expressed in natural inner voice (e.g., "I should follow the path toward the city gate").

Each reasoning sequence contains an average of 37.4 ± 11.7 tokens, achieving "a balance between detail and brevity" — long enough to capture meaningful planning but short enough to generate quickly. The annotation principles (Appendix C) emphasize naturalness and intentionality: thoughts should "sound natural and intention-oriented," not mechanical or overly literal, and should "provide guidance for the next 5–20 seconds of gameplay."

Hybrid thinking design. The critical innovation is that the model is trained to reason only when necessary, not at every step. The system prompt (Appendix E) explicitly instructs: "If the current situation continues from the previous analytical strategy, directly output the actions; only when the situation changes significantly, previous analysis becomes invalid, or new targets appear, you should provide necessary reasoning and output your thought process."

This is realized in the training data: most samples follow the instruction-following format (image + previous reasoning → new action), with reasoning appearing only at transition points. In the non-history setting, "most samples follow the same format as instruction-following data, where the model receives the previous reasoning and current visual observation as input. Only a small subset of data contains segments that generate new reasoning" (Section 5.3). In the history setting, trajectories start from the first frame after generating new reasoning and end upon the next reasoning generation, or earlier if they exceed 20 frames.

Forced reasoning recovery. At inference time, the paper implements a safety mechanism: "thinking models will be forced to enter thinking mode when it does not generate a new reasoning for more than 100 steps, which is useful to help it recover from a stuck state" (Section 8.3). This prevents the agent from looping indefinitely in unproductive behavior without reassessing its situation.

Why not always reason? The paper argues that constant reasoning is "computationally inefficient and prone to hallucinations in continuous, high-frequency control settings" (Section 2). In a real-time game, generating 40+ tokens of reasoning at every step would add hundreds of milliseconds of latency and likely cause the agent to miss timing-critical interactions. The hybrid approach means reasoning generation — which takes approximately 234ms for the first action chunk with reasoning versus 114ms without (Table 4) — occurs only at the 3.2-second average interval, keeping overall latency manageable.

Training configurations. The reasoning stage uses a learning rate of 1.83e-5 for the LLM (non-history) or 1.64e-5 (history), with the ViT frozen, a cosine learning rate schedule, gradient norm clipping at 1.0, the same AdamW optimizer, batch packing length of 32,768 tokens, batch size of 64, and 3 training epochs. Training uses 64 H100 GPUs, taking only 1 hour (64 GPU-hours) — the reasoning dataset is small enough to train quickly despite using twice the GPU count. The ViT is frozen to prevent the small reasoning dataset from distorting the visual representations learned during pre-training.

What reasoning achieves. The impact is dramatic: Lumine-Thinking with history achieves 93.4% overall success on the five subtasks of Act I (Table 5), compared to 66.8% for Lumine-Instruct and 53.4% for Lumine-Thinking without history. The instruct models "often fail to complete the full task chain" because "distractions encountered along the way, such as puzzles or enemies, cause the model to lose track of the main objective. Once disoriented in the overworld, the model tends to wander aimlessly and struggles to recover." In contrast, "thinking models are able to reflect and set appropriate goals for the current context and remain focused." Lumine-Thinking completed the full one-hour Act I in 56 minutes, versus 78 minutes for fresh human players and 53 minutes for expert human players.

Data Curation Pipeline: From Raw Gameplay to Training Datasets

The data pipeline (Figure 5) transforms 2,424 hours of raw gameplay recordings into three training datasets through a multi-stage process that progressively adds semantic richness.

Raw data collection (Appendix A). Seventy qualified annotators played Genshin Impact with brand-new accounts, progressing through the entire Mondstadt storyline and achieving over 80% map exploration. The data collection software standardizes the recording environment: all annotators played at 1080p with 100% scaling in Seamless Fullscreen mode, using only the four system-provided characters (Traveler, Amber, Kaeya, Lisa) with default key bindings and game settings.

Keyboard and mouse logging. The recording system captures both absolute cursor positions (via Win32 API SetWindowsHook for low-level input events) and relative mouse displacements (via DirectX DirectInput API polling every 5ms). This dual approach is necessary because neither alone suffices: absolute positions are unreliable in 3D overworld scenes where the game re-centers the cursor, while relative displacements are affected by Windows Enhance Pointer Precision (which applies nonlinear acceleration to mouse movement). For non-GUI scenes, the system sums relative movements between consecutive frames; for GUI scenes (detected via template matching), it uses absolute position differences.

Post-processing (Appendix A.4). The raw recordings are processed into standardized frame–action pairs through: (a) temporal alignment — synchronizing video frames with input logs by comparing timestamps and discarding misaligned initial segments; (b) keyboard state reconstruction — maintaining a full keyboard state and simulating the sequence of key-down and key-up events chronologically; (c) mouse movement discretization — quantizing to units of 5 pixels (X-axis) and 4 pixels (Y-axis); (d) key filtering — discarding keys not in the predefined mapping (Table 6); and (e) accuracy verification — replaying processed data segments in the game and comparing against original footage (the paper notes that faithful reproduction is possible over "short durations of a few seconds," but "over longer durations of more than ten seconds, or during sequences involving large camera rotations, a certain degree of drift emerges due to stochastic in-game mechanics").

Pre-training data. The processed frame–action pairs are filtered to remove 95% of idle actions and clips dominated by camera jitter, yielding 1,731 hours of data. No additional labeling is applied — the actions serve as their own supervision targets.

Instruction-following data. A 165-hour subset receives human annotation (38 activity categories, start/end timestamps within 20-second clips). This trains the Qwen2-VL-2B classifier, which labels all 2,424 hours. Transition points between categories are identified, 20-frame snippets are extracted and sent to GPT-4.1 for instruction generation, and the same action filtering as pre-training is applied, yielding 200 hours.

Reasoning data. From the 27 gameplay videos of Act I, annotators identify keyframes, write three-part inner monologues (previous summary, current analysis, next plan), and no action filtering is applied — the model must learn to wait appropriately at critical decision points. This yields 15 hours with 15,000 reasoning traces.

Context Management: Implementing Memory Without Specialized Modules

Lumine's memory system is implemented entirely through the model's context window, without any separate memory modules, databases, or retrieval mechanisms. The design uses a sliding-window mechanism with two types of memory distinguished by their retention policies (Section 6.1, Figure 6).

Short-term memory. The model maintains up to 20 recent image–action pairs in a multi-turn dialogue format within its context. These pairs are organized as alternating user (image) and assistant (action) turns, matching the conversational structure the base VLM was pre-trained on. This provides the model with explicit knowledge of what it just saw and what it just did — essential for temporal coherence and for avoiding the "overshooting" problem where the model incorrectly believes its previous action hasn't taken effect.

Long-term memory. The most recent reasoning step is preserved at the beginning of the context, after the system prompt. This reasoning serves as the "current objective" — it articulates what the agent is trying to accomplish and why, providing guidance that persists even as individual image–action pairs are evicted. When new reasoning is generated, it replaces the previous reasoning in this privileged position.

Eviction policy. A first-in-first-out (FIFO) policy is applied to the image–action pairs: when the number of pairs exceeds 20, the oldest pair is discarded. Crucially, the system prompt and reasoning are never evicted — they are always retained at the beginning of the context. This means the model always knows its current goal, even if it has forgotten the specific visual details of how it got to its current location.

Context flushing on new reasoning. When the model generates new reasoning, the entire context is flushed and re-accumulated from that point onward. This is a deliberate design choice: the old image–action pairs are relevant to the previous goal, not the new one. Flushing prevents the model from being confused by outdated visual context and ensures that the context window quickly fills with information relevant to the new objective. The paper notes that "the mechanism can be easily extended to maintain multiple reasoning segments if needed" (Section 6.1), suggesting that more sophisticated memory architectures (e.g., a hierarchy of reasoning goals with associated contexts) could be explored in future work.

Window length tradeoffs. The paper experiments with different window lengths for the instruction-following model (Figure 14) and finds that performance peaks at 10 frames in context and starts to decrease with more frames. The paper hypothesizes that this is related to data distribution — after action filtering "removes 95% noop and trivial movements, the data segments typically less than 20 frames, making it harder for the model to learn long-term dependencies effectively at longer range." The reasoning model uses 20 frames as its maximum, balancing the benefits of temporal context against the risk of distribution shift.

StreamingLLM interaction. To maintain efficiency with long contexts, the paper uses StreamingLLM (Xiao et al., 2023), which anchors an attention sink to the system prompt to prevent attention collapse when early tokens are evicted. The paper notes that "we observed a performance degradation in long-horizon tasks with the use of StreamingLLM, which is effectively alleviated by our context management strategy that clears the context when a new reasoning appears" (Section 6.2). This is an important detail: StreamingLLM alone degrades performance because the attention sink cannot perfectly preserve all historical information, but the periodic context flushing when reasoning changes resets the attention window to a state where all retained information is relevant, mitigating the degradation.

Real-Time Inference Optimization

Achieving real-time inference with a 7B-parameter VLM is a substantial engineering challenge. The paper implements a comprehensive optimization stack across communication, prefill, decoding, and infrastructure, achieving a 25.3× overall speedup (Figure 7) that reduces end-to-end latency from 3,655ms (baseline) to 129.8ms (fully optimized).

Communication optimizations (6ms network latency). The system maintains a persistent TCP connection to avoid repeated handshaking costs. Images are compressed to JPEG and Base64-encoded before transmission to reduce payload size. The streaming output mechanism means action chunks are executed as they are generated, without waiting for the full sequence.

Prefill optimizations. Historical key-value (KV) caches from prior interactions are maintained and reused, avoiding recomputation of attention for tokens that haven't changed. When the context window is saturated and early turns must be dropped, the StreamingLLM technique is applied: an attention sink is anchored to the system prompt, enabling a stable attention window even as earlier tokens are evicted. The vision encoder processes 1196 visual tokens in 39ms (fully optimized, Table 4).

Decoding optimizations. The compact action tokenization (one token per key press) minimizes the number of decoding steps — the average action chunk requires only 1.8 tokens and 1.02 forward steps (due to speculative decoding). The paper implements draft-model-less speculative decoding (Leviathan et al., 2023) that leverages the fixed delimiters in the action format: "we observe that ΔX and ΔY end with a space, ΔZ and K1, ..., K5 end with a semicolon, and K6 ends with the <|action_end|> token. We use a simple state variable to track the generator's stage to dynamically select the appropriate delimiter as a draft token." Standard rejection sampling ensures the final output matches the original distribution. This reduces the average forward steps per token from 1.0 to approximately 0.57 (1.02 steps for 1.8 tokens).

Infrastructure optimizations. Four key techniques are applied: (a) tensor parallelism across 4 NVIDIA H20 GPUs (degree 4, assigning one KV head per GPU since Qwen2-VL-7B has 4 KV heads); (b) W8A8 quantization using SmoothQuant (Xiao et al., 2023) for both the ViT and LLM prefill stages, reducing computation and memory bandwidth; (c) search-based tuning of GEMM kernels for the ViT, prefill, and decode stages, plus a custom one-shot all-reduce kernel for communication during decoding; and (d) CUDA graph capture that fuses the forward pass and rejection sampling process into a single graph, cutting CPU overhead during speculative decoding.

Latency breakdown. Table 4 provides precise timing: the first action chunk without reasoning takes 113.9ms total (6ms network + 6.8ms preprocessing + 39ms vision encoder + 52ms LLM prefill + ~8.4 tokens of decoding), well under the 200ms control cycle. With reasoning, the first chunk takes 234.0ms, which exceeds the 200ms threshold — but because reasoning occurs infrequently (every 3.2 seconds on average), "we empirically observe no noticeable impact on the visual experience, maintaining smooth and stable gameplay." The average action chunk takes 3.1ms to generate, and even the worst case (4 keys + semicolon) takes only 12.4ms, both comfortably under the 33ms per-chunk deadline.

4. Key Insights and Innovations

Innovation 1: Behavioral Pre-Training as the Primary Stage — Explicitly Rejecting the Instruction-First Paradigm

The most intellectually distinctive design choice in Lumine is the inversion of the standard VLA training curriculum. Prior data-driven agents — OpenVLA, RT-2, π₀, SIMA, CombatVLA, JAVIS-VLA — all follow a pattern where the primary pre-training stage uses instruction-following data: human-annotated (observation, instruction, action) triplets that teach the model to map language-grounded goals to motor outputs. This is the natural approach when coming from the language-modeling tradition — you have a foundation model that understands language, so you use language as the supervisory signal for action learning.

The paper explicitly argues this standard approach "leaves open the risk for continued scaling" because "human annotation is both costly and difficult to scale, while automatic labeling using VLMs remains unreliable due to their limited long-horizon understanding and insufficient domain knowledge" (Section 5.1). This is not merely an observation about cost — it is a claim about the nature of the skill hierarchy. The paper's implicit argument, supported by the scaling analysis in Figure 9b, is that visuomotor competence is foundationally independent of language and must be learned first, at scale, from raw behavioral data. Language can then be layered on top to direct and constrain the already-established motor repertoire, rather than being the primary medium through which motor skills are acquired.

This is a fundamental shift, not a refinement. The dominant assumption in VLA research has been that pre-trained VLMs already possess sufficient visual and linguistic understanding that the main challenge is teaching them to act — and that instruction-following data provides the most direct bridge. Lumine's position is different: the VLM's visual understanding is generic (internet images) and insufficient for domain-specific visual recognition (identifying specific game items, reading UI elements), and its motor repertoire is zero. Pre-training on raw gameplay teaches both simultaneously — the model learns to recognize game-specific visual patterns because they predict actions, and it learns actions because they are contingent on visual patterns. The instruction-following stage then adds a new conditioning variable (language) to an already-functioning visuomotor policy, rather than attempting to build the policy from scratch using language as scaffolding.

The evidence for why this matters comes from Figure 11: training only on instruction-following data without behavioral pre-training yields substantially worse performance, especially on hard tasks. The base model cannot effectively follow instructions to do things it doesn't already know how to do — language grounding channels existing competence rather than creating it. This echoes a finding from the robotics literature (e.g., RT-2's observation that internet-scale VLM pre-training helps but is insufficient without robot-specific data), but Lumine provides the cleanest demonstration in the game-playing domain by showing what happens when you skip the behavioral pre-training entirely.

A subtler implication: the paper shows in Figure 9b that different visuomotor capabilities emerge at predictably different data scales — object interaction at <100 hours, combat and GUI at ~1,000 hours, navigation and game mechanics at >1,800 hours. This is an emergent capability curve for behavioral pre-training, analogous to the scaling laws observed for language models, but for motor skills rather than perplexity. The paper doesn't formalize this as a law, but the empirical pattern — capabilities emerge in order of difficulty, with navigation (the most complex, requiring integration of multiple sub-skills) requiring the most data — suggests a predictable structure that could guide data collection for future domains. If you're building an agent for a new game, this pattern tells you which capabilities will appear first and how much data you'll likely need for each.

Innovation 2: Hybrid Thinking as a Learned Behavior Rather Than an Architectural Module

The field of embodied AI has oscillated between two extremes for incorporating reasoning. At one pole: hierarchical architectures (e.g., Figure's Helix, GR00T N1, Hi Robot) where a high-level reasoning model operates at low frequency (seconds) and a separate low-level policy model executes at high frequency (milliseconds), with information flowing unidirectionally from planner to executor. This provides temporal abstraction but introduces non-stationarity — the low-level policy must adapt every time the high-level model's outputs change, making joint optimization unstable. At the other pole: the ReAct paradigm (Yao et al., 2023) where the same model interleaves reasoning and action at every step, producing a thought then an action in lockstep. This is simple and end-to-end trainable but "can be computationally inefficient and prone to hallucinations in continuous, high-frequency control settings" (Section 2).

Lumine's hybrid thinking is a third paradigm: the model learns, through data, to decide when to reason rather than reasoning at every step or delegating reasoning to a separate module. This is realized as a behavior acquired during the third training stage, not as an architectural change — the model emits <|thought_start|> tokens when it determines that the current situation warrants explicit planning, and skips straight to <|action_start|> when the existing plan remains valid. The paper explicitly frames this as drawing inspiration from "hybrid thinking" in LLMs like Claude Plays Pokémon and Qwen3, where the model can flexibly decide whether to engage System 2 reasoning or default to System 1 reactivity.

What makes this intellectually distinctive is that it collapses the hierarchical architecture into a single model while preserving its computational benefits. A hierarchical system achieves efficiency by running the expensive planner rarely and the cheap executor frequently — but at the cost of architectural complexity and optimization difficulty. Lumine achieves the same efficiency pattern — reasoning at ~3.2-second intervals, actions at 200ms intervals — within a single autoregressive generation loop, by training the model to implicitly learn when its current plan has become stale (new objectives, unexpected obstacles, task completion) and only then generating new reasoning tokens.

The significance goes beyond performance. This is a unifying conceptual move: rather than treating reasoning as a separate module that must be integrated with a reactive controller, or as a mandatory step in every decision cycle, Lumine treats reasoning as one possible output modality among others — something the model can emit when useful and suppress when not. This is philosophically closer to how humans operate: we don't verbalize our goals at every step when walking down a hallway, but we do when we reach a junction and need to decide which way to turn. The model learns this pattern from data showing exactly this behavior — long stretches of reactive action punctuated by occasional explicit reasoning at decision points.

The evidence that this works as intended comes from the contrast between Lumine-Thinking and Lumine-Instruct on long-horizon missions (Table 5). The instruct model fails not because it lacks motor skills — it has the same pre-training and instruction-following data — but because it cannot self-generate goals and thus gets distracted, wandering aimlessly once the immediate context no longer implicitly specifies what to do. The thinking model succeeds because its reasoning explicitly maintains goal state: "I need to follow the quest marker to Mondstadt" persists as long-term memory even as individual frames are evicted from the sliding window. The fact that Lumine-Thinking generates 593 reasoning instances with only an 8.8% error rate (Figure 18) over a full hour of gameplay shows that the hybrid mechanism is both parsimonious (reasoning only when needed) and reliable (reasoning is mostly correct when generated).

Innovation 3: Action Tokenization That Preserves Semantic Interpretability Without Sacrificing Efficiency

Prior game-playing VLM agents face a fundamental tension in action representation. At one extreme, code-format actions (e.g., Cradle's turn(degree), move_forward(duration), attack()) are semantically rich — the VLM can understand what these functions do from its pre-training — but are verbose (many tokens per action) and abstract away continuous control (you can't express "hold Shift for 67ms then tap F" in a single function call). At the other extreme, action tokens or action heads (e.g., RT-2's discretized action bins, π₀'s flow matching, CombatVLA's specialized heads) are compact and enable precise continuous control, but are semantically opaque — the model must learn what these tokens mean from scratch, discarding the VLM's pre-trained knowledge of what "Shift" or "W" or "F" does in a game context.

Lumine's action space is an elegant resolution: represent actions as a textual format using the same tokens the VLM was pre-trained on, but structured compactly enough for real-time generation. The key insight is that keyboard keys and mouse movements carry "clear semantic intent that is naturally interpretable by VLMs" (Section 2) — the model already knows from its pre-training that "W" means forward movement in games, that "Shift" means sprint, that "F" typically means interact. By representing actions using these semantically meaningful tokens, Lumine leverages the VLM's existing knowledge rather than forcing it to learn new action primitives from scratch. The compact formatting (each key press is one token, actions are six 33ms chunks in a single string) ensures that generation latency stays within the real-time budget — a design that is simultaneously more semantically grounded than code-format approaches and more efficient than raw token learning.

This is an incremental but practically crucial refinement of existing VLA action representations. The contribution is not the idea of textual actions per se (GUI agents have used text for actions), but the specific combination of (a) fine-grained temporal modeling via action chunking (six 33ms chunks per 200ms decision), (b) semantic interpretability via standard key names, and (c) compactness via single-token-per-key encoding, all within a format that requires no architectural modifications to the base VLM. The paper's observation that prior approaches "fail to exploit the inherent semantics of keyboard and mouse operations, which are well captured by LLMs" (Section 4) is the diagnostic insight: the field had been treating game actions as either opaque motor commands (to be learned from scratch) or high-level function calls (to be reasoned about abstractly), missing the middle ground where actions are both semantically meaningful and temporally precise.

The practical impact is substantial though hard to quantify in isolation — it's one component of the 25.3× speedup, but the paper's latency breakdown (Figure 7, Table 4) shows that the compact action format directly enables the speculative decoding optimization (since fixed delimiters between action components can be used as draft tokens), which yields a 1.2× speedup on its own. More importantly, the semantic interpretability likely contributes to the model's ability to generalize to new games: when Lumine encounters Wuthering Waves or Honkai: Star Rail, the action tokens "W", "Shift", "F", "E", "Q" carry the same semantic meaning they did in Genshin Impact, allowing visuomotor transfer even when the visual environment changes. The paper doesn't isolate this effect, but the zero-shot transfer results would be far less plausible if actions were represented as learned tokens with no cross-game semantic consistency.

Innovation 4: Zero-Shot Cross-Game Transfer as Evidence for Learned Transferable Meta-Skills

The paper's most striking empirical result is not the in-domain performance (completing a one-hour mission in 56 minutes) but the zero-shot generalization: Lumine, trained exclusively on Genshin Impact data, completes a 100-minute mission in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail — games it has never seen, with different visual styles, different combat systems (real-time vs. turn-based), different navigation mechanics (open-world vs. hub-based), and different UI conventions. This is a qualitatively different kind of transfer than what prior game-playing agents have demonstrated, and it points toward a specific hypothesis about what Lumine has actually learned.

Prior agents that achieve transfer (e.g., SIMA across multiple games, or VLA models across robot embodiments) do so because they were trained on data from the target domains — the generalization is within the training distribution. Lumine's transfer is genuinely zero-shot: the model receives no fine-tuning, no prompt modification, and no game-specific adaptation when deployed in new games. The paper argues this means Lumine has acquired "transferable meta-skills, such as 3D navigation and 2D manipulation, that can be readily applied to other domains" (Section 9).

The intellectual contribution here is not just "our model generalizes well" — it is the identification of which skills transfer and why . The paper's cross-game analysis reveals a pattern: performance degrades gracefully as the structural similarity between source and target games decreases. Wuthering Waves, which "shares many structural similarities in terms of control schemes, combat loops, and gameplay rhythm" (Section 8.3.2), sees near-in-domain efficiency. Honkai: Star Rail, a turn-based RPG with hub-based exploration rather than open-world navigation, sees more degradation but still succeeds — the 3D navigation skills partially transfer (the model can move through environments and interact with NPCs), while the combat system (which is fundamentally different) causes the greatest difficulty. Black Myth: Wukong, with realistic graphics, hub-based design, invisible walls, auto-hiding UI, and Souls-like combat, sees basic competence but cannot complete longer tasks.

This pattern suggests Lumine has learned structurally generalizable visuomotor primitives — not game-specific button mappings, but abstract patterns like "follow the quest marker on the minimap," "move toward interactable objects when a prompt appears," "select dialogue options to advance conversations," "navigate menus using directional inputs and confirm buttons." These are meta-skills that transcend any particular game's implementation, and they transfer because they are grounded in general visual patterns (quest markers, interact prompts, dialogue boxes, menu layouts) that recur across games. The paper's finding that Lumine sometimes uses Genshin Impact terminology to describe entities in Wuthering Waves (calling monsters "Hilichurls," the protagonist "Traveler") is revealing: the naming is wrong, but the behavioral response is correct — the model recognizes "this is an enemy" or "this is a quest NPC" even when it uses the wrong label. This is evidence that the visuomotor mapping is more robustly learned than the linguistic labeling, which makes sense given the training curriculum (1,731 hours of behavioral data vs. 15 hours of reasoning data with language).

The significance of this finding extends beyond games. If visuomotor meta-skills acquired through large-scale behavioral pre-training on a single rich environment transfer to structurally similar environments without additional training, then the path to generalist embodied agents may not require training on every possible environment — it may require identifying a sufficiently rich source environment that covers the space of transferable skills, then verifying that skills acquired there compose appropriately in new contexts. Lumine's recipe provides a concrete methodology for testing this hypothesis: pre-train on one game, evaluate zero-shot on others, identify which skills transfer and which don't, and use the gaps to guide future data collection. The paper doesn't formalize this as a research program, but it is the natural implication of the cross-game results.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on a custom-built benchmark of 141 language-conditioned tasks in Genshin Impact, organized into four categories—Collection (62 tasks), Combat (21 tasks), NPC Interaction (21 tasks), and Puzzle (23 tasks)—with three difficulty levels: simple, hard, and unseen (Section 7, Figure 8). For long-horizon evaluation, the paper uses the game's main storyline missions: Prologue Act I (in-domain, ~1 hour), Acts II and III (reasoning OOD, ~4 hours combined), and Chapter I Act I in Liyue (fully OOD, ~1+ hours), plus cross-game evaluation in Wuthering Waves, Honkai: Star Rail, and Black Myth: Wukong (Section 8.3).
Base model. All Lumine variants are built on Qwen2-VL-7B-Base (Wang et al., 2024), a 7-billion-parameter vision-language model. The paper states this model is chosen for its "strong multimodal understanding and generation capabilities gained from large-scale pretraining on diverse web data" (Section 4). Two sizes (2B and 7B) are explored during scaling analysis, with 7B selected as the primary base due to superior scaling behavior (Section 8.1, Figure 9a).
Metrics. Task success rate (%) is the primary metric, assessed through human evaluation of whether the agent completed the specified objective. Each task is run five times unless stated otherwise (Section 7). For the long-horizon storyline missions, the paper reports per-subtask success rates (Table 5) and total completion time compared to human players. For reasoning quality, the paper manually classifies reasoning errors into six categories and reports error rates (Figure 18). For atomic ability evaluation during pre-training, human raters assign scores on a 5-point scale from "complete failure" (1) to "full mastery" (5) (Section 8.1). The MATH benchmark is not used in this paper—this is a game-playing agent, not a math reasoning system.
Baselines. The paper benchmarks against five state-of-the-art VLMs integrated into the Cradle framework (Tan et al., 2024): GPT-5 (OpenAI, 2025), Gemini 2.5 Pro (Google, 2025), Grok4 (xAI, 2025), Doubao1.6-Vision (Volcengine, 2025), and Qwen3-VL-235B-A22B-Thinking (Qwen Team, 2025). These models interact through predefined skill functions in code format (e.g., turn(degree), move_forward(duration), attack()) and maintain up to five recent steps as historical context, which the paper states "we found empirically to yield the best performance" (Section 7). All thinking models use their default thinking budgets, and the game is paused during API inference since "API responses can take as long as 30 seconds to return" (Section 7).
Generation budget / compute accounting. For the benchmark tasks, all agents operate under identical early-game conditions with the same four starter characters at specified levels (Section 7). For baseline models, the compute metric is effectively wall-clock time (with game pausing during inference). For Lumine, the paper reports training compute in GPU-hours (Table 3) and inference latency in milliseconds (Figure 7, Table 4), but there is no unified FLOPs-based comparison across methods—the baseline models are much larger (GPT-5, Gemini 2.5 Pro, etc. are hundreds of billions of parameters vs. Lumine's 7B) and use fundamentally different interaction paradigms (code-format skills vs. direct keyboard/mouse actions), making direct compute-matched comparison impossible.
Cross-validation / statistical protocol. For the benchmark tasks, "each task is run five times" and results are averaged across runs (Section 7). For the storyline missions, each of the five subtasks in Act I is run three times (Table 5). For Acts II and III and cross-game missions, single runs are reported due to their length (4.7 hours and 5+ hours respectively). There is no mention of confidence intervals, standard errors, or formal statistical testing anywhere in the paper—a notable absence given that some comparisons involve small numbers of runs (e.g., 3 runs per subtask, single runs for OOD missions). The difficulty estimation and strategy selection cross-validation protocols described in the reference example (two-fold cross-validation within difficulty bins) are not applicable here, as Lumine does not use per-prompt difficulty estimation or adaptive strategy selection—it is a single model evaluated on fixed tasks.

Main Quantitative Results

Pre-Training Scaling and Capability Emergence

The paper investigates how Lumine-Base's performance scales with pre-training data volume using the non-history setting during the first epoch (Section 8.1, Figure 9a).

Headline result: The 7B model shows monotonically improving training loss and benchmark success rate through the full first epoch (~1,731 hours of Genshin data), while the 2B model's benchmark performance begins to degrade after approximately 1,200 hours despite continuing loss decrease, "revealing the limited volume of smaller models" (Section 8.1). Before the divergence point, "both models exhibit steadily decreasing training loss alongside consistent improvements in benchmark performance," with the 7B model "consistently achieving lower loss than the 2B model" (Figure 9a).

Capability emergence ordering (Figure 9b, human-evaluated 5-point scale). The paper traces how five atomic capabilities develop as a function of pre-training data volume for the 7B model:

Object interaction (<100 hours): Emerges quickly. "At this stage, agents can consistently gather nearby resources... and interact with NPCs in their vicinity by pressing key F." Reaches score 4 (high success rate with occasional mistakes) early.
Basic combat and GUI manipulation (~1,000 hours): Become "smooth and reliable." Agents "can switch characters to chain skill combos, execute ranged attacks against elevated or distant targets" and "manage to handle common basic GUI events, from moving the cursor to select dialogue options, to clicking the close button in the top-right corner." Reaches approximately score 3.5–4 by 1,800 hours.
Game mechanics (>1,800 hours): "Pose remarkable challenges." Agents "can recognize game-specific puzzle elements and exhibit reasonable reactions, but their sparse occurrence in raw gameplay, coupled with the wide variety present in Genshin Impact, makes these mechanics significantly harder for agents to master." Reaches approximately score 2.5–3 by the end of training.
Navigation (~1,000 hours, solidifying >1,800 hours): Requires the most data. By the end, agents "exhibit robust road sense: they tend to follow the in-game roads to proceed, avoid obstacles such as trees and walls, halt at cliff edges, use the wind glider to prevent fall damage, and follow quest markers efficiently." Reaches approximately score 3.5–4, though "it sometimes exhibits hesitation, inefficient stamina management and mistimed actions, leaving room for further improvement."
Visual guidance following (>1,800 hours): The paper lists this as a separate capability in the text but does not provide a distinct curve in Figure 9b; it appears closely tied to navigation.

Key insight: The paper does not provide explicit score values for each capability at each data point—Figure 9b is presented as a qualitative line graph without numbered gridlines, making precise numerical comparisons impossible from the published version. This is a significant limitation for reproducibility and quantitative analysis.

Instruction-Following Performance

Simple tasks (Figure 10). Lumine-Instruct-NonHis achieves over 80% success across all four benchmark categories on simple tasks, representing a 61% performance gain over Lumine-Base-NonHis:

"While Lumine-Base-NonHis already significantly outperforms all baselines, the instruct model further achieves a 61% performance gain, reaching over 80% success across all four categories."

The paper does not report the exact percentage for each category in the main text, but Figure 10 shows the instruct model's bars at approximately: Collection ~85%, Combat ~80%, NPC Interaction ~85%, Puzzle ~80%. In contrast, all five baseline VLMs (GPT-5, Gemini 2.5 Pro, Grok4, Doubao1.6-Vision, Qwen3-VL-235B) show near-zero success on Combat and Puzzle tasks, with only GPT-5 and Qwen3-VL achieving modest success on Collection (~10–20%) and NPC Interaction (~20–30%).

Hard tasks (Figure 11). Lumine-Instruct-NonHis exhibits consistent performance on NPC Interaction hard tasks ("even when surrounded by multiple non-target NPCs acting as distractors, it can still successfully engage with the designated NPC"), but performance drops substantially on Combat, Collection, and Puzzle:

"The model exhibits a zero success rate against the Eye of the Storm on elevated terrain, a flying enemy that lands only briefly... the elevated terrain further increases difficulty by punishing movement errors with the risk of falling."

The paper notes that "Performance on Puzzle tasks drops the most, reflecting their comprehensive requirement for the abilities above and emphasizing the importance of a deeper understanding of in-game mechanisms" (Figure 11).

Ablation: without pre-training (Figure 11). Models trained exclusively on instruction-following data without behavioral pre-training show "overall lower performance, particularly on hard tasks." However, interestingly, "the model achieves slightly better performance in NPC Interaction tasks," suggesting "the pre-training data introduces a notable bias, where many trajectories involve players merely passing by NPCs without engaging in interaction" (Section 8.2.1).

Unseen tasks (Figure 12). Lumine-Instruct-NonHis demonstrates "strong generalization capabilities in NPC Interaction, achieving performance comparable to in-domain settings." Collection sees a "moderate but acceptable performance drop" because "new collectibles must be recognized solely by their visual appearance." Combat "also declines, as Lumine-Instruct-NonHis is unfamiliar with previously unseen enemy attack patterns and combat mechanics." Exact success rates for unseen tasks are not reported numerically in the main text; they are presented as bar charts in Figure 12.

Complex instruction following (Figure 13). Four case studies demonstrate that more detailed instructions—"incorporating prior knowledge or task decomposition that breaks a complex objective into a sequence of manageable steps"—substantially improve success rates: collecting an Anemoculus floating in the air goes from 20% (general instruction) to 100% (detailed instruction with climbing strategy), collecting an Anemoculus on water goes from 0% to 20%, collecting Iron Ore goes from 40% to 100%, and collecting a chest within a Wind Barrier goes from 40% to 80%. These are single-case demonstrations, not aggregate statistics.

Benefits of History

Context window length sweep (Figure 14). For Lumine-Instruct evaluated on the full benchmark, performance peaks at 10 frames of historical context:

"models that preserve multiple historical frames and actions in the context achieve substantially higher performance across all tasks compared to those limited to a single frame. Performance peaks when the model maintains 10 frames in context and starts to drop with more frames in the context."

The paper hypothesizes this decrease "may be related to the data distribution: after the action filtering operation... the data segments typically less than 20 frames, making it harder for the model to learn long-term dependencies effectively at longer range." The exact success rate differences between 1, 5, 10, 15, and 20 frames are shown only as a bar chart in Figure 14 without numerical labels.

History vs. non-history comparison (Figure 15). Across the full benchmark, the history model (10 frames) "exhibits a clear advantage in Collection and Puzzle tasks, achieving overall better performance." The exact quantitative differences are not reported in the main text. On combat-related tasks, the history model actually shows lower performance in the error analysis—the paper reports the history model makes more low-level control errors than the non-history model (Figure 16, 12.9% vs. 5.0% of errors attributed to low-level control), explaining its worse combat performance.

Error analysis (Figure 16). Across the full benchmark, errors are categorized into five failure modes for both non-history and history instruct models:

Multimodal Understanding accounts for the largest share: 57.5% (non-history) and 50.0% (history), with the history model making significantly fewer object detection failures (30.0% vs. 43.8%) but more cross-modal conflicts (20.0% vs. 10.0%).
Instruction Following: 23.8% (non-history) and 22.9% (history), with behavioral inconsistency (losing track of objectives mid-task) being the dominant subcategory.
Spatial Understanding: 13.8% (non-history) and 14.3% (history), primarily distance estimation errors.
Low-Level Control: 5.0% (non-history) vs. 12.9% (history)—the history model makes notably more of these errors, which "explains its lower performance on combat-related tasks."

The error analysis is conducted on 80 total error cases (the sum of the non-history and history counts suggests approximately 80 and 70 cases respectively in Figure 16, though exact totals are not stated). The sample sizes for each error subcategory are small (some as low as 1–2 cases), making the percentage breakdowns potentially unreliable.

Reasoning Performance

In-domain mission: Act I (Table 5). Lumine-Thinking (history) achieves 93.4% overall success across the five subtasks, compared to 66.8% for Lumine-Instruct (history), 53.4% for Lumine-Thinking-NonHis, and 6.6% for Lumine-Instruct-NonHis. The per-task breakdown (3 runs each):

Model	Task 1	Task 2	Task 3	Task 4	Task 5	Overall
Lumine-Thinking (history)	3/3	2/3	3/3	3/3	3/3	93.4%
Lumine-Instruct (history)	2/3	3/3	2/3	2/3	1/3	66.8%
Lumine-Thinking-NonHis	1/3	2/3	2/3	2/3	1/3	53.4%
Lumine-Instruct-NonHis	0/3	0/3	0/3	1/3	0/3	6.6%

Completion time: Lumine-Thinking completed the full Act I in 56 minutes, compared to "fresh human players with an average of 78 minutes and expert human players with an average of 53 minutes" (Section 8.3.1). The expert group "had substantial prior experience with the game and had played through the same story segment at least once within the preceding week."

Reasoning error analysis (Figure 18). During the entire Act I mission, the history model generated 593 reasoning instances with an 8.8% error rate (52 errors), compared to the non-history model's 708 instances with a 14.0% error rate (99 errors). The most frequent error type in both settings is Perceptual Misrepresentation (36.4% of non-history errors, 40.4% of history errors)—"describes objects or scenarios that do not exist in the observation or misinterprets the character's current status." Other error types: Premature Completion (~16–17%), Causal Misjudgment (~15–19%), Self-Contradiction (~13–15%), Instruction Unfollowing (~8–10%), and Linguistic Collapse (9.1% in non-history, not observed in history).

OOD mission generalization: Acts II and III. Lumine-Thinking "successfully completed the two acts consecutively within 4.7 hours, compared to an average of 3.6 hours for expert human players" (Section 8.3.1). These missions are included in pre-training data but excluded from reasoning data. The paper identifies five key factors accounting for inefficiency: (1) lack of proactive fast travel via teleportation, (2) limited understanding of the minimap, (3) lack of proactive health recovery, (4) limited memorization (oscillating between multiple quest markers, abandoning detours), and (5) combat proficiency below expert human level. No quantitative ablation is provided to isolate the contribution of each factor.

Fully OOD mission: Liyue Chapter I Act I. Lumine-Thinking "demonstrates in-domain level efficiency during the first hour of gameplay" navigating from Mondstadt to Liyue Harbor, evading capture, and locating Tartaglia. It "even manages to find the Adeptus dwelling deep within the mountains, after a long journey across rugged terrain and rivers." However, a hallucination causes it to accidentally cancel quest tracking, leading to "aimless exploration of the map for approximately two hours" before it recovers and completes the mission. No quantitative success metrics or completion times are provided for this mission.

Cross-Game Generalization

Wuthering Waves (similar genre). Lumine "impressively demonstrated in-domain-level efficiency during the first two main storyline missions, First Resonance and Echoing Marche, where Lumine completes these missions in 107 minutes, compared to an average of 101 minutes for fresh human players" (Section 8.3.2). The paper notes that "the launch date of Wuthering Waves was later than the knowledge cutoff of Qwen2-VL, so Lumine is unlikely to leverage prior knowledge about this game from the base model pretraining." Observed errors include occasional misinterpretation of on-screen prompts (reading "press F" as "press E") and domain bias in reasoning language (using Genshin Impact terminology).

Honkai: Star Rail (different genre—turn-based). Lumine "successfully completed the entire first chapter, Today is Yesterday's Tomorrow, in the Herta Space Station, win the Boss fight of Doomsday Beast, cleared the Simulated Universe tutorial, and even progressed into the second chapter, reaching a new planet, Jarilo-VI, using 7 hours, compared to the 4.7 hours typically taken by fresh human players" (Section 8.3.2). The main bottlenecks are navigation (attempting to jump across gaps that can't be jumped—behavior that works in Genshin Impact but not in Honkai: Star Rail) and combat (the turn-based system requires targeted skill usage and ultimate confirmation that Lumine's Genshin-trained reflexes do not handle). Against the Doomsday Beast boss, "Lumine chose to lower the game's difficulty and, by the narrowest margin, secured victory"—the model adaptively discovered and used the difficulty setting menu.

Black Myth: Wukong (different genre, realistic graphics). Lumine "demonstrates basic competence in both navigation and combat" but "several factors prevent it from completing longer tasks": invisible walls in realistic environments cause getting stuck, auto-hiding UI elements are misinterpreted as pre-rendered cutscenes (causing the agent to output noop actions and wait), health bar recognition fails, and "the game's high-difficulty, Souls-like combat, where survival requires enduring multiple waves of enemies" makes zero-shot play particularly difficult. No quantitative metrics are provided.

Ablation Studies and Robustness Checks

Model size comparison during pre-training (Figure 9a): The 2B model's benchmark performance degrades after ~1,200 hours of pre-training data while loss continues to decrease, whereas the 7B model maintains stable improvement on both metrics—demonstrating that 7B is the minimum scale needed to absorb the full 1,731-hour dataset without overfitting.

Pre-training vs. no pre-training for instruction-following (Figure 11): Training only on instruction-following data yields "overall lower performance, particularly on hard tasks," except for NPC Interaction where the no-pre-training model performs slightly better, suggesting pre-training introduces a bias toward ignoring NPCs.

Context window length for history models (Figure 14): Performance peaks at 10 frames and degrades with longer windows, attributed to mismatch with data distribution (most training segments are under 20 frames after action filtering).

Non-history vs. history training (Figure 15, Table 5): History models consistently outperform non-history counterparts, with the most dramatic difference in reasoning tasks (Lumine-Thinking history: 93.4% vs. non-history: 53.4% on Act I). The paper attributes the non-history model's poor performance partly to the 200ms asynchrony problem—without access to previous actions in context, the model incorrectly believes its actions haven't taken effect and overshoots targets.

Classifier size for instruction data labeling (Table 7): Both 2B and 7B classifiers achieve comparable accuracy (~83–84%), so the 2B model is selected for production labeling to maximize throughput. The marginal performance difference "between the 2B and 7B models is marginal" (Section B).

Thinking vs. non-thinking mode latency (Table 4): First action chunk without reasoning: 113.9ms; with reasoning: 234.0ms. The paper argues the 234ms latency (which exceeds the 200ms control cycle) is acceptable because "due to the low frequency of reasoning events, we empirically observe no noticeable impact on the visual experience" (Section 6.2).

Forced reasoning recovery (Section 8.3): The paper implements a safety mechanism where "thinking models will be forced to enter thinking mode when it does not generate a new reasoning for more than 100 steps." No ablation is provided to show performance with vs. without this mechanism, so its contribution to overall success is unknown.

GPT-4.1 as instruction generator and verifier (Section 5.2): No ablation is provided comparing GPT-4.1-generated instructions against purely human-written instructions, so the quality of the automatically generated 200 hours relative to the 165 hours of human-annotated data cannot be assessed from the paper.

Critical Assessment

Claim: "Lumine completes hours-long missions in real time within 3D open-world environments"

What the experiments demonstrate: Lumine-Thinking completes the one-hour Act I mission in 56 minutes (Table 5), the ~4-hour Acts II and III in 4.7 hours (a single run), and makes meaningful progress on the Liyue mission (fully OOD) over multiple hours. These are genuinely impressive demonstrations of real-time autonomous gameplay over long horizons.

What needs qualification: The phrase "on par with human-level efficiency" applies to Act I (56 minutes vs. 78 minutes for fresh players) but not to Acts II and III (4.7 hours vs. 3.6 hours for expert players, a 31% slowdown) or cross-game missions (107 minutes vs. 101 minutes for Wuthering Waves; 7 hours vs. 4.7 hours for Honkai: Star Rail, a 49% slowdown). The efficiency gap widens substantially as the domain diverges from training. Moreover, the Liyue mission includes a ~2-hour period of "aimless exploration" caused by a hallucination that canceled quest tracking—this is a catastrophic failure mode that the paper's single-run evaluation doesn't quantify in terms of reliability. A claim of "human-level efficiency" should be conditioned on in-domain tasks.

What is missing: Statistical reliability. The storyline missions are evaluated with at most 3 runs (Act I subtasks) or single runs (Acts II/III, Liyue, cross-game). With 3 runs per subtask on Act I, the 93.4% overall success rate has a 95% confidence interval spanning roughly 68–100% (exact binomial CI depending on how subtasks are aggregated). For the longer missions, n=1 means no estimate of variance is possible—we cannot distinguish between "Lumine reliably completes this mission slightly slower than humans" and "Lumine got lucky on this particular run."

Claim: "Lumine achieves over 80% success across diverse short-horizon benchmark tasks"

What the experiments demonstrate: Figure 10 shows Lumine-Instruct-NonHis bars at approximately 80–85% across all four categories on simple tasks. This is substantially above all baseline models (which are near zero on Combat and Puzzle). The 61% improvement over Lumine-Base-NonHis is clearly demonstrated.

What needs qualification: The 80%+ figure applies only to simple tasks. Hard tasks show substantially lower performance (Figure 11, exact values not reported numerically but visibly lower in the bar chart), and unseen tasks show moderate decline (Figure 12). The paper's abstract says Lumine "successfully completes... a broad spectrum of tasks," which is true, but the strong quantitative claim (80%+) is specific to simple, in-domain, short-horizon tasks.

What is missing: The baseline comparison is fundamentally unfair in ways that make the victory over GPT-5, Gemini, etc. less informative than it appears. The baseline models use a completely different interaction paradigm (code-format skill primitives vs. Lumine's direct keyboard/mouse generation), operate at vastly slower inference speeds (seconds to 30 seconds per action vs. Lumine's 200ms), and were not fine-tuned on any game-specific data. A more informative baseline would be: (1) fine-tune one of the baseline VLMs on the same 200 hours of instruction-following data using Lumine's action format, or (2) give Lumine the same code-format skill API and compare planning quality independent of execution quality. The current comparison conflates "Lumine's training recipe is effective" with "VLM-based agents using code-format skills are slow and imprecise," which are separate claims.

Claim: "Strong zero-shot cross-game generalization"

What the experiments demonstrate: This is the paper's most striking result and the evidence is qualitatively compelling—Lumine completes substantial missions in Wuthering Waves (107 minutes) and Honkai: Star Rail (7 hours, full chapter) without any fine-tuning or prompt modification. The fact that it works at all in Honkai: Star Rail, a turn-based game with fundamentally different combat, is genuinely surprising and supports the claim that transferable meta-skills have been learned.

What needs qualification: Performance degrades substantially with domain shift. Wuthering Waves (structurally similar): near-parity (107 vs. 101 minutes). Honkai: Star Rail (different combat system): 49% slowdown (7 vs. 4.7 hours). Black Myth: Wukong (realistic graphics, Souls-like): "basic competence" but cannot complete longer tasks. The "strong zero-shot generalization" claim is accurate for structurally similar games but weakens considerably as the genre gap widens. This is a gradual degradation curve, not a binary "generalizes/doesn't generalize."

What is missing: The paper does not isolate which components of Lumine enable cross-game transfer. Is it the behavioral pre-training on diverse gameplay? The instruction-following stage? The reasoning stage? The semantically interpretable action space? An ablation that tests Lumine-Base (no language grounding, no reasoning) on cross-game tasks would reveal whether the transfer is primarily a visuomotor phenomenon (learned during pre-training) or requires the higher-level capabilities from later stages. Without this, we don't know what part of the recipe is responsible for the most novel result.

Claim: "Behavioral pre-training teaches visuomotor primitives that emerge in a predictable order"

What the experiments demonstrate: Figure 9b qualitatively shows different capability curves reaching different score levels at different data volumes. Object interaction emerges earliest, navigation requires the most data. This is a reasonable qualitative finding.

What is missing: This analysis is not quantitatively rigorous. The 5-point scale is evaluated by "human analysis of the models' gameplay videos" with no reported inter-rater reliability, no blinding, and no operational definitions for each score level beyond brief descriptions (e.g., "Score 4: The agent demonstrates generally appropriate behavior with only occasional mistakes, achieving a high success rate"). The curves in Figure 9b lack gridlines or numerical labels, making it impossible to determine exact scores at exact data volumes. For a paper that emphasizes "open recipe" and reproducibility, this qualitative presentation of a central finding is a significant weakness. A proper capability emergence analysis would require: (1) multiple raters with reliability metrics, (2) precise operational definitions per capability, (3) numerical reporting of scores at multiple data checkpoints, and (4) ideally, automated evaluation where possible (e.g., "did the agent reach the quest marker within X seconds?").

More Broadly: Missing Experiments and Analyses

No combination of Lumine components with baseline VLMs. The paper demonstrates that Lumine outperforms GPT-5/Gemini/etc. on game-playing tasks, but never tests whether Lumine's training recipe (behavioral pre-training + instruction-following + reasoning on game-specific data) would improve those larger models even more. If Qwen2-VL-7B can be turned into a competent game agent with ~20,000 GPU-hours of training, what would happen with a 235B-parameter model? The scaling trend in Figure 9a (7B improves stably while 2B saturates) suggests larger models might benefit even more from this recipe, but this is untested.

No ablation of the 20% web data mixture during pre-training. The paper mentions mixing in "approximately 20% mixture of multimodal web data to retain general knowledge" during pre-training, but provides no comparison to pre-training on pure gameplay data. Given that cross-game generalization is a central claim, understanding whether the web data mixture contributes to (or potentially interferes with) transfer would be valuable.

No comparison to prior game-playing VLA models. CombatVLA and JAVIS-VLA are cited as related work and compared in Table 1, but never evaluated as baselines on the benchmark. Since these are the closest prior approaches (data-driven VLA models for game playing), comparing against them—even on a subset of tasks—would contextualize Lumine's contributions relative to the state of the art more precisely than comparisons against prompt-based VLMs.

No error analysis with statistical power. The error analysis in Figure 16 categorizes errors across models but reports percentages based on small absolute counts (e.g., "Poor Timing: 2 cases, 2.5%" for non-history). With such small samples, the percentage breakdown is more misleading than informative. A more useful analysis would report absolute counts and acknowledge the uncertainty, or pool errors across multiple evaluation runs to obtain larger samples.

Single game for training. All training data comes from Genshin Impact's Mondstadt region. The cross-game transfer results are impressive given this limitation, but the paper does not explore what happens when training data spans multiple games—which is the natural next step toward "generalist agents." The paper acknowledges this as future work (Section 9: "scale both the pre-training and reasoning datasets... across other games and domains") but the current results leave open the question of whether training on multiple games would improve cross-game transfer or cause interference.

No latency-matched baseline comparison. Lumine's 25.3× inference optimization is an engineering achievement, but the paper never evaluates what happens when you give the baseline models similar latency advantages—for instance, by running them on comparable hardware with similar optimization, or by simulating what their performance would be if they could operate at 5 Hz instead of ~0.03 Hz. The current comparison makes it impossible to determine whether Lumine's superior performance comes from better decision-making or simply from being able to act 100× more frequently.

6. Limitations and Trade-offs

Limitation 1: Single-Trajectory Gaslighting Has No Recovery Mechanism

The assumption or constraint. The hybrid thinking mechanism treats each reasoning step as the ground truth for subsequent behavior: when the model generates new reasoning, the entire context is flushed, the new reasoning becomes the "current objective" preserved as long-term memory, and all subsequent actions are conditioned on it. The paper explicitly describes this: "When new reasoning is generated, the context is flushed and re-accumulated from that point onward" (Section 6.1). This design assumes that reasoning, once generated, is correct enough to guide behavior until the next reasoning event — which occurs on average every 3.2 seconds, but can extend up to 100 steps (20 seconds) before the forced recovery mechanism triggers.

The consequence. When reasoning is wrong, the agent confidently executes a coherent but incorrect plan for an extended period with no internal mechanism to detect the error. The paper documents exactly this failure mode during the fully OOD Liyue mission:

"Lumine accidentally cancels quest tracking due to a hallucination. When attempting to reactivate it, the model outputs a left-click command combined with mouse movement, which the game misinterprets as a drag action, causing the operation to fail. However, the agent is overconfident that the task had been activated and subsequently closed the quest menu, leading to aimless exploration of the map for approximately two hours."

This two-hour failure is not a motor error or a perception error — it is a reasoning error that self-perpetuates because the reasoning-is-correct assumption prevents the agent from questioning its own conclusions. The error categories in Figure 18 confirm that "Premature Completion" — "a type of hallucination in which the model incorrectly assumes that a proposed goal has already been achieved" and "continues to generate reasoning that builds upon the nonexistent completion" — accounts for 16–17% of all reasoning errors. Each such error can cascade into minutes or hours of wasted behavior before environmental feedback (e.g., mission not progressing, getting stuck) eventually triggers new reasoning.

What evidence exists in the paper. The Liyue mission provides the clearest single-case evidence (Section 8.3.1), but the mechanism is visible throughout: the forced reasoning recovery at 100 steps (Section 8.3) exists precisely because the system has no other way to detect that its current plan is failing. The reasoning error analysis (Figure 18) shows that 8.8–14.0% of reasoning instances contain errors across the entire Act I mission, yet the paper does not measure what fraction of these errors lead to behavioral failures versus being corrected by subsequent reasoning. More critically, the paper provides no mechanism — architectural, algorithmic, or data-driven — for the agent to detect when its own reasoning is wrong before external feedback forces a correction.

Mitigation status. The forced reasoning recovery mechanism (triggering new reasoning after 100 steps without a reasoning event) is a coarse timeout-based safeguard, not a solution. It does not detect errors; it blindly forces a reasoning re-evaluation after a fixed interval regardless of whether the agent is succeeding or failing, which means it may interrupt successful behavior as well. The paper acknowledges this is a band-aid, not a fix, and the two-hour Liyue failure demonstrates that even 100-step recovery is insufficient when the agent is confident its hallucinated plan is correct and continues generating intermediate reasoning steps that reset the counter. The paper does not propose a verification mechanism, an uncertainty estimation approach, or a training strategy to reduce hallucinatory reasoning. This is a fundamental architectural limitation of the "context as memory" design that treats reasoning as authoritative rather than fallible.

Limitation 2: Difficulty Estimation and Strategy Selection Cost Is Entirely Unaccounted For

The assumption or constraint. The paper's three-stage training pipeline — 1,731 hours of pre-training, 200 hours of instruction-following, 15 hours of reasoning — requires carefully curated data at each stage. But the deployment of the trained model is treated as cost-free beyond inference latency: the agent is deployed as a single model that processes observations and generates actions, with no per-task adaptation, no difficulty estimation, and no strategy selection overhead. The paper acknowledges this implicitly by not discussing any deployment-time data collection or adaptation cost.

However, a critical hidden assumption is that the training data distribution covers all scenarios the agent will encounter sufficiently well that no deployment-time adaptation is needed. The paper's own results contradict this: on Acts II and III (reasoning OOD), the agent completes missions 31% slower than expert humans; on fully OOD Liyue missions, it suffers a two-hour hallucination-driven failure; on Black Myth: Wukong, it cannot complete longer tasks at all. Each of these degradations would require additional data collection, annotation, and training to address — but the paper's framework provides no mechanism for doing so efficiently. A practitioner who wanted to deploy Lumine in a new game would need to repeat the entire data collection and training pipeline (2,424 hours of gameplay, 70 annotators, 64 H100 GPUs for days to weeks), not merely fine-tune on a small amount of target-domain data.

The consequence. The headline results — completing Act I at human-level efficiency, 80%+ on benchmark tasks — are achieved after massive investment in data collection and training that is specific to Genshin Impact's Mondstadt region. The paper cannot tell us how much of this investment is necessary versus how much is sufficient: would 500 hours of pre-training data produce nearly the same results? Would 50 hours of instruction-following data suffice? Would reasoning data from a different mission transfer equally well? The scaling analysis (Figure 9a) shows that the 2B model saturates at ~1,200 hours while the 7B model continues improving through the full 1,731 hours, suggesting that the required data volume is model-dependent and may be substantial. For a new domain, a practitioner has no guidance on how much data to collect or which training stages are most critical to invest in.

This is the practical analog of the difficulty estimation problem discussed in the reference example: just as the reference paper's compute-optimal policy assumes difficulty can be estimated cheaply (which it cannot — 2,048 samples per question dominate the inference budget), Lumine assumes that sufficient training data for a new domain can be collected cheaply (which it cannot — 2,424 hours of gameplay from 70 annotators over three months is a substantial operational undertaking). The paper's "open recipe" framing papers over this cost by presenting the pipeline as if it were straightforward to replicate, when in fact the data collection infrastructure (custom recording software, quality control workflows, classifier training, GPT-4.1 verification) represents an engineering investment comparable to the model training itself.

What evidence exists in the paper. The cross-game transfer results (Section 8.3.2) are the closest the paper comes to measuring deployment cost: on Wuthering Waves (structurally similar), near-zero additional cost yields near-parity performance; on Honkai: Star Rail (different combat system), near-zero additional cost yields 49% slowdown; on Black Myth: Wukong (different graphics, combat, and navigation), near-zero additional cost yields failure on longer tasks. This gradient demonstrates that zero-cost deployment works only when the target domain is structurally similar to the training domain, and even then, performance degrades. The paper provides no experiment testing how much target-domain data would be needed to close these gaps — for example, fine-tuning Lumine on 10, 50, or 100 hours of Honkai: Star Rail data and measuring the improvement.

Mitigation status. The paper acknowledges this indirectly in its future work section: "There is significant potential to scale both the pre-training and reasoning datasets, not only within Genshin Impact but also across other games and domains, to enhance the model's generalization and robustness" (Section 9). But this frames the solution as more data rather than more efficient use of data — it does not address the core question of how much data is enough for a new domain, or whether techniques like few-shot adaptation, continual learning, or modular skill composition could reduce the data requirement. The paper offers no experimental evidence on data efficiency, minimum viable dataset sizes, or transfer learning dynamics across domains.

Limitation 3: The "Context as Memory" Design Has a Hard Capacity Ceiling That Is Already Bounding Performance

The assumption or constraint. Lumine implements memory entirely through the model's context window: a sliding window of up to 20 recent image–action pairs as short-term memory, plus the most recent reasoning step as long-term memory (Section 6.1). This design treats the transformer's attention mechanism as the sole memory architecture, assuming that all information needed for coherent behavior over hours-long missions can be captured within a 20-frame (~4-second) window of visual history plus a single reasoning sentence.

The consequence. The paper identifies five specific failure modes that are direct consequences of this memory constraint. On Acts II and III (Section 8.3.1):

Lack of proactive fast travel: Lumine rarely uses teleportation, instead following quest markers on foot across long distances. The paper attributes this to the absence of teleportation patterns in the first hour of reasoning data, but the deeper issue is that the agent cannot remember that it has previously visited a Statue of the Seven that could serve as a teleport waypoint — that information is not in its 4-second visual window. Human players use teleportation precisely because they remember the map layout over hours of gameplay.
Limited memorization: When multiple quest markers appear simultaneously, Lumine "is easily distracted, oscillating between different targets" (Figure 21a). When a direct path is blocked and a detour is needed, Lumine "often abandons the detour midway and returns to the starting point, drawn back by the quest marker's signal since it has forgotten that the direct path is blocked" (Figure 21b). In both cases, the 4-second memory window is insufficient to maintain the "detour in progress" or "already visited this marker" state.
Lack of proactive health recovery: The agent "does not actively restore the health of party members" because recovery patterns are absent from the reasoning data. But more fundamentally, the agent cannot maintain a running estimate of its health status over time — it only knows its current health from the current frame, and cannot compare it to health from 30 seconds ago to detect a downward trend.

On Honkai: Star Rail (Section 8.3.2), the navigation bottleneck is explicitly a memory problem: Lumine "attempts to jump across gaps toward a quest marker on the opposite platform, only to be blocked by an invisible wall... it more often returns to the same spot, repeatedly drawn by the quest marker and constrained by limited memorization." The agent cannot remember "I tried this path and it didn't work" because that information is evicted from the context window within 4 seconds.

The error analysis in Figure 16 confirms that the history model (20-frame context) makes fewer multimodal understanding errors than the non-history model (30.0% vs. 43.8% object detection failures), but the residual error rate is still substantial. This suggests that even 20 frames are insufficient for robust visual tracking — extending the window further might help, but Figure 14 shows that performance degrades beyond 10–15 frames, likely because the training data contains few segments longer than 20 frames after action filtering.

What evidence exists in the paper. The context window length sweep (Figure 14) is the most direct evidence: performance peaks at 10 frames and declines with longer windows, indicating that naively increasing the memory capacity is not a solution — the model cannot effectively use longer contexts because it wasn't trained on them. The five inefficiency factors on Acts II and III (Section 8.3.1) are qualitative but consistent with a memory bottleneck. The cross-game navigation failures (repeatedly attempting impossible jumps in Honkai: Star Rail, getting stuck at invisible walls in Black Myth: Wukong) demonstrate memory limitations in out-of-distribution settings where the agent needs to learn environment-specific constraints through trial and error but cannot retain those lessons.

Mitigation status. The paper explicitly acknowledges this limitation and identifies it as a primary direction for future work: "Future work should explore more sophisticated approaches that enable efficient memory retrieval and management over thousands turns of interactions" (Section 9). It notes that the current mechanism "can be easily extended to maintain multiple reasoning segments if needed" (Section 6.1), but provides no experiments or design proposals for such extensions. The mitigation is entirely deferred. No comparison is made to alternative memory architectures (e.g., retrieval-augmented generation, external knowledge bases, hierarchical summarization, or the periodic trajectory summarization used by prompt-based agents like Cradle and Voyager), leaving practitioners with no guidance on which memory approach to invest in.

Limitation 4: The System Cannot Exceed Human Demonstrator Performance

The assumption or constraint. Lumine is trained entirely through behavioral cloning — supervised learning on human demonstrations at every stage. The pre-training stage learns from raw human gameplay (1,731 hours), the instruction-following stage uses human-annotated activity labels and GPT-4.1-generated instructions conditioned on human actions, and the reasoning stage uses human-written inner monologues paired with human actions. At no point does the agent receive a reward signal, engage in trial-and-error learning, or explore beyond the distribution of human behavior. The paper is explicit about this constraint in its discussion of future work: "Lumine primarily learns from offline data, which is efficient but limits its ability to surpass human-level performance" (Section 9).

The consequence. Lumine inherits the biases, inefficiencies, and skill ceilings of its human demonstrators. The paper documents several specific manifestations:

The pre-training data introduces a "notable bias" where "many trajectories involve players merely passing by NPCs without engaging in interaction" (Section 8.2.1), which the instruction-following stage must explicitly correct — but can only correct to the extent that instruction data provides counterexamples.
On Acts II and III, Lumine is 31% slower than expert human players, and the identified inefficiencies — failure to use teleportation, limited minimap usage, no proactive health recovery — are behaviors that were simply not demonstrated frequently enough in the training data for the model to learn reliably. A reinforcement learning agent could discover these strategies through exploration and reward, but Lumine cannot.
On Honkai: Star Rail combat, Lumine "fails to understand that pressing E requires selecting a target, or that using an ultimate requires confirming with the spacebar" (Section 8.3.2). These are mechanics that a human would learn through trial and error (or by reading tutorials), but Lumine's behavioral cloning approach means it can only reproduce action patterns it has seen in Genshin Impact — where E does not require target selection — and has no mechanism to discover the correct behavior through interaction.
The forced reasoning recovery at 100 steps (Section 8.3) is a hand-crafted heuristic, not a learned behavior. The model never learns to recognize when it is stuck — it relies on an external timeout. A reinforcement learning agent could learn to detect and recover from stuck states autonomously.

More fundamentally, Lumine can never discover strategies that are better than what humans do. If there exists a combat rotation that deals 20% more damage than any human demonstrator used, or a navigation route that is 30% faster, or a puzzle solution that humans consistently miss, Lumine will never find them — not because the underlying visuomotor capabilities are insufficient, but because the training signal (imitation) provides no gradient toward improvement beyond the demonstration distribution.

What evidence exists in the paper. The paper provides no experiment comparing Lumine to a reinforcement learning baseline or an agent trained with online exploration. The comparison is exclusively against prompt-based VLMs (which are not trained on game data at all) and against human players (where Lumine underperforms experts on most metrics). The cross-game transfer results indirectly demonstrate the limitation: Lumine's Honkai: Star Rail combat performance is poor precisely because it relies on Genshin Impact behavioral priors that don't transfer, and it has no mechanism to adapt those priors through interaction. A model capable of online learning could improve through trial and error; Lumine cannot.

Mitigation status. The paper acknowledges this limitation and frames it as future work: "Building upon the strong foundation of the existing Lumine model, integrating online reinforcement learning could enable autonomous exploration and continuous self-improvement, further enhancing its performance beyond static, offline learning" (Section 9). This is a reasonable direction, but it is entirely speculative — the paper provides no experiments on whether the Lumine architecture and training pipeline are compatible with RL fine-tuning, what reward signals would be used, how exploration would be managed, or whether the behavioral cloning initialization would help or hinder subsequent RL training. The experience of other domains (e.g., AlphaGo's supervised learning → RL pipeline, or RLHF for language models) suggests that combining imitation and reinforcement learning is powerful but non-trivial, and the paper offers no evidence that Lumine's particular architecture is amenable to this combination.

Limitation 5: Single-Game Training With Uncharacterized Transfer Boundaries

The assumption or constraint. All training data — pre-training, instruction-following, and reasoning — comes from a single game (Genshin Impact) and, within that game, primarily from a single region (Mondstadt). The paper positions this as a strength — "Lumine's remarkable zero-shot generalization to unseen missions and even entirely unseen games suggests that the model acquires transferable meta-skills" (Section 9) — but it is also a fundamental constraint on what can be claimed about generality. The paper assumes, implicitly, that the structural similarities between Genshin Impact and other 3D games (control schemes, UI conventions, quest structures) are sufficient to enable transfer, and that the transfer gradient observed across three target games is representative of what would be observed across the broader space of interactive 3D environments.

The consequence. The paper's transfer results (Section 8.3.2) establish three data points on a transfer curve: Wuthering Waves (structurally similar, near-parity), Honkai: Star Rail (different combat system, ~49% slowdown), Black Myth: Wukong (realistic graphics + Souls-like combat, fails on longer tasks). These three points are suggestive but insufficient to characterize the transfer landscape. A practitioner considering Lumine for a new game — say, a first-person shooter, a driving game, a real-time strategy game, or a VR environment — has no way to predict whether transfer would yield near-parity, substantial degradation, or complete failure, because the paper provides no analysis of which specific features of the source and target environments determine transfer success.

The key unmeasured variables include:

Control scheme similarity: All three target games share WASD movement and mouse-looking with Genshin Impact. Would transfer work for a game with different movement conventions (e.g., click-to-move, controller input, or gesture-based controls)?
UI convention similarity: Wuthering Waves and Honkai: Star Rail share dialog systems, quest markers, and menu layouts with Genshin Impact. Would transfer work for a game with radically different UI (e.g., a minimal HUD, diegetic interfaces, or text-heavy CRPG menus)?
Visual style similarity: All target games use stylized 3D graphics. Black Myth: Wukong's more realistic rendering already causes failures (misinterpreting auto-hiding UI as cutscenes). Would transfer work for 2D games, pixel-art games, or games with abstract visual representations?
Task structure similarity: All target games are progression-based RPGs with quest structures and explicit objectives. Would transfer work for open-ended sandbox games, competitive multiplayer games, or puzzle games with no quest markers?

The paper's claim that Lumine has acquired "transferable meta-skills, such as 3D navigation and 2D manipulation" (Section 9) is a hypothesis consistent with the data but not rigorously tested against alternative explanations — for instance, that the transfer is primarily a function of Qwen2-VL's general visual capabilities (acquired during web-scale pre-training) rather than Lumine's game-specific training, or that the transfer works only for games that are close derivatives of Genshin Impact in game design space.

What evidence exists in the paper. The three-game transfer evaluation (Section 8.3.2) and the within-game OOD generalization (Acts II/III, Liyue — Section 8.3.1) are the only evidence. The paper provides no experiment that isolates the contribution of the pre-training, instruction-following, and reasoning stages to transfer — for instance, testing whether Lumine-Base (no language grounding, no reasoning) exhibits similar cross-game competence, which would suggest that the transfer is primarily a visuomotor phenomenon acquired during behavioral pre-training. The paper also provides no experiment testing transfer to a game that is structurally dissimilar in a controlled way (e.g., same combat system but different control scheme, or same control scheme but 2D instead of 3D) to characterize which features matter.

Mitigation status. The paper acknowledges the limitation of single-game training and identifies cross-game scaling as future work: "There is significant potential to scale both the pre-training and reasoning datasets, not only within Genshin Impact but also across other games and domains, to enhance the model's generalization and robustness" (Section 9). But this framing — "scale to more games" — does not address the deeper question of which games provide the most transferable skills or how to characterize the transfer landscape. A more systematic approach would involve training variants of Lumine on different source games (or combinations of games) and measuring transfer across a diverse benchmark, but the paper provides no such analysis, leaving practitioners to guess whether Lumine's recipe would work for their specific target environment.

Limitation 6: The Baseline Comparison Fundamentally Confounds Model Capability With Interaction Paradigm

The assumption or constraint. The paper compares Lumine (7B parameters, fine-tuned on 1,946 hours of game-specific data, generating direct keyboard/mouse actions at 200ms per decision) against five prompt-based VLMs (GPT-5, Gemini 2.5 Pro, Grok4, Doubao1.6-Vision, Qwen3-VL-235B — all substantially larger than 7B, none fine-tuned on game data) that interact through a completely different mechanism: predefined skill functions in code format invoked via function calls, with inference times of "as long as 30 seconds" per action requiring the game to be paused (Section 7). These two groups differ on at least four confounded dimensions: model size (7B vs. hundreds of billions), training data (game-specific vs. none), interaction paradigm (direct keyboard/mouse generation vs. code-format skill functions), and inference frequency (5 Hz vs. ~0.03 Hz).

The consequence. The near-zero baseline performance on Combat and Puzzle tasks (Figure 10) is uninterpretable: we cannot determine whether it reflects inferior reasoning by the baseline models (plausible, since they weren't trained on game data), inferior execution due to the coarse code-format skills (also plausible — turn(degree) cannot express fine-grained camera control, attack() cannot express timing-sensitive combos), or simply the fact that the baseline models must pause the game for seconds between actions in a real-time environment where combat demands sub-second reactions. This conflation undermines the paper's primary empirical claim that Lumine's training recipe produces superior game-playing capability.

A clearer experimental design would hold the interaction paradigm constant while varying the model and training. For instance: (a) fine-tune a baseline VLM (or Lumine's base model Qwen2-VL-7B) on the same instruction-following data but using the code-format skill API — this would test whether Lumine's direct action generation provides benefits beyond what fine-tuning alone provides; (b) deploy Lumine with the same code-format skill API and compare its planning quality to the baselines — this would isolate the contribution of the training recipe to high-level reasoning independent of low-level execution; (c) give the baseline VLMs access to Lumine's action space and latency — this would test whether the performance gap is primarily about training data or about inference speed and representation. None of these experiments are performed.

The paper's Table 1 comparison, which positions Lumine against prior game agents across multiple dimensions, is similarly confounded. Lumine is credited with a "5 hrs" task horizon, "Adaptive" reasoning, and "Real-Time" inference, while prior agents are marked as lacking one or more of these. But this comparison treats each dimension as binary rather than continuous, and it credits Lumine for capabilities that are enabled by its specific interaction paradigm (direct actions with action chunking) rather than by any generalizable advance in agent architecture. A prompt-based agent could, in principle, be given the same action space and latency optimizations — the paper doesn't test this, so we cannot determine how much of Lumine's advantage comes from its training recipe versus its engineering optimizations.

What evidence exists in the paper. The benchmark results (Figure 10) are the primary comparison, and the confounds are visible in the data: baseline models achieve non-zero but low performance on Collection and NPC Interaction tasks (which require less precise timing and can tolerate multi-second pauses), and near-zero on Combat and Puzzle tasks (which are time-sensitive). This pattern is consistent with the hypothesis that inference latency — not reasoning quality — is the dominant bottleneck for the baselines, but the paper does not test this hypothesis. The paper acknowledges the latency gap ("API responses can take as long as 30 seconds to return, we pause the game during inference," Section 7) but treats it as a fixed constraint of the baselines rather than a variable to be controlled.

The paper also never evaluates whether the baseline VLMs, if given Lumine's action space and sufficient inference speed, would perform better than they do with the code-format skills. The Cradle framework used for baselines defaults to skill primitives; the paper could have implemented a direct keyboard/mouse action generation mode for the baselines (using the same tokenization and action format as Lumine) and evaluated whether their performance improves, but this experiment is absent.

Mitigation status. The paper does not acknowledge this as a limitation. The baseline comparison is presented as a straightforward evaluation of Lumine against state-of-the-art VLMs, without discussion of the confounded dimensions or what conclusions can and cannot be drawn. The "Related Work" section (Section 2) and Table 1 position Lumine as advancing beyond prior agents across multiple capability dimensions, but without disentangling the contributions of training data, model architecture, action representation, and inference optimization. A practitioner reading this paper cannot determine whether Lumine's recipe would improve their own VLM-based agent (which might use a different interaction paradigm) or whether the gains are specific to the direct-action-generation approach and the latency optimizations that make it viable.

7. Implications and Future Directions

How This Work Changes the Landscape

Lumine represents a methodological reframing rather than a paradigm shift — it doesn't introduce fundamentally new architectures or learning algorithms, but it reorganizes how we think about building generalist agents for rich interactive environments by inverting the standard VLA training curriculum and demonstrating that behavioral pre-training on raw, unlabeled data should be the primary stage, not an afterthought.

The conceptual shift: from language-as-scaffold to behavior-as-foundation. The dominant assumption in the VLA literature has been that pre-trained VLMs bring sufficient visual and linguistic understanding to a new domain, and the primary challenge is teaching them to act — hence the emphasis on instruction-following data as the main training signal. Lumine's results argue the opposite: the base VLM's visual understanding is actually inadequate for domain-specific perception (the error analysis in Figure 16 shows multimodal understanding accounts for 50–58% of all instruction-following errors, with object detection failures being the dominant subcategory), and motor competence must be built from the ground up through large-scale imitation of raw behavior. Language then directs an already-functioning visuomotor system rather than constructing one. This isn't an architectural claim — Lumine uses the same Qwen2-VL backbone as any other VLA model — but a claim about data curriculum design, and the evidence that it matters comes from the ablation in Figure 11: models trained only on instruction-following data without behavioral pre-training show substantially degraded performance, especially on hard tasks.

What makes this reframing significant rather than obvious is that it contradicts the intuition that "bigger models trained on more internet data should need less domain-specific behavioral data." The baseline VLMs tested — GPT-5, Gemini 2.5 Pro, Qwen3-VL-235B — are substantially larger than Lumine's 7B parameters and were trained on vastly more data, yet they achieve near-zero success on combat and puzzle tasks (Figure 10). This suggests that scale and general pre-training do not substitute for domain-specific behavioral experience in visuomotor domains, at least given current architectures. The capability emergence analysis (Figure 9b) reinforces this: even for the 7B model, navigation — the most essential skill for open-world gameplay — requires over 1,800 hours of behavioral data to become reliable. Extrapolating, a model that skipped behavioral pre-training entirely would need an implausibly large instruction-following dataset to acquire equivalent motor competence, if it could acquire it at all.

Reconciling contradictions in the agent-building literature. The paper's three-stage curriculum provides a unified explanation for previously conflicting findings about VLM-based game agents. Prompt-based systems like Cradle and Voyager succeed at long-horizon planning but fail at real-time execution because they lack the behavioral pre-training stage — their models have never learned the visuomotor primitives needed for precise, low-latency control. Data-driven VLA models like CombatVLA and JAVIS-VLA succeed at short-horizon instruction following but fail at long-horizon autonomy because they lack the reasoning stage — their models can execute specific commands but cannot self-generate goals or maintain plans over time. Lumine's finding that the instruct model achieves only 6.6% success on the full Act I mission while the thinking model achieves 93.4% (Table 5) — despite identical pre-training and instruction-following data — demonstrates that both behavioral foundations and reasoning capabilities are necessary, and neither is sufficient alone. The conflicting results in prior work weren't contradictory findings about what VLMs can do; they were evaluations of incomplete systems that each addressed different necessary-but-insufficient components of the full agent stack.

Which research directions become more attractive. This work makes behavioral pre-training on raw interaction data the central research priority. If Lumine's core finding is correct — that visuomotor competence must be learned from large-scale behavioral data before language grounding or reasoning can be effectively layered on — then the key bottleneck for building agents in new domains is not better prompt engineering or more sophisticated reasoning strategies, but rather efficient collection and utilization of raw behavioral data. Research on self-supervised behavioral representations, data augmentation for visuomotor learning, and methods for transferring behavioral priors across domains becomes substantially higher-priority than it was before this paper.

Which directions become less attractive. Conversely, this work casts doubt on the sufficiency of prompt-only or few-shot adaptation approaches for embodied agent domains. The near-zero baseline performance (Figure 10) suggests that even the largest current VLMs, when dropped into a new interactive environment without domain-specific training, lack the fundamental perceptual and motor competence to perform basic tasks. If GPT-5 cannot pick a flower or talk to a visible NPC in a game environment — tasks that require basic object recognition and a single button press — then the path to generalist agents cannot rely on general-purpose models with task-specific prompts alone. Domain-specific behavioral data appears to be non-negotiable.

A new diagnostic: capability emergence curves. The paper's atomic ability evaluation (Figure 9b) introduces a methodology for characterizing what a model learns during behavioral pre-training as a function of data volume. This is a transferable diagnostic: for any new domain, one could define a set of atomic capabilities (e.g., for a driving simulator: lane-keeping, turning, stopping at signs, overtaking), train models on increasing amounts of raw behavioral data, and measure when each capability emerges. This would provide principled guidance for data collection — if navigation emerges at ~1,800 hours in your target domain, you know approximately how much data you need to collect before expecting reliable autonomous behavior. The paper doesn't formalize this as a methodology, but it is the natural operationalization of its findings for practitioners.

Follow-Up Research This Work Enables

Characterizing the transfer landscape: which source environment features predict zero-shot performance in a target environment? The paper establishes three data points on a transfer curve (Wuthering Waves: near-parity; Honkai: Star Rail: 49% slowdown; Black Myth: Wukong: failure on longer tasks) but provides no systematic analysis of why transfer degrades. A strong follow-up would define a set of measurable dimensions along which game environments differ — control scheme similarity (WASD+mouse vs. controller vs. click-to-move), visual style distance (stylized 3D vs. realistic 3D vs. 2D/pixel art), UI convention overlap (shared HUD elements, dialog systems, menu layouts), task structure similarity (quest-based progression vs. open-ended sandbox vs. competitive multiplayer), and physics model similarity (movement affordances: jumping, climbing, gliding) — and then train Lumine variants on one source game while systematically evaluating zero-shot transfer to target games that differ along exactly one dimension at a time. For example, a target game with identical controls and UI but different visual style would isolate the contribution of visual domain shift; a target game with identical visuals and UI but different control scheme would isolate the contribution of motor transfer. This would transform the paper's suggestive qualitative finding ("Lumine generalizes!") into a predictive model of when and why transfer succeeds, directly addressing Limitation 5.

Does behavioral pre-training on multiple games produce better transfer than pre-training on any single game? The paper demonstrates zero-shot transfer from a single source game, but leaves open the natural next question: if Lumine were pre-trained on gameplay data from Genshin Impact and Wuthering Waves and Honkai: Star Rail (or some subset), would its zero-shot performance on a held-out fourth game improve compared to pre-training on any single game? This is the central question for building genuinely generalist agents: does diverse behavioral experience compose additively (each new game adds transferable skills) or does it cause interference (different control schemes, visual styles, and mechanics conflict)? The experiment would require collecting comparable volumes of behavioral data from multiple games, training separate models on each individual game and on combinations, and measuring transfer to held-out games. The Lumine recipe makes this newly tractable because it provides a standardized training pipeline — the architecture, action space, and training hyperparameters are fixed; only the data source changes. A negative result (multi-game pre-training does not improve over the best single-game source, or actively hurts) would be equally valuable, as it would suggest that the transferable meta-skills posited by the paper are actually game-specific motor patterns that happen to overlap across structurally similar games, rather than truly abstract competencies.

What is the minimum viable behavioral pre-training dataset for a new domain? The paper's scaling analysis (Figure 9a) shows the 2B model saturating at ~1,200 hours while the 7B model continues improving through the full 1,731 hours, but this is measured on in-domain benchmark performance — not on downstream instruction-following or reasoning performance, which are the capabilities that matter for deployment. A practically crucial experiment would be: take Lumine-Base models trained on 100, 250, 500, 1000, and 1731 hours of behavioral data, fine-tune each through the full instruction-following and reasoning stages, and measure final task performance on both in-domain and cross-game benchmarks. This would reveal whether the benefits of behavioral pre-training saturate for downstream capabilities before they saturate for raw action prediction loss — it's possible that 500 hours of behavioral data provides 90% of the instruction-following performance that 1,731 hours provides, making the recipe substantially cheaper to replicate than the headline numbers suggest. The paper's current experiment (Figure 9a) only measures benchmark success during pre-training itself, leaving the transfer to downstream stages uncharacterized. This experiment also directly addresses Limitation 2 by providing practitioners with concrete guidance on data requirements.

Can the reasoning stage be replaced or augmented with test-time search? The paper's reasoning model generates a single chain of inner monologue autoregressively, which is efficient but susceptible to the cascading hallucination problem documented in Limitation 1 (e.g., the two-hour Liyue failure). A natural alternative — inspired by the test-time compute scaling literature — would be to have the model generate multiple candidate reasoning chains at each decision point, score them using some verifier (e.g., a separately trained outcome predictor, or consistency with the current visual observation), and select the best one before proceeding. This would trade increased inference cost for reduced reasoning error rates, and might substantially reduce catastrophic failures on OOD missions. The experiment would compare Lumine-Thinking's current single-chain reasoning against best-of-N reasoning (generate N reasoning chains, select via verifier), beam search over reasoning tokens, or a simpler consensus-based approach (generate multiple chains, take the action implied by the majority). The paper's reasoning error analysis (Figure 18) provides a direct metric for evaluating whether such methods reduce the 8.8% reasoning error rate. This is a natural extension that combines Lumine's architecture with the inference-time compute methodology, and it directly addresses the most consequential failure mode.

Integrating online reinforcement learning after behavioral cloning. The paper explicitly identifies the inability to surpass human demonstrator performance as a limitation (Limitation 4, Section 9) and suggests RL fine-tuning as the natural solution, but provides no experiments. A strong follow-up would take the fully trained Lumine-Thinking model and fine-tune it using online RL with a reward signal derived from game progress (e.g., quest completion events, enemy defeat indicators, distance-to-objective reduction). The key question is whether behavioral cloning provides a useful initialization for RL in this domain — it could either accelerate learning by providing reasonable exploration behavior, or hurt learning by creating a narrow policy distribution that RL struggles to escape. The experiment would compare RL-from-scratch, RL-from-Lumine-Base, and RL-from-Lumine-Thinking on a set of combat and navigation tasks where human performance is measurable and suboptimal strategies are known to exist. The paper's detailed characterization of Lumine's inefficiencies (failure to use teleportation, limited minimap usage, etc.) provides concrete targets for measuring whether RL can discover strategies that behavioral cloning missed.

Practical Applications and Downstream Use Cases

Automated game testing and quality assurance. The paper explicitly identifies this application (Section 9): Lumine "can autonomously explore game environments or interpret natural-language instructions to execute gameplay tasks, supporting debug detection, quality assurance, and large-scale usability evaluation in game development." The specific value proposition is grounded in Lumine's demonstrated capabilities: it can complete the full five-hour Mondstadt storyline, follow diverse short-horizon instructions at >80% success, and handle both 3D overworld navigation and 2D GUI manipulation — precisely the range of interactions that human QA testers perform. For a game studio developing a new open-world title, deploying Lumine (fine-tuned on early builds) could automate regression testing of main storyline quests, identify tutorial steps where the agent gets stuck (implying unclear instructions or buggy triggers), and stress-test UI flows across different screen resolutions and aspect ratios. The key advantage over scripted testing bots is Lumine's ability to handle unexpected states — the same robustness that lets it recover from getting stuck in Honkai: Star Rail would let it continue testing even when a quest trigger doesn't fire exactly as expected, providing coverage of edge cases that scripted bots would simply crash on. The paper doesn't provide a direct QA-relevant metric (e.g., bug detection rate or coverage relative to human testers), but the long-horizon completion results on OOD missions suggest basic viability.

AI-driven game assistants and streaming companions. The paper notes that Lumine "opens up possibilities for new forms of interactive entertainment, such as AI-driven game assistants and streaming where intelligent agents are capable of playing, commenting, and engaging with audiences in real time" (Section 9). This is not speculative given the demonstrated capabilities: Lumine already generates natural-language reasoning (first-person inner monologue averaging 37.4 tokens per reasoning event) that explains what it's doing and why, operates in real time (56 minutes for a one-hour human mission), and handles diverse gameplay types. A streaming deployment would take Lumine's existing reasoning output — currently used only internally as memory — and surface it as commentary, potentially with additional training to make the monologue more entertaining or audience-directed. The zero-shot cross-game transfer is especially relevant here: a streamer could deploy the same Lumine model across multiple games without reconfiguration, with the agent providing competent gameplay and commentary in each. The business model would be analogous to existing AI streamers (e.g., the viral Claude Plays Pokémon stream cited in the paper), but with the critical advantage of real-time operation — Lumine doesn't need to pause the game to think, making the viewing experience continuous and engaging rather than stuttering.

Scalable data generation for self-improving agents. Lumine's three-stage pipeline is inherently a data generation pipeline: the trained model can play through new content, and those playthroughs — including actions, observations, and reasoning traces — can become training data for the next iteration. This is the "iterative self-improvement loop" the paper gestures toward (Section 9). Concretely, a game studio with access to Lumine could: (1) train Lumine on existing gameplay data from version N of their game, (2) deploy Lumine to play through new content in version N+1, (3) filter successful playthroughs (using quest completion events as automatic reward signals), and (4) fine-tune Lumine on this new data, producing an agent adapted to the new content without requiring human demonstrators to play it first. The paper's results suggest this is feasible: Lumine already completes the full Act I mission in 56 minutes with 93.4% success, meaning the majority of its playthroughs could serve as training data for subsequent iterations. The key enabler that this paper provides (that prior VLA models lacked) is the reasoning stage — Lumine-Thinking can complete long-horizon missions autonomously, whereas Lumine-Instruct (which lacks reasoning) gets distracted and fails (6.6% on Act I), making it useless for autonomous data generation. This shifts the bottleneck from "can the agent complete the content at all?" to "can we filter the agent's successful completions efficiently?" — a much easier problem.

When to Prefer This Method

The paper does not articulate an explicit tradeoff decision framework comparing Lumine against named alternatives (e.g., "use Lumine when X, use Cradle/Voyager when Y, use RL-from-scratch when Z"). It positions Lumine primarily as an advance over prior approaches rather than as one option in a design space with clear tradeoffs. The comparisons in Table 1 and the baseline evaluation (Figure 10) demonstrate Lumine's superiority on the evaluated dimensions, but these are presented as evidence of progress rather than as a decision rule. The cross-game transfer results suggest a gradient — Lumine works better when the target environment is structurally similar to Genshin Impact — but this is an empirical observation about transfer distance, not a principled tradeoff against alternative agent architectures.

A forced decision matrix here would be fabricating a framework the paper does not provide. The paper's contribution is better understood as a recipe — a specific sequence of training stages, data curation procedures, and inference optimizations — that practitioners can replicate for their own environments, rather than as a method that occupies a particular point in a well-characterized tradeoff space with clearly named alternatives.