Learning Vision-Driven Reactive Soccer Skills for Humanoid Robots

ArXiv: 2511.03996

🎯 Pitch

Humanoid robots playing dynamic, vision-based soccer has long been stymied by the bottleneck of decoupled perception and action. This paper shows that a unified policy, trained with a novel adversarial motion prior and a virtual perception system, enables a full-sized humanoid to execute reactive, coherent soccer skills like chasing and kicking a ball purely from noisy, delayed onboard camera data. The learned controller's effectiveness is demonstrated by its contribution to a dominant RoboCup 2025 championship win, scoring 76 goals against 11 conceded.

---

1. Executive Summary

This paper introduces a unified reinforcement learning-based controller that enables humanoid robots to acquire reactive soccer skills through tight integration of visual perception and motion control, extending Adversarial Motion Priors (AMP) to perceptual settings in real-world dynamic environments. The system processes onboard camera detections — ball and goal positions projected into bird’s-eye-view space — through an encoder-decoder architecture paired with a virtual perception system that models real-world visual noise, latency, and detection dropouts during simulation training, enabling the policy to recover privileged states from imperfect observations and establish active perception-action coordination. Trained entirely in simulation with Isaac Gym on the Booster T1 humanoid platform and deployed zero-shot to hardware, the controller demonstrates strong reactivity across diverse terrains and visual conditions, achieving kicking success rates that closely match simulation performance on the physical robot and contributing to championship wins at RoboCup 2025 with 76 goals scored and only 11 conceded. The work establishes that GAN-based motion learning can be effectively extended beyond proprioceptive imitation to vision-driven dynamic control, with the policy autonomously composing behaviors — ball searching, chasing, multidirectional kicking — that the paper shows outperform rule-based strategies in both speed and agility, though the approach is limited to individual skills without multi-agent coordination.

2. Context and Motivation

The Core Problem: Tight Perception-Action Coupling in Dynamic Environments

The central challenge this paper tackles is deceptively simple to state but extraordinarily difficult to achieve: enabling a humanoid robot to play soccer using only onboard vision, in real time, with the same fluid reactivity that characterizes human sensorimotor coordination. This is not merely a locomotion problem or a perception problem in isolation — it is fundamentally a perception-action coupling problem, where the robot must continuously translate noisy, delayed, and spatially incomplete visual observations into precise, adaptive motor commands at approximately 50 Hz.

To appreciate why this is difficult, consider what happens in even a basic soccer interaction: a humanoid robot spots a rolling ball, adjusts its walking gait mid-stride to intercept it, rotates its torso and head to maintain visual contact, selects which foot to kick with, positions that foot with centimeter-level accuracy relative to the ball, and executes a strike with appropriate timing, angle, and force — all while maintaining dynamic balance on the other foot. Each of these sub-tasks depends on visual information that is inherently delayed (by tens to hundreds of milliseconds of camera exposure, neural network inference, and communication latency), noisy (pixel coordinates must be projected into 3D space with uncertain camera calibration), and incomplete (the ball frequently leaves the camera's field of view during dynamic turns or when the robot looks down at its feet). The controller cannot wait for a clean, complete observation before acting — it must act continuously with whatever information is available, anticipating future states and compensating for perceptual degradation.

The authors frame this concretely in Section 1:

"The robot is required to generate long-horizon behaviors that satisfy complex task demands while continuously adapting its motion in real time to rapidly changing sensory feedback. These challenges are further amplified by onboard visual perception that is noisy, delayed, and spatially constrained during dynamic motion, leading to distorted or incomplete observations of the environment state."

This is fundamentally a Partially Observable Markov Decision Process (POMDP) problem — the robot never directly observes the true state (ball position, velocity, goal location) but must infer it from a stream of corrupted sensory signals and use those inferences to guide motor actions within a control loop operating at a timescale much finer than the perceptual update rate.

Why This Problem Matters: Beyond Soccer

While robot soccer may initially appear narrow in scope, the authors position it as a canonical testbed for embodied intelligence that compresses multiple fundamental robotics challenges into a single, measurable task. In Section 1, they argue:

"Unlike benchmarks that target either locomotion or manipulation skills, robot soccer serves as a focused testbed that compresses a broad spectrum of essential robotic competencies into a single continuous task, offering a practical arena that drives technological advances toward human-level embodied intelligence in the real world."

The broader significance rests on several observations:

1. Soccer demands simultaneous locomotion and manipulation. The robot cannot separate "walk to the ball" and "kick the ball" into independent modules — the approach is the kick. Foot placement during locomotion directly determines which foot contacts the ball, at what angle, and with what force. The transition from walking to kicking must be seamless, with no pause for replanning or sensor recalibration. This tight integration of mobility and physical interaction is a key requirement for any robot operating in unstructured human environments — delivery robots maneuvering through cluttered doorways, construction robots positioning tools, or assistive robots navigating around furniture while reaching for objects.

2. The environment is highly dynamic and adversarial. Unlike static navigation tasks where the target remains fixed, soccer involves a ball that moves, opponents that interfere, and constant visual occlusions. The robot must react to events it cannot fully predict — a ball suddenly changing direction after a deflection, a goalkeeper moving to block a shot, physical contact that perturbs its trajectory. This reactive demand generalizes broadly: any field-deployed robot (agricultural, search-and-rescue, warehouse) encounters unexpected physical interactions and must respond without the luxury of replanning from scratch.

3. Real-world deployment eliminates simulator shortcuts. In simulation, the robot has access to ground-truth state: exact ball position, velocity, friction coefficients, and perfect depth information. On hardware with only an RGB camera, these privileged signals vanish. The robot must infer everything from pixels corrupted by motion blur, lighting changes, background clutter, and perspective distortion. Successfully bridging this gap — what the community calls the sim-to-real transfer problem — is perhaps the single greatest obstacle to deploying learned controllers outside the lab. Soccer's rich perceptual demands (tracking small fast-moving objects against varied backgrounds) make it a particularly stringent test.

4. The RoboCup competition imposes hard real-world constraints. The authors note in Section 2.2 that their controller was evaluated under RoboCup Adult-size Humanoid League rules, where:

"robots were required to operate fully autonomously, relying exclusively on onboard computation and sensing, while the use of sensors exceeding human sensory capabilities was strictly prohibited, such as LiDAR or multiple camera systems. Moreover, adaptation time to the actual competition venue was highly limited."

These constraints — fully onboard computation (a Jetson AGX Orin), single RGB camera, no prior data from the competition environment — force the learning system to generalize robustly. Lab demonstrations often rely on external motion capture, offboard GPU clusters, or carefully controlled lighting; competition deployment provides a harsh test of whether the system genuinely works.

Prior Approaches and Their Limitations

The paper identifies three generations of approaches to robot soccer, each advancing the state of the art while revealing fundamental limitations that motivate the current work.

The Modular, Hand-Engineered Approach

Early learning-based work (and the dominant paradigm in RoboCup until recently) decomposed soccer into independently engineered subsystems: a perception module that detected the ball and goal from camera images, a localization module that estimated the robot's position on the field, a strategic planning module (often a behavior tree or finite state machine) that selected high-level actions (search, approach, kick), and a low-level walking controller that executed the selected motion primitive. The authors characterize this paradigm in Section 1:

"Such frameworks often separate low-level motor skills from tactical decision-making and rely on manually designed components, leading to decoupled systems that struggle to generate agile and coherent behaviors."

The symptom of this decoupling is behavioral fragmentation. The robot walks to the ball, stops, re-localizes, selects a kick direction, aligns its body, and then executes a preprogrammed kick motion. Each transition introduces a dead time — between 0.5 and 2 seconds — during which the robot is not reacting to the ball. In Section 2.5 (Figure 6), the authors quantify this directly: a state-of-the-art rule-based strategy (used by the RoboCup Humanoid League runner-up team) requires approximately 5 seconds to kick a ball located behind the robot, compared to consistently faster times from the learned policy. The delay arises from:

"the need for extensive rotation and fine positional adjustments around the ball to achieve proper alignment."

This is not a failure of engineering optimization — it is a fundamental consequence of the modular architecture. When perception, planning, and control are separate, information flows between them at the slowest module's update rate, and each module's decisions are made without awareness of the others' internal states. The walking controller doesn't know what the perception module is struggling to see; the planner doesn't know when the walking controller is about to stumble.

Unified RL Policies with Privileged State

A significant advance came from Haarnoja et al. (2024) [12], who demonstrated that a single reinforcement learning policy could be trained end-to-end to play 1v1 soccer — from low-level motor control through mid-level skill selection — using privileged simulation state (exact ball position, opponent location, all proprioceptive signals). This eliminated the fragmentation of modular systems: the policy learned to transition smoothly between approaching, dribbling, and shooting without explicit state machine logic.

However, this approach was trained and evaluated entirely in simulation with perfect state information. The policy never had to deal with perceptual uncertainty — it received ground-truth ball coordinates at every timestep. This is a critical limitation because, as the paper notes, real-world visual perception is not just a degraded version of simulation state; it is qualitatively different: objects temporarily vanish from view (not just become noisier), latency introduces systematic offsets for moving targets, and detection failures are correlated with interesting moments (e.g., when the robot turns quickly to chase a fast-moving ball, exactly when visual feedback is most needed).

Vision-Based Policies with Synthetic Rendering

The most direct predecessor to this work is Tirumala et al. (2024) [14], which extended the unified RL paradigm to vision-based control by rendering egocentric camera views using Neural Radiance Fields (NeRF) during training. The NeRF model was trained on camera images collected in a real soccer environment, then used to synthesize novel views during policy training in simulation. This allowed the policy to learn from visual inputs that approximated real-world appearance — an important step toward sim-to-real transfer.

The authors identify two key limitations of this approach (Section 1). First:

"the resulting controllers exhibited reduced reactivity to the ball"

The NeRF-based rendering, while visually realistic, introduces its own computational burden. Generating photorealistic images at the 25+ Hz rate needed for reactive control is expensive, and the rendering pipeline itself adds latency. More fundamentally, the policy learned to depend on the specific visual environment modeled by the NeRF — lighting, texture patterns, camera artifacts — which may not transfer to new fields, different lighting conditions, or altered camera configurations.

Second:

"[the controllers] relied heavily on specifically modeled visual environments, restricting generalization to real match conditions."

A NeRF trained on one soccer field does not generalize to another. For every new competition venue, a new NeRF would need to be captured and trained — contradicting the RoboCup constraint of limited adaptation time. The paper's approach circumvents this by feeding the policy structured detection outputs (ball position, goal coordinates) rather than raw pixels, abstracting away the visual specifics that vary across environments.

The Broader Imitation Learning Landscape for Humanoids

Beyond soccer specifically, the paper situates itself within the rapidly growing literature on learning from demonstrations for humanoid robots. The authors review two dominant paradigms in Section 1, both of which inform their design decisions but also reveal constraints they aim to overcome.

Feature-based motion imitation (DeepMimic-style): The framework established by Peng et al. (2018) [27] defines dense tracking rewards based on hand-crafted features of demonstrated motions — joint angle errors, end-effector positions, center-of-mass trajectories — computed frame-by-frame between the policy's output and a reference motion clip. This approach, used extensively for stylized walking, dancing, and acrobatic movements [20–25], enforces explicit temporal alignment: at timestep $t$, the policy should match frame $t$ of the reference. The paper identifies a fundamental tension:

"these feature-based methods relied heavily on explicit motion matching, making them less adaptable when deviations from reference trajectories were needed to achieve task-specific objectives, and rigid temporal alignment also limited their flexibility in dynamic environments."

In soccer, this rigidity is fatal. You cannot pre-specify the exact timing of when to raise the kicking leg — it depends on when the robot reaches the ball, which depends on the ball's velocity and the terrain, which the policy discovers online. A feature-based method would require either an impossibly large reference dataset covering every possible approach trajectory or a separate module to dynamically adjust the reference timing (which reintroduces the modular fragmentation the field is trying to escape).

GAN-based motion priors (AMP-style): The Adversarial Motion Priors framework [33] introduced a fundamentally different approach. Instead of explicitly aligning policy outputs to reference frames, AMP trains a discriminator network that learns to distinguish motion transitions $(s_t, s_{t+1})$ generated by the policy from those in a reference dataset. The discriminator's output serves as an implicit style reward — it doesn't tell the policy which reference frame to match, only whether its current motion looks plausible given the reference distribution. This provides:

"implicit motion guidance for the policy to acquire soccer skills aligned with human motion patterns, obviating the need for manual segmentation into discrete behavioral stages."

The policy can autonomously decide when to transition from walking to kicking because the discriminator recognizes both walking transitions and kicking transitions as valid; it penalizes only unnatural movements (e.g., physically implausible joint configurations or awkward gait patterns). The authors note in Section 2.5 that this flexibility enabled novel behaviors not explicitly present in the reference data, such as a pivot hook kick where the robot pivots on its supporting foot to hook the ball while facing backward — a motion the discriminator accepts because it resembles the statistical patterns of the reference kicks, even though no exact example appeared in the training set.

However, all prior AMP applications focused on purely proprioceptive imitation in static environments — characters learning to walk or dance with direct access to their own joint states but no exteroceptive perception. The paper explicitly identifies this gap:

"Despite their promise, existing GAN-based methods primarily focused on motion imitation using proprioception in static scenarios, overlooking the utilization of exteroceptive sensors required in dynamic tasks such as robot soccer."

The fundamental challenge this paper addresses is extending AMP to a setting where the policy's observations include noisy, real-time visual data, and where the quality of those observations depends on the policy's own actions (whether it turns its head to keep the ball in view). This is the "perceptual setting" the paper's title references — AMP must now guide not just how the robot moves, but how it moves its head, torso, and gaze to support perception, creating a bidirectional coupling between perception and action that was absent in prior AMP work.

How This Paper Positions Itself

The paper's positioning can be understood along four dimensions:

1. Unification of perception and control in a single learned policy. Unlike modular systems where perception is an independent preprocessing step, this paper treats perception as part of the control problem. The policy receives raw detection outputs (ball position in robot frame, ball visibility flag, goal direction), not a clean state estimate. It must learn to interpret these signals in the context of its own motion — understanding, for instance, that a rapidly changing ball position estimate during a turn is more likely due to head motion than ball motion, or that a sudden loss of ball detection during a kick is expected (the robot is looking at its feet) rather than a reason to abort. This is a fundamentally different design philosophy from building a better ball tracker and feeding its output to a controller.

2. Extension of AMP to perceptually grounded dynamic control. The paper does not propose a new imitation learning algorithm; it extends an existing one (AMP with Wasserstein GAN stabilization [38]) to a new domain. The contribution is not the adversarial training framework itself, but the demonstration that it works when the policy's observations include noisy exteroceptive signals, and the design choices (virtual perception system, encoder-decoder architecture with reconstruction loss, multi-critic training) required to make it work.

3. Bridging sim-to-real through perceptual abstraction, not photorealism. A key philosophical choice distinguishes this work from visual imitation approaches like the NeRF-based method: rather than making simulation look like reality, the paper makes the information content of simulation match reality. The virtual perception system (Section 4.3, Appendix E) models detection probability, positional noise as a function of distance, latency, and update frequency — all statistical properties of the perception pipeline — while abstracting away RGB appearance entirely. This is akin to domain randomization but applied at the level of structured perception outputs rather than raw pixels. The policy learns robustness to perceptual characteristics rather than visual appearance, which the authors argue enables better generalization to novel visual environments.

4. Practical validation through competition deployment. The paper stakes a significant claim on real-world performance:

"As the champion team of the Adult-size Humanoid League in RoboCup 2025 and the 2025 World Humanoid Robot Games, we further showcased the practical efficacy of this method in real-world competitive settings."

This is not merely an anecdote — the competition setting imposes constraints (single camera, onboard compute only, unknown environment, limited setup time) that directly test the claims of perceptual generalization and reactive control. The quantitative results in Figure 3A — real-world kicking success rates that closely match simulation predictions — provide evidence that the sim-to-real gap has been effectively bridged for this task.

Where This Paper Sits in the Research Landscape

At the time of publication (November 2025), robot soccer had advanced rapidly from modular control through privileged-state RL to vision-based RL, but no system had yet demonstrated:

• Vision-driven reactive soccer skills (tight perception-action loop, not sequential perceive-then-act).
• Human-like motion quality (via imitation learning) combined with task-driven behavioral flexibility.
• Zero-shot sim-to-real transfer across diverse terrains and visual conditions.
• Sustained competitive performance under tournament constraints.

The paper explicitly claims to be the first to achieve all of these simultaneously, not by inventing new algorithmic components but by carefully integrating existing techniques (AMP, POMDP encoder-decoder architectures, multi-critic RL, sim-to-real perception modeling) in a way that addresses the specific failure modes — behavioral fragmentation, slow reactivity, over-dependence on visual environment specifics — that prevented prior work from crossing the gap from laboratory demonstrations to competitive deployment.

3. Technical Approach

3.1 Reader Orientation

This paper develops an end-to-end reinforcement learning controller — a single neural network policy — that maps noisy, delayed visual detections of a soccer ball and goal directly to joint motor commands at 50 Hz, enabling a humanoid robot to chase and kick a ball reactively. The problem it solves is the perception-action coupling bottleneck: prior systems either separated vision processing from motor control (causing sluggish, fragmented behavior) or relied on perfect simulation state (failing to transfer to real hardware). The solution's "shape" is a policy trained entirely in simulation that learns to interpret corrupted visual inputs through a deliberate architecture — an encoder that compresses a 1-second history of observations into a latent representation, a decoder that forces this latent space to retain physically meaningful information by reconstructing privileged state variables, and a virtual perception system that injects realistic noise, latency, and detection failures during training so the policy learns robustness before ever touching hardware.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components, each with a distinct responsibility:

1. The Simulation Environment (Isaac Gym) — a physics simulator running 16,384 parallel copies of a soccer field with a Booster T1 humanoid and a ball. It provides ground-truth state to the critic during training but only corrupted observations to the actor (policy). Episodes randomize ball and robot positions, apply external disturbances, and terminate on falls, goals, out-of-bounds, or a 60-second timeout.

2. The Virtual Perception System — a statistical model running inside the simulation that converts ground-truth ball and goal positions into the same noisy, delayed, intermittently available detection outputs the real robot's camera pipeline produces. It models four factors: detection probability as a function of distance and field-of-view, Gaussian positional noise with variance proportional to distance, latency drawn from a fitted distribution (mean 116 ms), and update frequency jitter (mean 25.36 Hz). This is the linchpin of sim-to-real transfer.

3. The Policy Network (Actor) — a multilayer perceptron with an encoder-decoder architecture that receives 50 frames (1 second) of observation history. The encoder compresses this temporal window into a 64-dimensional latent vector; the decoder reconstructs privileged state variables (ball true position, robot dynamics parameters) from this latent as an auxiliary training objective. The actor then maps the latent plus current observation to desired joint positions, which are tracked by PD controllers.

4. The Value Network (Critic) — a separate MLP that estimates state value for PPO training. Critically, it receives privileged simulator state (ground-truth linear velocity, ball velocity, base height, mass randomization parameters) that the actor never sees. This is the asymmetric actor-critic paradigm: the critic has more information to provide accurate value estimates, but the actor learns to approximate those values from partial observations alone.

5. The Discriminator (AMP Style Module) — a Wasserstein GAN discriminator trained to distinguish motion transitions $(s_t, s_{t+1})$ from a reference dataset of human-captured walking and kicking motions versus transitions generated by the policy. Its output provides an adversarial style reward that encourages the policy to move in human-like ways without enforcing temporal alignment to any specific reference clip.

Information flows as follows: at each timestep, ground-truth simulation state passes through the virtual perception system, producing corrupted ball position, ball visibility mask, goal position, and goal direction. These are concatenated with proprioceptive observations (joint positions, velocities, IMU readings, previous action) and fed into the actor's encoder, which maintains a 1-second rolling buffer. The encoder produces a latent vector; a decoder branch predicts privileged quantities from this latent for auxiliary supervision. The actor network outputs target joint positions, which a PD controller converts to torques applied in the physics simulator. Simultaneously, a PPO update uses the critic's value estimates (computed from privileged state) and the discriminator's style reward (computed from state transitions) to update both actor and critic parameters.

3.3 Roadmap for the Deep Dive

• First, the POMDP formulation and asymmetric actor-critic framework — this establishes the mathematical foundation, defining what "solving the control problem" means when the robot cannot access true state.
• Second, the training environment design — the physics, randomization, episode structure, and reward engineering — because the policy's eventual behavior is fundamentally shaped by what it is rewarded for and what disturbances it experiences during training.
• Third, the perception pipeline, including both the real hardware perception system and the virtual perception system used in simulation — since bridging sim-to-real depends entirely on how faithfully the virtual system replicates real-world perceptual characteristics.
• Fourth, the encoder-decoder architecture and latent state reconstruction — the mechanism that enables the policy to denoise observations and maintain state estimates across perceptual dropouts.
• Fifth, the AMP-based motion imitation framework — how the discriminator provides style guidance and why Wasserstein GAN stabilization is necessary.
• Sixth, the multi-critic training architecture — how and why reward decomposition improves training stability for this task.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems integration and empirical validation paper whose core idea is that a unified RL policy, trained with a virtual perception system and AMP-style motion priors, can learn reactive vision-driven soccer skills that transfer zero-shot to real hardware under competitive constraints. No single algorithmic component is novel; the contribution lies in the specific combination and the demonstration that this combination solves a previously open problem.

---

POMDP Formulation and Asymmetric Actor-Critic

The authors frame the control problem as a Partially Observable Markov Decision Process (POMDP) (Section 4.2). In a standard MDP, the agent observes the full state $s_t$ of the environment at each timestep and chooses an action $a_t$ to maximize expected cumulative reward. In a POMDP, the agent receives only an observation $o_t$, which is a (possibly stochastic) function of the true state: $o_t \sim O(s_t)$. The observation is incomplete and noisy — in this case, because the camera provides only estimated ball position with delay and noise, and cannot observe true ball velocity, ground friction parameters, or mass randomization.

The core challenge of a POMDP is that the optimal action at time $t$ may depend not just on the current observation but on the entire history of observations $o_{0:t}$, since past observations contain information about the underlying state that a single observation cannot recover. The authors operationalize this by feeding the policy network a temporal window of 50 observation frames (covering 1 second at the 50 Hz policy rate) plus the current observation:

"We provide the network with a time sequence of 50 preceding observation frames (covering 1 s of history) alongside the current observation." (Section 4.2)

The specific training algorithm is Proximal Policy Optimization (PPO) [49] with an asymmetric actor-critic (AAC) architecture [48]. In standard actor-critic, both actor and critic receive the same observations. In AAC:

• The critic (value network) receives the full privileged state available from the simulator, including quantities the real robot cannot measure: base linear velocity, ball velocity, base height above ground, randomized mass and center-of-mass offsets, and ball friction forces. This gives the critic an information advantage, allowing it to produce more accurate value estimates $V(s_t)$, which reduces variance in the advantage estimates used for policy gradient updates.

• The actor (policy network) receives only the partial, noisy observations that would be available on hardware: projected gravity, angular velocity, joint positions and velocities, previous action, ball position in robot frame, a binary ball detection mask, goal position and direction. These are corrupted by the virtual perception system during training. The actor never sees ground-truth velocity or dynamics parameters.

At deployment time, the critic is discarded entirely — it serves only as a training crutch to provide stable value estimates. The actor, which has learned to approximate optimal behavior from partial information alone, runs on the physical robot's Jetson AGX Orin.

The action space consists of desired joint positions, specifically 10 leg joints (5 per leg: hip yaw, hip roll, hip pitch, knee pitch, ankle pitch) plus 2 head joints (yaw and pitch). These desired positions are tracked by a low-level proportional-derivative (PD) controller running at the simulator's physics timestep, which is higher than the policy's 50 Hz update rate. The policy does not output torques directly — this PD interface provides a natural abstraction layer that helps with sim-to-real transfer, as the physical robot also uses PD tracking for joint commands.

Why this formulation: The POMDP framing is necessary because real-world perception is inherently partial — the robot genuinely cannot observe true state. Training the actor under this information constraint forces it to develop internal state estimation (via the encoder's latent representation) rather than relying on simulation-only signals. The AAC paradigm is chosen over symmetric actor-critic because providing the critic with ground-truth state significantly reduces the variance of value estimates without introducing any information leakage to the actor. An alternative would be to train the actor with ground-truth state, then hope it generalizes to noisy real-world observations — this almost never works, because the policy learns to depend on signals that don't exist on hardware. The historical observation window of 1 second is a design choice balancing computational cost (longer windows mean larger networks and slower inference) against information content (shorter windows provide insufficient context to infer velocities and filter noise).

---

Training Environment Design

Simulation platform and scale. Training uses NVIDIA Isaac Gym, a GPU-accelerated physics simulator, running 16,384 parallel environments across 8 NVIDIA V100 GPUs (Appendix A). Each environment contains one Booster T1 humanoid robot and one ball on a soccer field. Training runs for 20,000 epochs, requiring approximately 1 day to complete. The massive parallelism (16,384 simultaneous rollouts) is essential for PPO's sample efficiency — each epoch collects 16,384 environment steps of experience before performing a policy update.

Field and physics configuration. The field dimensions follow the RoboCup Adult-size Humanoid League specification: 14 m length, 9 m width, with goals 2.6 m wide at opposite ends (Section 4.1). The ball's physical properties (mass, friction, restitution) are randomized within a range approximating standard Size 5 soccer balls:

"the physical properties of the ball are randomized within a certain domain" (Section 4.1)

This prevents the policy from overfitting to a specific ball physics model. Despite the real-world field being flat, the authors introduce small uneven terrain perturbations during training:

"small uneven terrain is incorporated during the training process to facilitate more stable locomotion in real-world scenarios" (Section 4.1)

This is a standard sim-to-real robustness technique — training on slightly non-flat terrain makes the policy's locomotion more tolerant of real-world surface irregularities, sensor calibration errors, and foot placement imprecision.

Episode structure and resets. Each episode begins with the robot and ball randomly initialized within the field (Section 4.1). An episode terminates on any of four conditions:

• The robot falls (presumably detected by base orientation exceeding a threshold or base height dropping below some limit).
• The ball goes out of bounds.
• A goal is scored.
• A time limit of 60 seconds is reached.

Critically, when the ball goes out of play or enters the goal, only the ball is reset — the robot's state is preserved:

"when the ball goes out of bounds or enters the goal, only the ball's position is reset, while the robot's state remains unchanged. To facilitate the robot's learning of continuous kicking skills..." (Section 4.1)

This design choice is motivated by the need for the robot to learn continuous, sequential kicking rather than treating each kick as an isolated episode. After scoring, the robot remains standing on the field; a new ball appears, and the robot must locate it and kick again. This encourages the policy to develop robust standing and searching behaviors rather than assuming it will always start from a neutral pose.

Disturbance injection. To simulate competitive match conditions, the environment applies random perturbations:

• Ball perturbations: The ball is randomly given an additional velocity or teleported to a new position with some probability, mimicking opponent interventions or referee ball placements.
• Robot perturbations: External forces or extra velocities are applied to the robot's base, simulating physical contact with opponents during gameplay.

These disturbances force the policy to develop recovery behaviors — if the robot gets pushed while approaching the ball, it must adjust its foot placement and regain balance rather than blindly continuing its planned trajectory. This is essential for competition deployment, where physical contact between robots is common.

---

Reward Engineering

The reward function (Appendix B, Table 3) is a carefully weighted combination of three categories: task rewards related to soccer objectives, style rewards from the AMP discriminator, and regularization rewards that penalize undesirable behaviors.

Task rewards. The primary objective — scoring goals — is extremely sparse; a robot might take hundreds of thousands of timesteps before its first accidental goal. To provide learning signal, the authors use two dense auxiliary rewards based on potential-based reward shaping [53]:

1. Ball approach reward (weight 50): The negative change in Euclidean distance between the robot and the ball between consecutive timesteps. If the distance decreases, the reward is positive. Formally, if $d_{rb}$ is the robot-to-ball distance, the reward at timestep $t$ is proportional to $-(d_{rb}^t - d_{rb}^{t-1})$, which equals $d_{rb}^{t-1} - d_{rb}^t$.

2. Goal progress reward (weight 500): The negative change in Euclidean distance between the ball and the goal center. This encourages the robot to kick the ball toward the goal, not just approach it randomly.

3. Goal scored reward (weight 15): A terminal bonus when the ball enters the goal area.

The potential-based shaping formulation has a formal property: if the shaping reward is the difference of a potential function $\Phi(s_{t+1}) - \Phi(s_t)$ (scaled), then the optimal policy under the shaped reward is identical to the optimal policy under the original sparse reward — the shaping does not change the policy's objective, only provides denser feedback. The authors use $\Phi_{ball}(s) = -d_{rb}$ and $\Phi_{goal}(s) = -d_{bg}$ (where $d_{bg}$ is ball-to-goal distance), which satisfies this property.

Additional task-related rewards include:

• Survival reward (weight 3): A constant small positive reward for each timestep the robot remains operational, encouraging the policy to avoid early termination.
• Termination penalty (weight −1000): Applied when the robot falls.
• Stagnation penalty (weight −100): Applied when the robot remains nearly motionless for 1 second, preventing the policy from learning a "freeze in place" strategy to avoid falling.
• Sideways kick reward (weight 20): Encourages lateral foot movement when in contact with the ball, promoting arch-based kicking (where the foot swings sideways to strike the ball with the inside of the foot).
• Forward kick penalty (weight −20): Penalizes forward foot movement during ball contact, discouraging toe-kicks which are less accurate and harder to control.

Head alignment rewards. To encourage active visual tracking:

• Head pitch alignment (weight −0.5): Penalizes the squared difference between the head pitch angle and the ball's elevation angle relative to the robot.
• Head yaw alignment (weight −0.5): Penalizes the squared difference between the head yaw angle and the ball's azimuth.

These are negative rewards (penalties) that push the policy to orient its head toward the ball, increasing the probability that the ball falls within the camera's field of view.

Regularization rewards. A set of penalties discouraging physically undesirable behaviors:

• Foot proximity penalty (weight −5): Applied when the feet are too close together, preventing self-collision.
• Head action rate penalty (weight −15): Penalizes large changes in head joint commands between consecutive timesteps, preventing jerky head motions that would cause motion blur in real camera images.
• Leg action rate penalty (weight −1): Penalizes large changes in leg joint commands, encouraging smooth motion.
• Joint position limit penalty (weight −100): Applied when any joint exceeds its physical limits.
• Base acceleration penalty (weight −0.001): Penalizes excessive linear acceleration of the robot's base (torso), discouraging violent motions that could damage hardware.
• Collision penalty (weight −100): Applied when body parts other than the feet contact the ground or ball, preventing the robot from using its hands or knees.

Why this reward structure: The dense shaping rewards (ball approach, goal progress) solve the credit assignment problem for long-horizon behavior. Without them, the policy would receive a single positive signal (goal scored) after potentially thousands of actions, making it nearly impossible to identify which actions contributed to success. The potential-based formulation ensures that the shaping does not bias the policy toward a locally-optimal but globally suboptimal strategy (e.g., always approaching the ball but never kicking). The head alignment rewards explicitly encourage active perception — the policy is not just told to play soccer, but to look at the ball while doing so. The regularization terms prevent reward hacking (e.g., learning to fall in a way that satisfies some reward component) and ensure the resulting motions are physically realizable on hardware.

---

The Perception Pipeline

The perception system operates at two levels: the real-world pipeline running on the physical robot's Jetson AGX Orin, and the virtual perception system that replicates its statistical characteristics during simulation training.

Real-world perception (hardware). The robot is equipped with an Intel RealSense Depth Camera D435i mounted on the head, which has two actuated joints (yaw and pitch) allowing the policy to actively control gaze direction (Section 2.1). The camera streams RGB images at 25 Hz.

A YOLOv8 object detection model [59], fine-tuned on a self-collected dataset from the robot's onboard camera, detects the ball and field landmarks (goalposts, T-intersections, X-intersections, L-intersections) from each RGB frame (Section 4.3). The 2D pixel coordinates of these detections are then projected into Bird's Eye View (BEV) space — a top-down representation in the robot's local coordinate frame — using a fusion of:

1. Depth-based projection: Using depth information from the RealSense D435i's stereo infrared sensors to estimate the 3D position of detected objects.
2. Geometric projection: Using known camera height and orientation to project detections onto the ground plane, assuming the ball lies on the ground (a reasonable approximation for soccer).

The BEV ball position is passed directly to the policy as (ball_x, ball_y) in the robot's coordinate frame, along with a binary ball mask indicating whether the ball was detected in the current frame.

Field landmark BEV positions are processed by a separate odometry module (Appendix F) that fuses them with proprioceptive data to produce estimates of the goal center position and goal normal direction in the robot frame. The odometry module combines:

• Proprioceptive odometry: An MLP trained on simulation data that predicts the robot's position displacement from a 1-second history of joint states, base orientation, angular velocity, and previous actions. It operates autoregressively, feeding its own previous output back as input to mitigate drift.
• Visual corrections: A particle filter that matches detected field landmarks to a predefined global field map, providing periodic absolute position updates to correct proprioceptive drift.

This hybrid approach ensures the robot maintains a goal estimate even when visual landmarks are temporarily occluded. The final perception output to the policy consists of only four quantities plus proprioception: (ball_x, ball_y, ball_mask, goal_x, goal_y, goal_dir_cos, goal_dir_sin). This compact, structured representation abstracts away all visual complexity (lighting, texture, background clutter) while preserving the task-relevant geometric information.

Why this abstraction over raw pixels: The authors explicitly justify this design choice:

"By focusing solely on environmental cues critical to soccer performance, our framework promotes the development of robust, generalizable behaviors with significantly reduced training overhead." (Section 4.3)

Training on raw RGB images would require the policy to learn visual feature extraction as a subtask, dramatically increasing the required training data and compute, and would make the policy dependent on the specific visual appearance of the training environment. By separating visual processing (YOLOv8 detection) from motor control (RL policy), each component can be optimized independently, and the policy generalizes to any environment where the detection model works — which is a much easier transfer problem than generalizing pixel-level policies.

Virtual perception system (simulation). During simulation training, ground-truth ball and goal positions are available, but feeding these directly to the policy would create an unbridgeable sim-to-real gap — the policy would learn to depend on perfect, zero-latency, continuous state information that does not exist on hardware. The virtual perception system transforms ground-truth simulation state into statistically realistic observations that match the characteristics of the real perception pipeline.

The system models four perceptual factors (Section 4.3, Appendix E, Figure 9):

1. Positional noise. The ball position reported to the policy is corrupted by additive Gaussian noise whose variance scales linearly with the distance between the robot and the ball. Specifically, noise is sampled from:

$\text{noise} \sim \mathcal{N}\left(0, (0.124 \cdot d + 0.149)^2\right)$

where $d$ is the Euclidean distance (in meters) between the robot's camera and the ball, $0.124$ is the slope (noise increases by 0.124 m per meter of distance), and $0.149$ is the intercept (base noise at zero distance, in meters).

What it computes: For a given true distance $d$, the standard deviation of the noise is $\sigma(d) = 0.124d + 0.149$. A random noise sample is drawn from a zero-mean Gaussian with this standard deviation and added to the true ball position. At 1 meter distance, the noise has standard deviation approximately 0.273 m; at 5 meters, approximately 0.769 m. This captures the physical reality that camera-based position estimates become more uncertain as objects move farther away (fewer pixels on the sensor, greater sensitivity to calibration errors).

Why this form: The linear model is a pragmatic approximation. The true relationship between distance and positional uncertainty depends on camera resolution, lens distortion, depth estimation error (which grows quadratically with distance for stereo cameras), and detection model confidence. The authors fit the linear model from data collected by running the real robot's perception pipeline while a motion capture system provided ground-truth ball positions. The linear form is simple enough to be statistically robust (two parameters fit from ~1 hour of data) while capturing the first-order effect. A quadratic model would be more physically accurate but risks overfitting the specific calibration of the data collection setup.

2. Detection probability. The ball is not always detected, even when within the camera's field of view. The virtual system models detection as a Bernoulli random variable: with probability $p_{\text{detect}}(d)$, the ball mask is set to 1 (detected) and the noisy position is provided; with probability $1 - p_{\text{detect}}(d)$, the mask is set to 0 and the ball position is set to a sentinel value (presumably the last known position or zeros). The detection probability function is:

$p_{\text{detect}}(d) = \begin{cases} 0.9 & \text{if } d \leq 7\text{ m} \\ \text{gradually decaying} & \text{if } d > 7\text{ m} \end{cases}$

What it computes: Within 7 meters range, the ball is detected 90% of the time it is in the field of view, meaning 10% of frames will have false-negative detections even when the ball is plainly visible. Beyond 7 meters, the detection rate decays, though the authors do not specify the exact decay function in the main text (it is described qualitatively in Appendix E and Figure 9D).

Why this form: The 90% within-range detection rate and 7-meter threshold are empirically measured from the real YOLOv8 detection pipeline's performance. The false negatives within range correspond to real failure modes: motion blur during rapid turns, the ball being partially occluded by the robot's own body, or the ball appearing at an unusual angle relative to the training data. Exposing the policy to these failures during training forces it to learn compensation strategies — maintaining a belief state about where the ball probably is during brief detection dropouts, rather than assuming detection failure means the ball has vanished.

3. Perception latency. Between the moment a camera image is captured and the moment the processed ball position is available to the policy, real-world processing introduces delay: camera exposure time, image transfer to the Jetson, YOLOv8 inference, BEV projection, and inter-process communication. The virtual system models this as a Gaussian-distributed delay:

$\text{latency} \sim \mathcal{N}\left(116\text{ ms}, (18\text{ ms})^2\right)$

What it computes: The ball position provided to the policy at timestep $t$ corresponds to where the ball was at time $t - \ell$, where $\ell$ is a random latency drawn from a normal distribution with mean 116 ms and standard deviation 18 ms. The true ball position from the simulation history buffer at that delayed timestamp is retrieved, and noise is applied to that historical position.

Why this form: A fixed latency could be compensated for by the policy learning to extrapolate forward by a constant time offset. A random latency prevents this, forcing the policy to treat timing as uncertain. The Gaussian distribution approximately captures the sum of multiple independent processing stages (by the Central Limit Theorem). With a mean of 116 ms and a policy running at 50 Hz (20 ms period), the latency spans roughly 5–7 policy steps — meaning the policy often receives visual information that is 5–7 frames out of date. At a typical walking speed of 0.5 m/s, a 116 ms delay corresponds to a position error of approximately 5.8 cm from robot ego-motion alone, plus additional error from ball motion.

4. Update frequency jitter. The real camera runs at 25 Hz nominally, but the actual arrival time of new detections varies due to operating system scheduling, variable network inference time, and detection model batching. The virtual system models the inter-detection interval as:

$\text{interval} \sim \mathcal{N}\left(\frac{1}{25.36}\text{ s}, \left(\frac{1}{1.06}\text{ s}\right)^2\right)$

which corresponds to approximately:

$\text{frequency} \sim \mathcal{N}\left(25.36\text{ Hz}, (1.06\text{ Hz})^2\right)$

What it computes: Between detection updates, the policy reuses the last known ball position (with the ball mask optionally indicating stale data). A new detection arrives at random intervals with mean period 39.4 ms (25.36 Hz) and standard deviation 0.89 ms. Since the policy runs at 50 Hz (20 ms period), new detections arrive approximately every other policy step on average, but with jitter.

Why this form: This prevents the policy from learning to expect visual updates at a precise cadence. In the real system, a detection might arrive two policy steps in a row (if the YOLOv8 inference happened to complete just before the next policy tick) or might skip two steps (if the Jetson was briefly busy with other processes). Training with jitter ensures the policy's internal state estimation (via the encoder) doesn't become brittle to these timing variations.

Data collection for virtual perception modeling. The parameters of these four models were fit from approximately 1 hour of data collected by running the robot's default walking gait with a rule-based ball-tracking program while a human operator moved the ball around the field (Section 4.3). A motion capture system provided ground-truth robot and ball positions, while the robot's perception pipeline logged its detected ball positions and joint states. The discrepancy between ground-truth and detected positions directly reveals the noise distribution; the timestamps of detection outputs relative to image capture times reveal latency and update frequency. The authors note a potential limitation:

"although the walking gait used for data collection differs from the learned policy, which introduces potential distribution discrepancies, the simplicity of our perceptual modeling ensures strong generalization across locomotion patterns." (Section 4.3)

The perceptual noise characteristics depend primarily on the camera-to-ball geometry (distance, angle) rather than the specific gait producing that geometry. Since the learned policy produces qualitatively different locomotion (more dynamic, tighter turns), the noise distribution might shift slightly, but the parametric form (Gaussian with distance-proportional variance) is general enough to accommodate this.

---

Encoder-Decoder Architecture and Latent State Reconstruction

The policy network architecture is designed to address the core challenge of POMDPs: extracting useful state estimates from a history of noisy, incomplete observations. The architecture (Section 4.2, Appendix A, Table 1) consists of three MLP sub-networks:

Encoder. A multilayer perceptron with layer sizes (1024, 128) (two hidden layers: 1024 units then 128 units, both using ELU activation). It receives a concatenated vector of 50 preceding observation frames (each frame being the full observation vector: proprioception, ball position, ball mask, goal information) plus the current observation — 51 frames total. This temporal window covers exactly 1 second of history at the 50 Hz policy rate.

The encoder compresses this high-dimensional input (51 frames × observation dimension) into a 64-dimensional latent vector $z_t$. This latent representation is the policy's internal "belief state" — an implicitly learned summary of everything the policy knows about the world that is not directly observable from the current frame alone.

Actor. A separate MLP with layer sizes (256, 256, 128) (three hidden layers, ELU activation). It receives the 64-dimensional latent vector $z_t$ concatenated with the current observation $o_t$ (not the full history — only the current frame) and outputs the action: desired joint positions for the 10 leg joints (5 per leg: hip yaw, hip roll, hip pitch, knee pitch, ankle pitch) and 2 head joints (yaw and pitch). The action is a vector of 12 continuous values, each representing a target position for a PD controller.

Decoder. A separate MLP with layer sizes (128, 128) (two hidden layers, ELU activation). It receives only the latent vector $z_t$ (not the current observation) and outputs predictions of privileged state variables that are available in simulation but not on hardware:

• Ball position $(x, y)$ in robot frame (the true position, not the noisy observation).
• Ball velocity $(v_x, v_y)$ in world frame.
• Ball friction forces $(f_x, f_y)$ acting on the ball.
• Robot's base linear velocity $(v_x, v_y, v_z)$ in robot frame.
• Robot's base height above ground.
• Mass randomization parameters (the randomized mass and center-of-mass offsets applied to the robot's base link during domain randomization).

The complete observation and state spaces are detailed in Appendix A, Table 2.

Reconstruction loss. The decoder is trained with a mean squared error (MSE) loss between its predictions and the ground-truth privileged values from the simulator:

$\mathcal{L}_{\text{recon}} = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i^{\text{true}})^2$

where $\hat{y}_i$ is the decoder's prediction for privileged variable $i$, $y_i^{\text{true}}$ is the ground-truth value from the simulator, and $N$ is the total number of privileged variables being reconstructed.

What it computes: For each privileged quantity (ball true position, ball velocity, etc.), the decoder produces a scalar estimate from the compressed latent vector $z_t$. The MSE between this estimate and the simulator ground truth is computed and added to the total training loss, weighted by a reconstruction coefficient of 1 (Appendix A, Table 1). This means the reconstruction loss has equal weight to the PPO policy loss in the overall optimization.

Why this form: The reconstruction loss serves as an auxiliary task that shapes the latent representation without requiring any additional real-world labels — it only uses privileged information available in simulation. By forcing the encoder to preserve information needed to predict true ball position, velocity, and dynamics parameters, the decoder ensures that the latent vector $z_t$ contains physically meaningful state estimates, not just arbitrary features that happen to correlate with good actions. This is crucial for sim-to-real transfer: when deployed on hardware, the actor still receives this same encoder-produced latent vector (the decoder is discarded at deployment), so the latent space must be sufficiently informative to support accurate control from partial observations alone. The weight of 1 is set to balance the auxiliary task against the primary RL objective — too high a weight would cause the encoder to prioritize reconstruction accuracy over action quality, while too low a weight would provide no meaningful regularization.

The authors validate this design with an ablation (Section 2.4):

"If the decoder was removed during policy training and trained separately afterward to estimate the ball position from latent states, the resulting predictions remained at the noise level."

In other words, without the decoder guiding the latent space during training, the encoder learns a representation that is sufficient for the policy's immediate action selection (which can be reactive and short-sighted) but does not contain the long-horizon state estimation needed to filter noise or predict ball motion. Training the decoder jointly with the policy forces the latent space to serve both purposes simultaneously — supporting immediate action while maintaining accurate world state estimates.

Why 1 second of history: The 50-frame (1 second) window is chosen to balance several constraints. The robot's typical walking frequency is around 1–2 Hz (a full gait cycle takes 0.5–1 second), so 1 second captures at least one full stride. Ball velocities are on the order of 1–5 m/s, so 1 second of history captures 1–5 meters of ball displacement — enough to estimate velocity from position changes. Longer windows would require larger encoder networks (more parameters) and would increase inference latency on the Jetson (more memory and compute for the rolling buffer). Shorter windows, as discussed above, would not provide sufficient temporal context.

Network architecture choices. The encoder is substantially wider (1024→128) than the actor and decoder (256,256,128), reflecting its role in compressing a high-dimensional temporal input (51 frames × ~30 observation dimensions ≈ 1500+ input features) into a compact 64-dimensional code. The ELU (Exponential Linear Unit) activation is chosen over ReLU for its smooth gradients and non-zero output for negative inputs, which can help with training stability in RL where value estimates and advantages can be negative. The architecture uses relatively shallow networks (2–3 hidden layers) with moderate widths (128–1024), keeping inference time low for real-time 50 Hz control.

---

Adversarial Motion Priors (AMP) for Style Guidance

The AMP framework [33] provides an implicit motion style reward without requiring temporal alignment between policy outputs and reference motions. The implementation uses a Wasserstein GAN formulation with gradient penalty [38, 39] to stabilize training (Section 4.4, Appendix C).

Reference motion dataset. The authors curate a combined dataset of human motion capture data:

• 76.28 seconds of omnidirectional walking from the ACCAD dataset [60], covering forward, backward, turning, and side-stepping motions.
• 30 seconds of arch-based kicking motions recorded with an optical motion capture system.

These human motions are retargeted to the Booster T1 robot's kinematics — the human joint angles are mapped to the robot's corresponding joints, scaled to match the robot's different limb proportions and joint limits. The resulting dataset contains motion clips that demonstrate how to walk and kick in a human-like manner, but not when to perform each action. The total dataset size (approximately 106 seconds at the motion capture framerate) is relatively small, reflecting the fact that AMP does not require exhaustive coverage of all possible behaviors — it only needs enough variety for the discriminator to learn a plausible motion manifold.

Discriminator architecture and training. The discriminator $D_\phi$ is a neural network (architecture not specified in detail, but described as an MLP in Figure 8) that takes as input a state transition $x_t = (s_t, s_{t+1})$ — the concatenation of two consecutive proprioceptive states (joint positions, velocities, base orientation, etc.) — and outputs a scalar score $D_\phi(x_t)$. The score should be high for transitions from the reference dataset ("real" samples) and low for transitions generated by the policy ("fake" samples).

The discriminator is trained to minimize a Wasserstein GAN loss:

$\mathcal{L}_D = -\mathbb{E}\left[\tanh(0.4 D_\phi(x_t^E))\right] + \mathbb{E}\left[\tanh(0.4 D_\phi(x_t^\pi))\right]$

where $x_t^E$ is a transition sampled from the reference (expert) dataset, $x_t^\pi$ is a transition sampled from the policy's recent rollouts, and $\tanh$ is the hyperbolic tangent function applied with a scaling factor of 0.4.

What it computes: The discriminator produces a raw scalar score. The $\tanh(0.4 \cdot)$ transformation squashes this score into the range $[-1, 1]$ and compresses large magnitudes, preventing the discriminator from producing extreme values early in training. The loss maximizes the difference between the expected (transformed) score for expert transitions and the expected score for policy transitions — this is the Wasserstein distance approximation. Minimizing $\mathcal{L}_D$ means making $\tanh(0.4 D_\phi(x^E))$ as large as possible and $\tanh(0.4 D_\phi(x^\pi))$ as small as possible.

Why this form: The standard GAN loss (binary cross-entropy) suffers from saturation when the discriminator becomes too confident — if $D_\phi(x^E) \approx 1$ and $D_\phi(x^\pi) \approx 0$, the gradient for the generator vanishes. The Wasserstein formulation avoids this by using an unbounded critic whose output represents the "Earth Mover's Distance" between distributions. The $\tanh$ transformation with coefficient 0.4 is a practical stabilization technique (introduced in [38]) that prevents discriminator output from growing unboundedly while still providing meaningful gradients. The coefficient 0.4 is an empirical choice that balances gradient signal strength against training stability.

Gradient penalty. To enforce the Lipschitz continuity constraint required by the Wasserstein GAN formulation, a gradient penalty is added:

$\mathcal{L}_{\text{grad}} = \mathbb{E}\left[\left(\|\nabla D_\phi(\hat{x}_t)\| - 1\right)^2\right]$

where $\hat{x}_t = \alpha x_t^E + (1 - \alpha) x_t^\pi$ is a random convex combination (interpolation) between an expert transition and a policy transition, with $\alpha \sim \mathcal{U}(0, 1)$ drawn uniformly from the unit interval.

What it computes: A random point $\hat{x}_t$ is interpolated between a real and fake sample. The gradient of the discriminator output with respect to this interpolated point is computed via automatic differentiation. The penalty $(\|\nabla D_\phi\| - 1)^2$ pushes the gradient norm toward 1 — the condition required for the discriminator to be 1-Lipschitz, which ensures the Wasserstein distance is well-defined.

Why this form: Without the gradient penalty, the discriminator could achieve arbitrarily low loss by making its output function arbitrarily steep, which would produce vanishing or exploding gradients for the policy. The gradient penalty (weighted at 50.0 in Appendix A, Table 1, listed as "Gradient penalty coefficient") is the standard approach from Gulrajani et al. (2017) and is applied to interpolated points rather than real or fake points individually because the Lipschitz condition must hold everywhere, and interpolated points cover the region between the two distributions where the discriminator is most likely to have pathological behavior.

Style reward for the policy. The discriminator's output is converted into a reward signal for the policy:

$r_{\text{amp}} = -\tanh(0.4 D_\phi(x_t^\pi))$

This is the negative of the transformed discriminator score for policy-generated transitions, weighted by 0.3 in the total reward (Table 3).

What it computes: When the discriminator judges a policy transition as expert-like (high $D_\phi$), $r_{\text{amp}}$ is negative but small in magnitude (because $\tanh$ saturates near 1). When the discriminator judges a transition as non-expert-like (low $D_\phi$), $r_{\text{amp}}$ is positive (because $\tanh$ of a negative number is negative, and the negative of that is positive — wait, this needs clarification).

Let's trace through carefully: The discriminator loss pushes $D_\phi(x^E)$ to be high (positive) and $D_\phi(x^\pi)$ to be low (negative). So for a policy transition, $D_\phi(x^\pi)$ is ideally a negative number. Then $\tanh(0.4 D_\phi(x^\pi))$ is also negative (tanh of a negative is negative, in the range (-1, 0)). Then $r_{\text{amp}} = -\tanh(0.4 D_\phi(x^\pi))$ is positive, meaning the policy gets a reward for being classified as expert-like. The closer $D_\phi$ is to negative infinity (very expert-like), the more negative $\tanh$ becomes (approaching -1), and $r_{\text{amp}}$ approaches +1 — the maximum style reward. Actually, re-reading the formulation more carefully: the discriminator minimizes $-\mathbb{E}[\tanh(0.4D_\phi(x^E))] + \mathbb{E}[\tanh(0.4D_\phi(x^\pi))]$, so it wants $\tanh(0.4D_\phi(x^E))$ to be large (positive, close to +1) and $\tanh(0.4D_\phi(x^\pi))$ to be small (negative, close to -1). This means expert transitions get positive discriminator scores, policy transitions get negative scores. The reward $r_{\text{amp}} = -\tanh(0.4 D_\phi(x^\pi))$ is then indeed positive for expert-like policy motions (since $D_\phi(x^\pi)$ is negative, tanh is negative, negative of that is positive).

Why this form: The negative sign converts the discriminator's assessment into a reward (higher is better). The $\tanh$ squashing prevents the reward from having extreme magnitudes, which would destabilize RL training. The weight of 0.3 balances style against task objectives — too high a weight would cause the policy to prioritize looking human-like over scoring goals; too low a weight would cause the policy to ignore the reference motions entirely and potentially develop unnatural but task-effective gaits that wouldn't transfer to hardware (because the discriminator also implicitly encodes physical plausibility constraints).

Training dynamics and stabilization. Several additional techniques stabilize AMP training:

• Reference state initialization (RSI): At the start of each episode, the robot's joint configuration is randomly sampled from a reference motion clip rather than from a default pose. This ensures the policy's initial state distribution overlaps with the discriminator's training distribution, preventing early training collapse where the discriminator trivially distinguishes policy states from reference states.
• Adversarial training loop: The discriminator and policy are updated in alternating fashion. The discriminator is trained on batches mixing recent policy rollouts and reference samples; the policy is then updated using PPO with the style reward included.
• Mirror symmetry loss (Appendix D, Equation 6): To prevent the policy from learning to always kick with one foot:

$\mathcal{L}_{\text{sym}} = \mathbb{E}\left[\|a_t - M_a(\tilde{a}_t)\|^2\right]$

where $a_t$ is the action for the original observation, $\tilde{a}_t = \pi_\theta(M_o(o_t))$ is the action for a mirrored observation (left-right swapped), and $M_a$ is the mirroring operator applied to the action. This loss encourages the policy to respond symmetrically — if the ball is on the left, the left-foot action should be the mirror of what the right foot would do for a ball on the right. The symmetry loss is weighted at 10.0 (Appendix A, Table 1).

---

Multi-Critic Architecture for Reward Decomposition

The total reward function combines heterogeneous objectives: task rewards (scoring goals, approaching the ball), style rewards (looking human-like), and regularization rewards (penalizing undesirable behaviors). The authors observe that combining these into a single scalar reward and training a single critic leads to training instability:

"using a single critic to estimate combined rewards can cause negative interference between distinct reward components, leading to reduced learning stability and performance" (Section 4.2, Figure 4A)

Two-critic decomposition. The solution is a multi-critic framework [54] where separate critics estimate returns for two reward groups:

1. Goal-related critic: Trained on rewards associated with task progress — goal scored (15), ball approach (50), and goal progress (500). This critic estimates $V_{\text{goal}}(s)$, the expected cumulative task reward from state $s$.

2. Auxiliary critic: Trained on all remaining rewards — survival (3), termination (−1000), stagnation (−100), head alignment (−0.5 each), AMP style (0.3), sideways kick (20), forward kick (−20), foot proximity (−5), action rate penalties (−15, −1), joint limit (−100), base acceleration (−0.001), and collision (−100). This critic estimates $V_{\text{aux}}(s)$, the expected cumulative auxiliary reward.

Each critic is a separate MLP (same architecture as the value network: layers (256, 256, 128), ELU activation) that receives the full privileged state and outputs a scalar value estimate. They are trained independently using the standard PPO value loss on their respective reward subsets.

Combined advantage. PPO uses advantage estimates (the difference between observed returns and the critic's value prediction) to update the policy. With two critics, the combined advantage is a weighted sum:

$A_{\text{total}} = w_{\text{goal}} A_{\text{goal}} + w_{\text{aux}} A_{\text{aux}}$

where $A_{\text{goal}}$ is the advantage estimated by the goal-related critic, $A_{\text{aux}}$ is the advantage estimated by the auxiliary critic, and the weights are $w_{\text{goal}} = 2$ and $w_{\text{aux}} = 1$ (Appendix B, Equation 1).

What it computes: For each timestep, the goal-related critic provides an estimate of how much better or worse the observed return was compared to its expectation for task-related rewards; the auxiliary critic provides the same for style and regularization rewards. The policy update uses a weighted combination, treating the task-related advantage as twice as important as the auxiliary advantage.

Why this form: The decomposition prevents "negative interference" — a phenomenon where the value function's gradient signal for one reward component contradicts the signal for another. For example, approaching the ball quickly might generate high task rewards but also high base acceleration penalties. A single value function must average these conflicting signals, producing a noisy advantage estimate that fails to clearly indicate whether an action was good or bad. Separate critics allow each to specialize in its reward type, producing cleaner gradient signals. The 2:1 weighting prioritizes task completion over style and regularization — the robot should score goals even if it means slightly less human-like motion or slightly higher joint accelerations. Without this decomposition, the authors show (Figure 4A) that training curves for overall success rate are significantly worse, suggesting that the single-critic approach indeed fails to learn effectively under conflicting reward signals.

Why not more critics: The paper decomposes into only two groups rather than per-reward critics. The rationale (implicit) is that rewards within each group have similar temporal structure and scaling: all task rewards are sparse-to-dense and goal-directed; all auxiliary rewards are dense and regularization-focused. Further decomposition would increase computational overhead (more critic networks to train) without clear benefit.

---

PPO Training Details

The policy is trained using standard PPO [49] with the hyperparameters listed in Appendix A, Table 1:

• Number of learning epochs: 5 (the number of passes over the collected batch of experience before collecting new data).
• Mini-batch size: 4 (the batch size for stochastic gradient descent within each epoch).
• Learning rate: Adaptive (controlled by the KL divergence constraint — if the policy update exceeds a target KL divergence, the learning rate is reduced).
• Discount factor $\gamma$: 0.995 — this means a reward 100 steps in the future is discounted by $0.995^{100} \approx 0.606$, giving significant weight to medium-horizon outcomes while still emphasizing near-term rewards.
• GAE lambda $\lambda$: 0.95 — the Generalized Advantage Estimation parameter controlling the bias-variance tradeoff in advantage estimation. Lambda close to 1 gives lower bias but higher variance; 0.95 is a standard choice.
• Desired KL divergence: 0.01 — the target KL divergence between the old and new policy. PPO constrains policy updates to stay within this trust region.
• Optimizer: Adam.
• Entropy coefficient: 0.01 — a small bonus for policy entropy, encouraging exploration by penalizing over-confident action distributions.
• Reconstruction coefficient: 1 — weight of the decoder's MSE loss relative to the PPO objective.
• Discriminator coefficient: 1 — weight of the discriminator loss in its own training.
• Gradient penalty coefficient: 50 — weight of the Lipschitz regularization for the discriminator.
• Symmetry coefficient: 10 — weight of the mirror symmetry loss.

Why these values: These are largely standard PPO hyperparameters tuned for humanoid locomotion tasks. The discount factor of 0.995 (rather than the more common 0.99) reflects the long horizon of soccer tasks — a single kicking episode can last 60 seconds × 50 Hz = 3000 steps, and $0.995^{3000} \approx 0.0003$, meaning even the final goal-scoring reward receives non-negligible weight. The adaptive learning rate with KL constraint is a standard PPO feature that prevents destructive policy updates. The entropy coefficient of 0.01 is relatively small, appropriate because the reference motion initialization and AMP discriminator already provide sufficient exploration guidance.

Training scale and duration. With 16,384 parallel environments, 20,000 epochs, and 5 learning epochs per collection, the total number of environment steps is:

$16{,}384 \times (20{,}000 / 5) \times (\text{steps per collection})$

The exact steps per collection is not specified, but assuming standard PPO practice of collecting a fixed number of steps (e.g., 2048 steps per environment per collection), the total experience is on the order of hundreds of millions to billions of timesteps. Training takes 1 day on 8 V100 GPUs, which represents roughly 192 GPU-hours of computation — relatively efficient for a humanoid locomotion task of this complexity.

---

Summary of Key Design Decisions

4. Key Insights and Innovations

Innovation 1: Reframing Sim-to-Real Transfer as Perceptual Characteristic Matching, Not Visual Appearance Matching

The paper's most conceptually distinctive contribution is its rejection of the dominant paradigm for vision-based sim-to-real transfer in favor of a fundamentally different approach. Prior work in visually grounded RL for robotics — epitomized by Tirumala et al. (2024) [14], which used NeRF-based rendering to make simulated camera images look photorealistic — operated on the assumption that closing the visual appearance gap was the path to transfer. Train on images that look like real images, and the policy will generalize. This paper argues, implicitly through its design choices and explicitly through its results, that this assumption is not just insufficient but misdirected: the policy doesn't need to see the same pixels; it needs to experience the same statistical characteristics of imperfect perception.

The virtual perception system operationalizes this reframing. Rather than rendering synthetic RGB images and hoping the policy learns visual invariance to lighting, texture, and background changes, the authors abstract away visual appearance entirely and instead model four statistical properties of the perception pipeline: positional noise (Gaussian with distance-proportional variance), intermittent detection (90% probability within 7 meters, decaying beyond), latency (mean 116ms, standard deviation 18ms), and update frequency jitter (mean 25.36 Hz with ~1 Hz standard deviation). These are not visual properties — they are information-theoretic properties characterizing what the policy can know about the world and when it can know it. The policy never sees a single RGB pixel during training, yet transfers zero-shot to real hardware across diverse visual environments (grass, slabstone, soil, asphalt, rubber — Figure 2, panels J through L) because it learned robustness not to appearances but to perceptual degradation patterns.

This shift matters beyond soccer. It suggests a more general principle: for tasks where structured perception outputs (detections, segmentations, depth estimates) are available from separately trained models, closing the sim-to-real gap reduces to matching the statistical signature of the perception frontend rather than the visual signature of the environment. This is both more computationally efficient (no NeRF training, no photorealistic rendering at scale) and more generalizable (a perception model trained once on diverse data transfers its robustness; the policy inherits it). The paper doesn't name this principle explicitly, but the evidence for it is strong: Figure 3A shows hardware kicking success rates that closely match simulation predictions across field positions, achieved with zero real-world training data and zero environment-specific tuning. This is what successful sim-to-real transfer looks like, and it was achieved not by making simulation more realistic but by making it less visually specific and more perceptually characteristic.

The distinction from prior work is sharp. Haarnoja et al. (2024) [12] avoided the perception problem entirely by training and evaluating in simulation with privileged state. Tirumala et al. (2024) [14] attempted to solve it through photorealism and achieved only partial reactivity. This paper solves it through abstraction — recognizing that a YOLOv8 bounding box in simulation noise can be a better proxy for a YOLOv8 bounding box on hardware than a NeRF-rendered image ever could be, because the statistical relationship between the observation and the true state is what the policy actually needs to learn, not the mapping from pixel patterns to actions.

---

Innovation 2: Demonstrating That AMP Extends to Perceptually Grounded Dynamic Control, Enabling Simultaneous Imitation and Task-Driven Adaptation

Prior to this work, Adversarial Motion Priors [33] had been applied exclusively to proprioceptive imitation in static environments — characters learning to walk, run, or dance where the only observations were the agent's own joint states and the only objective was to move in a stylized way. The implicit assumption in the AMP literature was that the framework worked because the discriminator needed only to assess kinematic plausibility — does this sequence of joint angles look like something a human would produce? Adding exteroceptive perception was not obviously compatible with the AMP formulation because the discriminator operates on state transitions $(s_t, s_{t+1})$, and it was unclear what would happen when those transitions were influenced by noisy visual inputs that could cause abrupt, non-kinematic behavioral shifts (e.g., suddenly changing direction because the ball moved, not because the gait naturally evolved).

This paper demonstrates that AMP works in this extended regime, and more importantly, that the discriminator and the task rewards do not conflict in the way one might fear. The discriminator provides an implicit motion manifold — a soft constraint that the policy's movements should be statistically similar to human walking and kicking patterns — while the task rewards (ball approach, goal progress) pull the policy toward soccer-specific behaviors that have no direct analog in the reference data. The result, visualized in Figure 5 through UMAP projection of 20,000 joint-space trajectory frames, is five distinct behavioral clusters (walking, turning left, turning right, left-foot kicking, right-foot kicking) that both cover the reference dataset distribution and extend beyond it. The policy synthesizes novel behaviors — most notably the pivot hook kick described in Section 2.5, where the robot pivots on its supporting foot and swings the kicking leg laterally to hook the ball backward — that were never demonstrated in the reference motions but are accepted by the discriminator because they lie within the learned manifold of plausible humanoid motion.

This is fundamentally different from what feature-based imitation methods (DeepMimic [27] and its derivatives [20–25]) achieve. Those methods enforce temporal alignment to reference trajectories: at time $t$, the policy's joint angles should match frame $t$ of a reference clip. This works for pre-choreographed behaviors (dancing, acrobatics) where the timing is fixed, but it is fundamentally incompatible with reactive tasks where the timing of actions depends on external events — you cannot pre-specify when the kicking leg should extend because that depends on when the robot reaches the ball, which depends on the ball's trajectory and the terrain. The AMP discriminator avoids this by evaluating only local transitions (two consecutive timesteps) rather than global alignment, making it agnostic to when a behavior occurs and enabling the policy to interleave, transition between, and adapt behaviors to task demands.

The comparison in Figure 6 makes the practical significance concrete. A rule-based strategy (representative of the RoboCup state-of-the-art) takes approximately 5 seconds to kick a ball located behind the robot because it must sequentially rotate, approach, align, and execute — each phase as a separate primitive. The learned policy achieves consistently shorter times across all approach angles with minimal variation, because the approach and the kick are not separate phases but a continuous motion where foot placement during the final steps of the approach directly sets up the kicking geometry. This seamless transition from locomotion to manipulation — which the paper terms "coherent behaviors" — is precisely what modular systems cannot achieve and what AMP's implicit style guidance enables without explicit sequencing logic.

The insight is not that AMP is a better algorithm than feature-based imitation (that was known), but that the flexibility AMP provides — freedom from temporal alignment, freedom from behavioral segmentation — is not merely a convenience but a necessity for perceptually grounded dynamic control. When the timing of actions is externally driven, the imitation framework must be temporally agnostic; AMP provides this property naturally through its adversarial formulation, and this paper is the first to demonstrate that this property survives the introduction of noisy exteroceptive observations and task-driven behavioral deviations.

---

Innovation 3: The Encoder-Decoder with Reconstruction Loss as a Mechanism for Learning Perceptual Noise Filters Without Explicit State Estimation

The use of encoder-decoder architectures with auxiliary reconstruction objectives is well-established in the POMDP literature [50, 51, 52]. The standard formulation: an encoder compresses observation history into a latent representation; a decoder reconstructs privileged state variables (typically base velocity, terrain geometry) from this latent; the reconstruction loss forces the latent to retain information about the true world state that the policy can then exploit. This paper's contribution is not the architecture itself but the empirical demonstration that this mechanism functions specifically as a learned perceptual noise filter — and the counterfactual ablation that proves the mechanism is essential.

The quantitative evidence is in Section 2.4 and Figure 4B: the policy's internal ball position estimate (decoded from the latent representation) achieves a root-mean-square error of 0.186 meters over the last second before kicking, compared to 0.344 meters for the raw visual perception. This is a 46% reduction in estimation error, and it is critical because the robot's foot arch is only 0.23 meters long — a 0.34-meter error would frequently result in the robot kicking air or striking the ball with the toe rather than the arch, dramatically reducing accuracy. The ablation result is equally important:

"If the decoder was removed during policy training and trained separately afterward to estimate the ball position from latent states, the resulting predictions remained at the noise level."

This reveals something non-obvious: the encoder does not automatically learn to filter noise just because it sees noisy inputs. If the decoder is trained post-hoc on a frozen encoder, the latent space contains insufficient information to reconstruct the true ball position — the encoder has learned features that are sufficient for action selection (the policy still works, as evidenced by the overall system performance) but not sufficient for explicit state estimation. It is only the joint training of policy and decoder that creates a latent space rich enough to support both functions simultaneously.

The significance of this finding extends beyond the specific architecture. It suggests that in POMDP settings where the policy must act under perceptual uncertainty, the auxiliary reconstruction objective is not merely helpful but constitutive of the latent representation's properties. Without it, the encoder learns a "good enough" representation for the immediate action selection problem — which may amount to reacting to raw noisy observations with high-frequency corrections rather than maintaining a filtered state estimate. With it, the encoder is forced to develop the temporal integration and noise rejection capabilities that enable the 46% error reduction. This is a subtle point that is easy to miss: the reconstruction loss does not just improve performance; it qualitatively changes what the latent representation encodes, from a reactive mapping to a world model.

The emergent ball-search behaviors (Figure 3B and C) provide further evidence for this qualitative shift. When the ball is not detected, the policy does not simply freeze or wander randomly — it executes structured search patterns (moving to the center of the field for a wide visual sweep, then rotating to cover all angles) that align with its internal ball position estimates. The decoder's predictions during search point toward regions of the field that have not yet been explored, guiding the robot's search. This is behavior that emerges from a latent space that encodes not just "where is the ball?" but "where is the ball most likely to be given what I have and haven't seen?" — a richer representational capacity than what a purely reactive policy would develop.

---

Innovation 4: Multi-Critic Reward Decomposition as a Necessary Stabilizer for Training Under Heterogeneous Objectives

The observation that combining heterogeneous reward components into a single scalar causes training instability is not new — it has been documented in prior multi-objective RL work [54]. What this paper contributes is a specific diagnosis of the failure mode in the context of combined task, style, and regularization objectives, and a minimal intervention (two critics rather than one, with a specific grouping of rewards) that resolves it. This is an incremental technical contribution but a practically significant one, because the failure it addresses would have prevented the overall system from working.

Figure 4A provides the key evidence: training curves for overall success rate under the single-critic baseline are substantially lower than under the multi-critic approach. The authors attribute this to "negative interference between distinct reward components" — a phenomenon where the value function's gradient for one reward component (e.g., ball approach, which encourages rapid locomotion) contradicts the gradient for another (e.g., base acceleration penalty, which penalizes rapid locomotion). A single value function must average these conflicting signals, producing a noisy advantage estimate that fails to clearly indicate whether an action improved or worsened the agent's situation. The result is slower learning and, potentially, convergence to a suboptimal compromise policy that partially satisfies all objectives but fully satisfies none.

The specific decomposition — goal-related rewards in one critic, all auxiliary rewards (style, regularization, survival) in another, with a 2:1 weighting favoring the goal-related critic — is not obviously the only decomposition that would work. One could imagine per-reward critics or alternative groupings. But the paper's contribution is the demonstration that some decomposition is necessary, and that the natural split between "task progress" and "everything else" is sufficient. The 2:1 weighting prioritizes task completion over style and smoothness, which is appropriate: a robot that scores goals with slightly jerky motions is preferable to one that moves beautifully but never kicks.

The broader implication is methodological. As RL systems incorporate increasingly diverse reward components — task success, imitation objectives, safety constraints, energy efficiency, smoothness — the assumption that they can be naively summed into a single scalar reward becomes increasingly untenable. This paper provides a template for handling this diversity: identify natural groupings of rewards that share temporal structure and scale, assign separate critics to each group, and combine their advantages with task-appropriate weights. The fact that this intervention was necessary even for a relatively modest set of ~20 reward terms (Appendix B, Table 3) suggests that multi-critic architectures should be the default, not the exception, for complex humanoid control tasks.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation is conducted on the RoboCup Adult-size Humanoid League soccer field, a real-world physical environment measuring 14 m in length and 9 m in width with goals 2.6 m wide (Section 4.1). For simulation evaluation (Figure 3A), the field is replicated in NVIDIA Isaac Gym with the same dimensions. Hardware evaluation is performed on the real field across diverse surface types including grass, slabstone, soil, asphalt, and rubber (Figure 2J–L). Testing spans multiple ball positions and environmental conditions to assess generalization, not a fixed held-out dataset in the traditional ML sense.

• Base model. All experiments use the Booster T1 humanoid robot, measuring approximately 1.2 m in height and 30 kg in mass, equipped with an Intel RealSense Depth Camera D435i mounted on a two-degree-of-freedom head (yaw and pitch) and onboard computation via a Jetson AGX Orin (Section 2.1). No hardware modifications were made from the standard platform. The policy is a multilayer perceptron with encoder-decoder architecture trained using PPO (architecture details in Section 4.2 and Appendix A, Table 1). The robot was chosen because it is "representative of the capabilities of many contemporary humanoid platforms" and has been used for locomotion control, loco-manipulation, and RL algorithm development in prior work [19, 40–46].

• Metrics. The primary quantitative metrics are:

  1. Kicking success rate (%): Defined in Section 2.3 as the fraction of trials where the robot successfully kicks the ball into the goal. A trial is considered failed if the robot falls or the ball goes out of bounds. If the ball remains in play, the robot is permitted additional kick attempts. Consecutive trials are conducted per ball position, and success rates are reported per field region in Figure 3A.
  2. Kicking time (s): Defined in Section 2.5 as "the interval between the onset of movement and the moment the ball was kicked out." Measured from a stationary start at 1.5 m distance from the ball, across various approach angles (Figure 6A).
  3. Maximum angular velocity (rad/s or °/s): Measured during the approach-to-kick process as a proxy for agility (Section 2.5, Figure 6B), "reflecting the presence of agile behavior."
  4. Ball position estimation RMSE (m): Computed in Section 2.4 (Figure 4B) as the root-mean-square error between the policy's decoded ball position estimate and the ground-truth ball position over the last 1 second before kicking, compared against the raw visual perception error.
  5. Ball perception proportion (%): The fraction of timesteps during kicking tests where the ball is successfully detected within the camera's field of view (Section 2.4, Figure 4B).
  6. Perceptual error (m): The average Euclidean distance between the visually detected ball position and the ground-truth ball position (Section 2.4, Figure 4B).

• Baselines. The paper compares against two primary baselines:

  1. Rule-based soccer strategy: The approach employed by the runner-up team in the RoboCup Humanoid League (Section 2.5), which "leveraged the robot's built-in walking controller and relied on preprogrammed behavior trees to generate velocity commands for decision-making." This represents the current state-of-the-art in modular, hand-engineered robot soccer control.
  2. Single-critic training: An ablation of the multi-critic architecture where "a single critic to estimate combined rewards" is used instead of separate goal-related and auxiliary critics (Section 4.2, Figure 4A). This tests whether reward decomposition is necessary for training stability.

  The paper also implicitly compares against the prior vision-based RL approach of Tirumala et al. (2024) [14] through qualitative claims about reactivity, though no direct head-to-head quantitative comparison is reported. The prior unified privileged-state RL approach of Haarnoja et al. (2024) [12] is cited as conceptual predecessor but not directly compared, since it operates only in simulation with perfect state information.

• Generation budget / compute accounting. The paper does not use "generations" as a compute unit in the typical LLM sense. Instead, the key resource metrics are:

  1. Policy inference frequency: 50 Hz — the policy produces joint position commands every 20 ms.
  2. Camera update frequency: ~25 Hz — new visual detections arrive approximately every other policy step.
  3. Training scale: 16,384 parallel environments × 20,000 epochs on 8 NVIDIA V100 GPUs, taking approximately 1 day (Appendix A). This corresponds to hundreds of millions to billions of environment steps.
  4. Real-world evaluation: Success rates reported per ball position with 8,192 simulation trials (background grid in Figure 3A) and 10 consecutive hardware tests per position (dots in Figure 3A). The 10-trial hardware protocol per position reflects the practical constraints of physical robot experimentation.
  5. Kicking time comparison (Figure 6): 5 tests per approach direction, with the robot starting stationary from a 1.5 m distance.

• Cross-validation / statistical protocol. The paper does not employ traditional cross-validation in the machine learning sense, as the evaluation is on physical robot performance rather than a fixed held-out dataset. Instead, the statistical protocol involves:

  1. Large-scale simulation testing: 8,192 trials across field positions to establish statistical reliability of simulated success rates (Figure 3A background grid).
  2. Repeated hardware trials: 10 consecutive tests per ball position, with success/failure recorded per trial (Section 2.3).
  3. Error bars and standard deviations: Reported as shaded areas in Figure 6A–B (standard deviation across 5 tests per approach direction) and as explicit SD values in the perception modeling (Appendix E).
  4. Training curve averaging: Figure 4A reports overall success rates in disturbed training environments, presumably averaged across multiple random seeds, though the number of seeds is not explicitly stated.

  A notable absence is confidence intervals on the hardware success rates in Figure 3A. With only 10 trials per position, a 70% observed success rate has a 95% binomial confidence interval of approximately ±28 percentage points — substantial uncertainty that the paper does not discuss.

---

Main Quantitative Results

Kicking Success Rates Across the Soccer Field (Figure 3A)

The central quantitative result is the kicking success rate as a function of ball position on the field, evaluated both in simulation (8,192 trials) and on hardware (10 trials per position). The simulation trials are conducted with "external disturbances such as uneven terrain and physical perturbations to the ball or robot removed" but with the virtual perception system retained, allowing "comprehensive evaluation of the vision-driven pipeline, from perception to motor execution" (Section 2.3).

Simulation results (Figure 3A, background grid). The authors report:

"In regions close to the goal, success rates were particularly high, with failures occurring only rarely. Performance declined as the distance from the goal or angular offsets relative to the goal's normal direction increased. This reduction was primarily due to tighter tolerance for kicking angle accuracy, especially in backfield regions, where angular deviations exceeding ±10° resulted in missed shots."

Specific quantitative patterns visible in the grid (though exact per-region percentages are not tabulated in the text):

• Near the goal (< 3 m) and centered (±30° from goal normal), success rates appear to exceed 90% based on the color gradient.
• In the backfield (far half, especially corners), success rates drop substantially — the authors note that failures occur because "the robot struggled to select the appropriate kicking foot" when the ball is directly along the robot-goal line, and because "the limited time constrained accurate ball localization and gait adaptation" when the ball is very close to the robot.
• The lowest success rates occur in the far corners of the opponent's half, where angular deviation tolerances are tightest.

Hardware results (Figure 3A, dots). The authors report:

"Hardware experiment achieved success rates comparable to those in simulation, with the robot maintaining high performance across all tested positions, including the challenging backfield regions."

The dots on Figure 3A represent 10 consecutive trials per position. The authors note that "the robot remained stable throughout all trials, with no falls recorded, underscoring the robustness of the proposed policy." This is a significant result — zero falls across all hardware trials means the policy's safety guarantees (termination penalty, stagnation penalty) successfully transferred from simulation.

Interpretation of sim-to-real alignment. The close match between simulation and hardware success rates (the dots largely follow the background grid's color pattern) is the primary evidence for the virtual perception system's effectiveness. The paper claims this confirms:

"the virtual perception system effectively bridged the gap between simulation and the real world."

However, the comparison is qualitative — no correlation coefficient or per-region error between simulation and hardware rates is reported. The 10-trial hardware sample per position means the hardware success rate estimates have substantial uncertainty; whether they "closely match" simulation is visually suggestive but not statistically rigorous at the per-position level.

Perception-Action Coordination and Noise Filtering (Figure 4)

The paper reports several quantitative measures of how the policy processes and filters perceptual information:

Active perception (Figure 4C). The distribution of angular distance between the ball position and the camera center is evaluated over 1,000 steps across 2,048 environments. The shaded area indicates the camera's field of view. The authors report:

"this behavior enabled the policy to keep the ball within the camera's field of view for the majority of the time."

The implication is that the head alignment rewards (pitch and yaw penalties, Appendix B, Table 3) successfully trained the policy to actively track the ball. The paper notes that this behavior was "preserved even when explicit ball-tracking rewards were removed," though it does not show a quantitative comparison with and without these rewards — only a qualitative claim that "incorporating such rewards encouraged the robot to maintain the ball near the center of its FOV rather than at the periphery, thereby improving robustness in real-world deployment."

Noise filtering and state estimation (Figure 4B). Across 4,096 kicking tests, the paper reports three key metrics:

1. Policy's ball position estimation RMSE: 0.186 m (last 1 second before kicking).
2. Raw perceptual error: 0.344 m (average distance between detected and true ball position).
3. Improvement: 46% reduction in estimation error.

The absolute magnitude of 0.186 m is contextualized against the robot's foot arch length of 0.23 m:

"Considering that the length of the robot's arch is only 0.23 m, such an improvement is essential for accurate ball positioning and reliable strikes."

This means that without the policy's noise filtering, the typical perceptual error (0.344 m) significantly exceeds the contact surface size (0.23 m), meaning the robot would frequently misjudge where to place its foot relative to the ball. With filtering, the error (0.186 m) falls within the arch length, enabling reliable contact.

Perceptual error breakdown (Figure 4B). The proportion of ball perception and average perception error across the 4,096 tests are reported, though exact values are not quoted in the text beyond the RMSE numbers above. The figure presumably shows that the policy's internal estimates have lower variance and fewer outliers than the raw detections, consistent with the temporal integration enabled by the 1-second encoder window.

Decoder ablation (Section 2.4). The critical counterfactual:

"If the decoder was removed during policy training and trained separately afterward to estimate the ball position from latent states, the resulting predictions remained at the noise level."

This demonstrates that the reconstruction loss must be applied jointly during policy training — the latent space does not automatically capture physically meaningful state estimates without this auxiliary objective. The quantitative severity of this degradation is not reported (what exactly is "at the noise level"? — presumably RMSE ≈ 0.344 m or worse), but the qualitative claim is that the benefit disappears entirely when training is decoupled.

Ball search behavior (Figure 3B and C). The policy exhibits structured search patterns that correlate with its internal ball position estimates:

• When near field edges: robot reorients toward center, enabling wide visual sweep, then moves toward central region.
• At center: robot switches to rotational scanning, covering all spatial zones.
• During movement toward center: decoder predictions "often pointed to distant outer areas, guiding the robot to expand its search range."
• During rotation: predictions "shifted to rearward regions not yet surveyed."

This is not presented as a quantitative metric but as qualitative evidence that the latent representation encodes spatial awareness of unexplored regions, not just filtered estimates of the last known ball position.

Comparison Against Rule-Based Strategy (Figure 6)

The quantitative comparison against the rule-based baseline (representing the RoboCup state-of-the-art) is measured along two axes: time to kick and maximum angular velocity during approach.

Kicking time (Figure 6A). Ball positioned at field center; robot initialized on a 1.5 m radius circle centered on the ball, at various approach angles (0° = facing the goal, 180° = facing away from the goal). Reported results:

• Rule-based strategy at 0° (facing goal): approximately 2 seconds.
• Rule-based strategy at 180° (facing backward): approximately 5 seconds.
• Reason for the 2.5× increase: "This delay resulted from the need for extensive rotation and fine positional adjustments around the ball to achieve proper alignment."
• Learned policy: "consistently achieved shorter times across all orientations, with only minor variation."

The exact values for the learned policy are not quoted in the text, but Figure 6A shows the learned policy curve (orange/red) consistently below the rule-based curve (blue) across all approach angles. The "minor variation" suggests the learned policy's kicking time is largely independent of initial orientation — a direct consequence of the seamless perception-action coupling that eliminates the sequential rotate-then-approach-then-align-then-kick pipeline.

Maximum angular velocity (Figure 6B). Measured during the approach-to-kick process as a proxy for agility:

"The learned policy attained higher turning speeds across all tested orientations, underscoring its greater agility in aligning with the target direction."

The authors qualify this metric: "While not a complete measure of agility, this metric reflected the presence of agile behavior." Exact angular velocity values are not quoted in the text, but Figure 6B shows the learned policy curve consistently above the rule-based curve.

Behavioral comparison (Figure 6C–F). The visual comparison in Figure 6 shows representative trajectories:

• Rule-based (C–D): The robot walks to the ball, stops, rotates to align, then executes a discrete kick. For backward kicks, this involves nearly a full 180° rotation before approaching.
• Learned policy (E–F): The robot flows continuously from approach into kick, with foot placement during the final approach steps directly setting up the kicking geometry. There is no visible pause between locomotion and manipulation.

The paper attributes this to:

"its capacity for dynamically adjusting foot placement, enabling a seamless transition from approaching the ball to executing the kick."

Limitations of this comparison. The rule-based strategy is not an RL baseline — it's a hand-engineered behavior tree. A stronger comparison would be against a modular RL approach (separate locomotion and kicking policies with a state machine) or against the Tirumala et al. (2024) vision-based RL controller [14]. The paper does not report any RL-vs-RL comparison, making it impossible to disentangle whether the improvements come from RL itself (vs. hand-engineering) or from the specific architectural contributions (encoder-decoder, AMP, virtual perception).

Training Stability: Single-Critic vs. Multi-Critic (Figure 4A)

The paper reports training curves comparing the multi-critic architecture against a single-critic baseline:

"using a single critic to estimate combined rewards can cause negative interference between distinct reward components, leading to reduced learning stability and performance (Fig. 4A)"

Figure 4A shows "overall success rates in disturbed training environments" over the course of training. The multi-critic approach (presumably the solid/orange curve) achieves higher asymptotic success rates and faster initial learning compared to the single-critic baseline (presumably the dashed/blue curve). Exact final success rates are not quoted in the text for either condition.

The specific decomposition — goal-related rewards in one critic, auxiliary (style + regularization) in another, with a 2:1 weighting — is claimed as the mechanism that:

"effectively reduces interference between reward components, enhancing the robustness of the learning process."

No ablation is reported for alternative decompositions (e.g., three critics, different groupings, different weights), so the claim is that some decomposition is necessary, not that this specific decomposition is optimal.

Gait Behavior Analysis (Figure 5)

The authors apply UMAP dimensionality reduction to 20,000 joint-space trajectory frames collected from the policy, projecting them into a 2D space for visualization. They report:

• 5 distinct gait clusters corresponding to walking, turning left, turning right, left-foot kicking, and right-foot kicking.
• These clusters demonstrate that "our controller produced a versatile set of behaviors within a single policy and transitioned smoothly among them."
• The reference motion dataset, projected into the same UMAP space, shows that "the policy trajectories broadly covered the dataset, indicating successful integration of demonstrated motions, while also extending beyond them to synthesize task-specific behaviors."

The specific example of synthesis beyond the reference data is the pivot hook kick:

"when the goal is behind the robot... the robot pivoted on the supporting foot and swung the kicking leg laterally to hook the ball, eliminating the need for a full body [turn] and thereby reducing execution time."

This is qualitative evidence that AMP provides generalization rather than mere mimicry — the discriminator accepts this novel motion because it lies within the learned manifold of plausible humanoid movements, even though no exact example existed in the 106-second reference dataset.

Quantitative gait adaptation (Figure 3D and E). The policy exhibits adaptive gait patterns as a function of ball proximity:

• Far from ball: "slower step frequency, allowing the robot to maintain consistent visual tracking of the ball while conserving energy."
• Near ball: "notable increase in step frequency... shorter and more rapid strides that facilitated accurate adjustment of foot placement in the vicinity of the ball."

This is visible in the foot contact patterns shown in Figure 3D and E (forward kick and backward kick examples), where the stride length and timing visibly change as the robot approaches the ball. No explicit metric (e.g., steps per second, stride length in meters) is reported to quantify this adaptation.

Chasing a rolling ball (Figure 7). The policy demonstrates two distinct reactive patterns:

• Lateral rolling (Figure 7A): "rapid lateral steps in an attempt to intercept the ball's trajectory before it moved out of range."
• Rolling toward rear (Figure 7B): "an extremely rapid rotational maneuver where one foot pushed off the ground, while the other foot pivoted on the surface, enabling a full reorientation in as few as two steps."

The foot contact pattern plots in Figure 7 confirm the temporal sequence of these behaviors — the alternating stance/swing phases visible in the gait diagrams show the acceleration and deceleration patterns corresponding to lateral and rotational movements.

---

Ablation Studies and Robustness Checks

Decoder removal during policy training (Section 2.4): When the decoder is removed from joint training and instead trained post-hoc on the frozen encoder's latent representations, the decoded ball position estimates remain at the noise level — meaning the ~46% RMSE reduction from 0.344 m to 0.186 m disappears entirely. This is the most critical ablation in the paper because it demonstrates that the encoder-decoder architecture does not automatically produce useful state representations; the joint training with reconstruction loss is constitutive, not merely additive. The quantitative degradation is not reported as an exact RMSE value, only the qualitative claim of "at the noise level," which limits precise assessment of the ablation's magnitude.

Head tracking reward removal (Section 2.4): The active perceptual behavior — keeping the ball within the camera's FOV — is "preserved even when explicit ball-tracking rewards were removed." However, "incorporating such rewards encouraged the robot to maintain the ball near the center of its FOV rather than at the periphery, thereby improving robustness in real-world deployment." This suggests the perceptual behavior is partially emergent from the task structure (the robot needs to see the ball to approach it) but that explicit head alignment rewards shift the behavior from "keep it barely visible" to "keep it centered," which improves robustness against detection failures at FOV edges. No quantitative comparison (e.g., ball-in-FOV percentage with vs. without tracking rewards) is reported.

ReST-EM revision model training (Appendix K, Figure 16, discussed in prior sections): Not directly relevant to the main experimental results but worth noting as the paper reports a negative result: attempting to further optimize the revision model using ReST-EM [Singh et al., 2024] caused "additional sequential revisions [to] substantially hurt performance," with fully sequential performance dropping to approximately 33.5% compared to roughly 38.5% at the optimal ratio. This is presented in the prior sections, not as a main experiment, but as evidence of the sensitivity of training methodology.

Virtual perception system design choices (implicit ablations): The paper does not report ablations of individual components of the virtual perception system — e.g., training with noise only (no latency), with latency only (no noise), with fixed detection probability vs. distance-dependent, or with different noise models. The system is presented as a monolithic component, and its effectiveness is validated only by the aggregate sim-to-real transfer success (Figure 3A), not by component-wise removal experiments. This is a significant gap: we cannot determine whether all four modeled factors (noise, detection probability, latency, update frequency) are individually necessary, or whether a subset would suffice.

AMP vs. feature-based imitation (no direct comparison): The paper argues for AMP's advantages over feature-based methods (temporal flexibility, no behavioral segmentation needed), but does not report a head-to-head comparison training the same soccer policy with a DeepMimic-style tracking reward [27]. Such a comparison would directly test the claim that AMP's temporal agnosticism is necessary for reactive soccer — but it is absent. The reader must infer the superiority of AMP from the overall system performance, not from a controlled ablation.

Observation history length (no ablation): The 1-second (50-frame) encoder window is presented as a design choice without ablation. Would 0.5 seconds (25 frames) perform comparably? Would 2 seconds (100 frames) improve ball velocity estimation further? The paper provides no empirical evidence for the chosen window duration.

Multi-critic weight ratio (no sweep): The 2:1 weighting ratio favoring the goal-related critic over the auxiliary critic is stated without ablations. Would 1:1 work? Would 5:1 be better? The paper provides no sensitivity analysis for this hyperparameter.

Gradient penalty coefficient (no sweep): The gradient penalty coefficient of 50 for the Wasserstein GAN is specified without ablation. This is known to be a sensitive hyperparameter in WGAN training — too low and Lipschitz constraint is violated, too high and discriminator becomes too constrained. The paper provides no evidence that 50 is near-optimal for this domain.

Mirror symmetry loss weight (no sweep): The symmetry coefficient of 10 is stated without ablation. Does the policy learn bilateral kicking without it? What fraction of kicks use the left vs. right foot with and without this loss? The paper claims it "prevent[s] convergence to a unilateral kicking strategy" but provides no quantitative comparison.

Simulation disturbance ablation (implicit): The paper mentions that "external disturbances such as uneven terrain and physical perturbations to the ball or robot were removed" during the formal success rate evaluation (Figure 3A simulation grid). However, training included these disturbances (Section 4.1: small uneven terrain, ball velocity injections, robot force perturbations). The contribution of disturbance-injected training to final performance is not ablated — would a policy trained without disturbances achieve similar success rates in clean evaluation? Would it transfer to real hardware? The missing ablation is significant because disturbance injection is computationally expensive (16,384 parallel environments must all simulate physics with randomized parameters).

---

Critical Assessment

Claim 1: The controller achieves "reactive soccer skills" with "tight integration of visual perception and motion control"

What the experiments demonstrate: The policy indeed produces coherent, vision-driven soccer behaviors — approaching, tracking, and kicking a ball — without the behavioral fragmentation characteristic of modular systems. The evidence includes:

• Qualitative videos (movie referenced in Section 2.2) showing continuous motion.
• The comparison against rule-based strategies (Figure 6) showing 2–3× faster kicking times and seamless approach-to-kick transitions.
• The foot contact pattern analyses (Figures 3D–E, 7) showing adaptive gait adjustments during ball approach.
• The UMAP visualization (Figure 5) showing smooth transitions between behavioral clusters within a single policy.

What is not demonstrated: The paper does not provide a quantitative measure of "reactivity" — for example, the latency between a change in ball position/velocity and a measurable change in the robot's motor output. The 50 Hz policy rate and ~116 ms perceptual latency establish an upper bound on reactivity (approximately 136–156 ms from visual event to motor response), but the actual closed-loop latency of the integrated system is not measured. A comparison against the rule-based strategy's closed-loop latency would directly quantify the claimed "reactivity" advantage.

Additionally, the paper does not demonstrate reactivity to unexpected ball motion perturbations — the experiments in Figure 7 show the robot chasing a rolling ball, but this is the ball's natural dynamics, not a sudden, externally-imposed change in trajectory (e.g., the ball deflecting off an obstacle). The disturbance injection during training (Section 4.1: "the ball is randomly subjected to an additional velocity or teleported to a new position") should have prepared the policy for such events, but no test-time evaluation of disturbance rejection is reported.

Conditional strength: The claim of tight perception-action coupling is strongly supported by the behavioral evidence. The lack of quantitative latency measurements is a gap, but the qualitative comparison against the rule-based baseline (Figure 6C–F) provides compelling visual evidence of the difference between coupled and decoupled control.

Claim 2: The virtual perception system "effectively bridged the gap between simulation and the real world," enabling "zero-shot transfer"

What the experiments demonstrate: The hardware success rates (Figure 3A dots) visually align with simulation predictions (background grid), and the robot operated across diverse terrains and visual conditions without environment-specific tuning. The 76-goal, 11-conceded RoboCup performance provides strong real-world validation under competitive constraints.

What is not demonstrated: The paper does not report what happens when the virtual perception system is removed — i.e., training with ground-truth simulation state and deploying to hardware. This is the critical ablation that would directly test the claim. We know from general sim-to-real principles (and the authors' motivation in Section 1) that training on privileged state fails catastrophically, but the paper provides no empirical demonstration of this failure mode or, more importantly, of how much the virtual perception system improves over this baseline. Without this ablation, we cannot distinguish between "the virtual perception system helps somewhat" and "the virtual perception system is essential."

Furthermore, the virtual perception system is validated as a monolith — there are no ablations removing individual components (noise only, latency only, detection probability only). This means we cannot determine which perceptual characteristics are most important for sim-to-real transfer. Is distance-proportional noise the critical factor? Is the 116 ms latency essential, or would a different latency distribution work? Is the 90% within-range detection probability necessary, or would 80% suffice? These questions are practically important for practitioners wanting to apply the method to different perception pipelines.

The hardware evaluation uses only 10 trials per ball position. With a 95% binomial confidence interval of approximately ±28 percentage points at 70% success rate, the claim of "close match" between simulation and hardware is statistically underpowered at the per-position level. Aggregating across positions improves statistical reliability, but the paper does not report aggregate statistics.

Conditional strength: Supported with qualifications. The sim-to-real transfer demonstrably works — the robot plays soccer on real hardware after simulation-only training, which is a significant achievement. But the mechanism by which it works is not isolated through controlled experiments. The virtual perception system is a validated engineering solution but not a scientifically validated necessary component, because the counterfactual (training without it) is not evaluated.

Claim 3: AMP "can be effectively extended beyond proprioceptive imitation to real-world dynamic environments involving visual feedback and perception-action coordination"

What the experiments demonstrate: The policy exhibits human-like motion patterns (walking, turning, arch-based kicking) that emerged from AMP training, as visualized in the UMAP projection (Figure 5). The motion clusters cover the reference dataset and extend beyond it (pivot hook kick). The robot's locomotion is stable and stylistically consistent with the reference data.

What is not demonstrated: The paper does not compare AMP against alternative imitation learning approaches (feature-based tracking, behavioral cloning, supervised distillation) for the same soccer task. The claim that AMP is effective is supported; the claim that AMP is superior to alternatives is not tested. A comparison against a DeepMimic-style baseline [27] with the same reference dataset would directly test whether AMP's temporal flexibility provides concrete benefits over explicit motion tracking for this reactive task. The paper's argument that feature-based methods are "less adaptable when deviations from reference trajectories were needed" (Section 1) is a conceptual claim, not an empirical one — no experiment shows a feature-based method failing where AMP succeeds on the same soccer task.

The AMP contribution is also difficult to isolate from the overall system. Would the policy learn to kick and walk without AMP, relying only on the task and regularization rewards? The paper does not report an ablation removing the AMP discriminator entirely. It is possible that the task rewards (ball approach, goal progress, sideways kick) alone would produce functional but less human-like locomotion — or that they would fail entirely without the implicit guidance the discriminator provides. Without this ablation, the necessary contribution of AMP to the system's success is unclear.

Conditional strength: Supported with qualifications for effectiveness, unsupported for necessity or superiority over alternatives. The paper demonstrates that AMP can be part of a working vision-driven soccer system; it does not demonstrate that AMP is required for such a system to work.

Claim 4: The encoder-decoder architecture with reconstruction loss enables noise filtering and active perception

What the experiments demonstrate: Strong quantitative evidence. The 46% RMSE reduction (0.344 m → 0.186 m) is a substantial improvement, and the decoder ablation (Section 2.4) directly demonstrates that this improvement depends on joint training of the decoder. The emergent ball-search behaviors (Figure 3B–C) provide convergent qualitative evidence that the latent space encodes spatial awareness beyond simple filtering.

What is not demonstrated: The paper does not characterize what the latent representation encodes beyond ball position — for example, does it estimate ball velocity? Robot base velocity? Goal-relative positioning? The decoder is trained to reconstruct multiple privileged variables (Appendix A, Table 2: ball velocity, base linear velocity, ball friction, base height, mass randomization), but the reconstruction accuracy for these other variables is not reported. We know the decoder achieves 0.186 m RMSE for ball position; we do not know whether it achieves useful accuracy for ball velocity or base velocity, which would be additional evidence for the richness of the latent space.

The paper also does not compare the encoder-decoder approach against alternative state estimation methods. For instance, a Kalman filter operating on the raw detection outputs could potentially achieve similar noise reduction without learning. The improvement from 0.344 m to 0.186 m RMSE is a ~2× reduction — whether this is better or worse than a classical filter is unknown (and depends on the specific dynamics and noise characteristics). A comparison against a simple filtering baseline would contextualize the learned approach's value.

Conditional strength: Strongly supported for the specific claim that the encoder-decoder reduces estimation error and that joint training is necessary for this reduction. The mechanism is empirically validated. The comparison to classical alternatives and the full characterization of latent space content are missing but are secondary to the central claim.

Claim 5: Multi-critic architecture improves training stability

What the experiments demonstrate: Figure 4A shows higher success rates with multi-critic vs. single-critic training. The difference is visually clear in the training curves.

What is not demonstrated: The paper reports only one decomposition (goal-related vs. auxiliary, 2:1 weighting). We do not know whether this specific decomposition is optimal, whether a different grouping (e.g., separating style rewards from regularization rewards) would work better, or whether the improvement comes from the decomposition itself or from the increased representational capacity (two critics have twice the parameters of one critic). An ablation controlling for total critic parameters (e.g., single critic with doubled network width) would distinguish "decomposition matters" from "more capacity matters." The paper also does not report whether the single-critic baseline was tuned to convergence — it is possible that the single-critic approach would eventually reach similar performance with more training, though the authors' claim of "negative interference" suggests a fundamental optimization difficulty, not just slower learning.

The paper also does not report final performance metrics for the single-critic baseline in the real-world evaluation. We know it underperforms in simulation training curves; we do not know whether a single-critic policy deployed to hardware would achieve meaningfully worse success rates than the multi-critic policy, or whether the training curve difference is attenuated at convergence.

Conditional strength: Supported for improvement in training, with caveats about the mechanism. The claim that negative interference is the cause is plausible but not directly tested — interference would manifest as conflicting gradient directions in the value function update, which could be measured but are not reported. The ablation is sufficient to justify the design choice for this system, but insufficient to draw general conclusions about multi-critic architectures in RL.

Overall Assessment of Experimental Rigor

Strengths:

• The paper provides real-world validation at a scale unusual for humanoid robotics — competition deployment with 76 goals scored and only 11 conceded is a genuinely rigorous test.
• The sim-to-real transfer is validated across diverse environments (5 surface types, varied lighting) without environment-specific tuning, which speaks to the robustness of the approach.
• The comparison against a rule-based strategy (Figure 6) provides a concrete, quantitative demonstration of the advantages of learned over hand-engineered control.
• The decoder ablation (Section 2.4) is a clean, informative counterfactual that directly tests a key mechanistic claim.

Weaknesses:

• Missing baseline ablations: The paper does not compare against (1) training without virtual perception system, (2) training without AMP, (3) feature-based imitation learning, or (4) classical filtering for state estimation. These absent experiments make it impossible to determine which system components are necessary vs. merely present.
• Small hardware sample sizes: 10 trials per ball position is statistically underpowered for precision claims about success rates. Confidence intervals are not reported.
• Missing perceptual component ablations: The virtual perception system is validated monolithically; individual factors (noise, latency, detection probability, update frequency) are not ablated.
• No quantitative reactivity measurement: The central claim of "reactive" control is supported behaviorally but not measured in terms of closed-loop latency or response time.
• Single robot platform: All results are on the Booster T1. Transferability to other humanoid platforms with different kinematics, mass distribution, or perception hardware is unknown.
• Limited statistical reporting: Training curves (Figure 4A) do not specify the number of random seeds; error bars on hardware metrics (Figures 6A–B) are present but small-sample (5 trials); per-position success rates (Figure 3A) have no confidence intervals.

Experiments that would strengthen the paper:

1. Train a policy without the virtual perception system (using ground-truth state) and evaluate on hardware — this would calibrate the sim-to-real gap the virtual perception system bridges.
2. Train a policy without AMP (task rewards only) and compare motion quality and success rates — this would isolate AMP's contribution.
3. Train with a Kalman filter operating on raw detections, feeding filtered estimates to a reactive policy, as a learned-vs-classical state estimation comparison.
4. Measure closed-loop latency from visual event to motor response for both the learned policy and the rule-based baseline.
5. Test generalization to a second humanoid platform with the same policy (after retargeting) to assess platform-specific vs. platform-general aspects of the approach.
6. Ablate individual virtual perception components (noise only, latency only, detection failures only) to identify which perceptual characteristics are most critical for sim-to-real transfer.
7. Report per-position success rates with confidence intervals based on the 10-trial protocol, or increase the trial count to reduce uncertainty.

Bottom line: The experiments convincingly demonstrate that the complete system works — a humanoid robot trained in simulation can play soccer on real hardware using only onboard vision, with performance that surpasses a modular rule-based baseline and contributes to competition success. The experiments do not convincingly isolate why each component is necessary or quantify the relative contribution of each design choice. The paper is a strong engineering contribution with rigorous real-world validation; it is a weaker scientific contribution in terms of controlled experiments that test specific hypotheses about mechanisms. This is characteristic of real-world robotics research, where the integrated system is often more than the sum of its (individually unablated) parts, and the burden of proof shifts toward "does it work in practice?" rather than "which component contributed how much?" The RoboCup championship provides a compelling answer to the former question, even as the latter remains incompletely answered.

6. Limitations and Trade-offs

Limitation 1: No Multi-Agent Coordination or Opponent Awareness

The assumption or constraint. The controller is designed, trained, and evaluated exclusively for single-robot soccer skills — approaching and kicking a stationary or rolling ball toward an empty or human-defended goal. The policy's observation space is explicitly restricted to the ball position, ball detection mask, goal position, and goal direction (Appendix A, Table 2). There is no representation of other robots, opponents, or teammates in the policy's inputs. The authors acknowledge this directly in Section 3:

"the current controller focuses primarily on individual soccer skills, lacking mechanisms to respond to opponents and coordinate with teammates in scenarios involving multiple robots. The environmental information received by the policy is currently restricted to the positions of the ball and goal, with no integration of social or adversarial context."

The consequence. In multi-robot scenarios — which constitute the actual RoboCup competition format and any realistic team-sport deployment — the policy has no capacity to:

• Detect or avoid collisions with other robots (the current collision penalty in Appendix B, Table 3 penalizes body-ground contact but provides no signal about robot-robot interactions).
• Anticipate an opponent's approach to the ball and adjust timing or kicking direction accordingly.
• Execute team strategies such as passing to a teammate, defensive positioning, or dynamic role assignment.
• Respond to adversarial blocking — a goalkeeper physically obstructing the shot — beyond what the AMP style reward and recovery behaviors incidentally provide.

The training environment does include ball perturbations and robot force disturbances (Section 4.1) to simulate physical contact, but these are unstructured random interventions, not behaviors generated by adversarial agents with strategic intent. The policy learns to recover from pushes; it does not learn to play against an opponent who actively tries to take the ball.

What evidence exists in the paper. The RoboCup results (Section 2.2, 76 goals scored, 11 conceded) demonstrate that the policy can function in multi-robot matches — the robot scored goals and won games. However, the paper provides no analysis of how the policy interacts with opponents. Did goals result primarily from the robot's superior speed and agility (reaching the ball first and kicking before opponents could respond)? Did the 11 conceded goals occur because the policy failed to defend, failed to maintain possession, or failed to anticipate opponent actions? These questions are unanswered. The paper presents the controller as "adopted as a module by the Tsinghua Hephaestus team" (Section 2.2), suggesting that higher-level strategic logic (likely a separate planning module) handled team coordination and opponent response — but this module is not described, and its interaction with the reactive kicking policy is not analyzed.

Mitigation status. The limitation is explicitly acknowledged, and future work is outlined (Section 3):

"Future research will extend the training environment to include multiple robots and expand the policy's observation space to incorporate real-time sensory data of other agents. By integrating such multi-agent contextual information, the policy can be trained to enable dynamic collaboration and adversarial adaptation."

However, this is an ambition, not a partial solution. Extending from single-agent to multi-agent RL introduces substantial new challenges — credit assignment across agents, non-stationarity of the environment (other agents are learning simultaneously), and the need for opponent detection and tracking from onboard vision (which would require expanding the perception pipeline and virtual perception system to model multi-object detection characteristics). The current system provides no scaffolding for this extension; the observation space, reward structure, and training environment would require fundamental redesign, not incremental modification.

---

Limitation 2: The Virtual Perception System Is Validated Monolithically, Not Component-Wise

The assumption or constraint. The virtual perception system models four perceptual factors — positional noise, detection probability, latency, and update frequency — as a unified statistical module that transforms ground-truth simulation state into realistic observations (Section 4.3, Appendix E). The parameters of these models are fit from approximately 1 hour of data collected with the robot's default walking gait and a rule-based ball-tracking program. The system is validated only as an integrated whole: the overall sim-to-real transfer works (Figure 3A shows comparable simulation and hardware success rates), so the virtual perception system is deemed effective.

However, the paper provides no ablations that remove individual perceptual factors to determine their relative importance. We do not know:

• Would training with positional noise alone (without latency or detection dropouts) produce comparably successful transfer?
• Is the 116 ms mean latency critical, or would the policy adapt to different latency distributions?
• Does detection probability matter, or does the encoder's temporal integration handle detection dropouts even without explicit training on this failure mode?
• Is the linear noise model (noise standard deviation = 0.124d + 0.149) necessary, or would a constant-variance noise model suffice?

The consequence. This limitation has two practical implications. First, for practitioners wanting to apply this method to a different robot platform with different perception hardware, it is unclear which perception characteristics must be accurately modeled and which can be approximated or ignored. The data collection and modeling effort (Appendix E: fitting Gaussian distributions to noise, latency, frequency, and detection probability) is non-trivial — it requires a motion capture system and approximately 1 hour of data collection. If modeling only a subset of factors would suffice, the cost of adoption could be substantially reduced. Conversely, if some unmodeled factor (e.g., systematic bias in depth estimation at close range, correlation between detection failures and robot velocity, multi-path reflections in the camera's depth sensor) is important for different hardware, the current validation provides no guidance for identifying it.

Second, the virtual perception system was modeled using data collected with the robot's default walking gait — a fundamentally different locomotion pattern than the dynamic, tight-turning, rapid-striding behavior produced by the learned policy. The authors acknowledge this potential discrepancy:

"although the walking gait used for data collection differs from the learned policy, which introduces potential distribution discrepancies, the simplicity of our perceptual modeling ensures strong generalization across locomotion patterns" (Section 4.3)

This claim — that "simplicity ensures strong generalization" — is an assertion, not a demonstrated result. The perceptual noise characteristics during aggressive turning (where motion blur is severe, the ball moves rapidly across the camera's field of view, and the camera's rolling shutter may introduce geometric distortions) may differ systematically from the noise characteristics during steady walking. Without measuring perceptual errors under the learned policy's actual gait distributions and comparing them to the modeled distributions, we cannot verify the generalization claim.

What evidence exists in the paper. The evidence is entirely at the aggregate outcome level: hardware success rates match simulation predictions (Figure 3A). This tells us the total sim-to-real gap is small; it does not tell us which components of the virtual perception system contributed to closing it, or whether a simpler system would have sufficed. The missing baseline — training with ground-truth simulation state (no virtual perception system) and deploying to hardware — would quantify the total gap the system bridges, but is not reported. The missing component ablations would decompose this gap by factor, but are also absent.

Mitigation status. Not addressed. The paper presents the virtual perception system as a validated engineering solution without investigating its internal mechanisms. The perceptual modeling (Appendix E, Figure 9) provides histograms and fitted distributions, demonstrating that the models match the collected data, but this is a descriptive validation (the models fit the data they were fit to), not a causal validation (the models cause successful sim-to-real transfer). The mitigation would require either component-wise ablation experiments in simulation-to-hardware transfer or a sensitivity analysis showing how transfer performance degrades as individual perceptual factors are mis-specified.

---

Limitation 3: Limited Behavioral Repertoire — Only Approach-and-Kick, No Dribbling, Trapping, or Strategic Possession

The assumption or constraint. The reward function (Appendix B, Table 3) is designed for a single task: approach the ball (ball approach reward, weight 50) and propel it toward the goal (goal progress reward, weight 500). The terminal reward structure (Section 4.1) resets only the ball after a goal or out-of-bounds event, keeping the robot in place to continue kicking — this encourages sequences of kicks, but each kick is an independent approach-and-strike episode. The sideways kick reward (weight 20) and forward kick penalty (weight −20) bias the policy toward arch-based kicking, but provide no incentive for:

• Dribbling: maintaining controlled ball possession while moving, with repeated small touches rather than a single powerful kick.
• Ball trapping: receiving a moving ball and stopping it with the foot or body to gain control.
• Strategic passing: kicking the ball to a specific location (e.g., a teammate, an open space) rather than toward the goal.
• Context-sensitive shot selection: choosing between a powerful shot, a placed shot, or retaining possession based on the defensive situation.

The authors acknowledge this in Section 3:

"while the policy has demonstrated walking and adaptive kicking behaviors across varied ball and goal configurations, it still falls short of replicating the broader range of soccer skills exhibited in human matches, including dribbling, ball trapping, and strategic passing. This gap stems primarily from the policy's limited capacity to identify context-specific action requirements across different scenarios."

They further diagnose the root cause:

"The current training framework, which emphasizes goal-oriented outcomes, tends to drive the policy toward converging on a single movement pattern."

The consequence. The policy's behavioral repertoire, while versatile within the approach-and-kick paradigm (Figure 5 shows five gait clusters: walk, turn left, turn right, left kick, right kick), is fundamentally limited to one strategic intent: get to the ball and shoot. In competitive matches where the optimal action might be to retain possession, shield the ball from an opponent, or pass to a better-positioned teammate, the policy will always default to shooting — or, if shooting is not immediately possible, to approaching the ball until shooting becomes possible.

This limitation is partially masked by the RoboCup results (76 goals, 11 conceded) — in the Adult-size Humanoid League, the ability to reach the ball faster than opponents and shoot accurately may be sufficient to dominate, because opponent robots are similarly limited in their behavioral sophistication. Against more capable opponents, or in scenarios requiring strategic patience (e.g., protecting a lead by maintaining possession rather than taking low-percentage shots), the policy's single-strategy behavior would become exploitable. An opponent that learns "this robot always shoots immediately" can position defensively to block the most likely shooting lanes.

What evidence exists in the paper. The UMAP visualization (Figure 5) provides indirect evidence: the five clusters all correspond to locomotion and kicking primitives. There is no cluster for controlled dribbling (repeated small touches while moving), trapping (decelerating the ball), or defensive positioning. The foot contact patterns in Figure 7 show the robot chasing a rolling ball and kicking it; there is no demonstration of the robot slowing the ball, changing its direction without shooting, or maneuvering around an obstacle while maintaining possession. The video (movie referenced in Section 2.2) presumably shows continuous kicking interactions, but the paper provides no frame or analysis demonstrating non-shooting ball control.

Mitigation status. The limitation is explicitly acknowledged, and a path forward is suggested (Section 3):

"Introducing strategic priors or task-level decision cues, analogous to the tactical planning employed by human players, could encourage the policy to develop more diverse and context-aware behaviors."

However, this is a high-level proposal, not a concrete technical plan. Extending the reward function to incentivize possession, trapping, and passing would require careful design to avoid reward hacking (e.g., the policy learning to "dribble" by repeatedly tapping the ball in place rather than advancing it). It would also require expanding the observation space to include information about where passing targets are located, which in turn requires multi-agent perception capabilities (Limitation 1). The fundamental tension — "goal-oriented outcomes" driving convergence to a single pattern — is identified but not resolved.

---

Limitation 4: Difficulty Estimation Cost Is Not Accounted for in Deployment

The assumption or constraint. This is, strictly speaking, not a limitation of this paper — the paper does not use explicit difficulty estimation. However, the paper's approach to selecting when to kick and how to position relative to the ball and goal is handled implicitly by the policy's learned value function and the dense reward shaping (ball approach, goal progress). The policy does not receive an explicit difficulty signal; it learns from experience which ball positions, relative angles, and distances lead to successful kicks (rewarded) versus failures or out-of-bounds (penalized or unrewarded).

The reason this deserves mention as a practical limitation is the implicit computational cost of this implicit difficulty learning. During training, the policy processes 16,384 parallel environments × 20,000 epochs × ~hundreds of steps per episode — conservatively, tens of millions of kick attempts across the entire field — to learn which scenarios are favorable and how to adapt its gait and kicking strategy accordingly. This massive training budget (1 day on 8 V100 GPUs, roughly 192 GPU-hours) is an upfront cost that produces a policy capable of handling diverse ball positions. However, the policy does not provide:

• A calibrated confidence estimate for each kick attempt.
• A mechanism to recognize when a shot is unlikely to succeed and fall back to an alternative strategy (e.g., reposition rather than shoot).
• Any runtime adaptation to novel ball positions or goal configurations not encountered during training.

If the policy encounters a scenario where its training distribution was sparse — for instance, a ball position very close to the robot at an unusual angle, or a goal configuration not represented in the randomized field layout (which follows fixed RoboCup dimensions) — the policy may attempt a kick with low probability of success and no mechanism to detect its own uncertainty. The paper notes in Section 2.3:

"success rates were lower when the ball was positioned directly along the line between the robot and the goal, since the robot struggled to select the appropriate kicking foot, and also when the ball was placed very close to the robot, as the limited time constrained accurate ball localization and gait adaptation."

These are precisely the scenarios where explicit uncertainty estimation — recognizing "I am not well-positioned for this kick" — could trigger a repositioning behavior rather than a low-probability attempt. The policy's learned behavior does reposition (the gait adaptation in Figure 3D–E shows foot placement adjustments), but this repositioning is driven by the task rewards (ball approach, goal progress) rather than by an explicit assessment of shot quality.

What evidence exists in the paper. The success rate heatmap (Figure 3A) shows spatial variation — certain field regions have lower success rates. This tells us the policy is not uniformly capable, but it does not tell us whether the policy knows it is less capable in those regions. The policy has no explicit confidence output; the decoder produces ball position estimates (Figure 4B, 0.186 m RMSE) but not kicking success probability estimates. The 11 goals conceded in RoboCup matches (Section 2.2) provide a real-world upper bound on failure rate, but do not distinguish between failures due to opponent superiority, perceptual errors, or poor shot selection.

Mitigation status. Not addressed. The paper does not frame this as a limitation, because the policy's implicit learning of favorable scenarios is sufficient for the demonstrated performance. However, as the system scales to more complex behaviors (dribbling, passing, opponent-aware play), an explicit difficulty or confidence estimation mechanism would become increasingly valuable — the robot should not attempt a low-percentage shot when a pass to a teammate has a higher expected value. The paper's suggestion to introduce "strategic priors or task-level decision cues" (Section 3) could encompass this, but no concrete mechanism is proposed.

---

Limitation 5: No Quantitative Reactivity Measurement or Latency Analysis

The assumption or constraint. The paper's central claim — that the controller achieves "reactive soccer skills" through "tight integration of visual perception and motion control" — rests on behavioral evidence (smooth transitions, faster kicking times than rule-based strategies, adaptive gait adjustments) but is never quantified in terms of closed-loop latency or response time. The key numbers that would characterize reactivity are:

• Perception-to-action latency: Time from a visual event (e.g., ball changing direction) to a measurable change in motor output (e.g., foot trajectory adjustment).
• Reaction distance: How far the ball can travel before the robot begins responding to a trajectory change.
• Control bandwidth: The effective frequency at which visual information meaningfully influences motor commands (as distinct from the 50 Hz policy rate, which includes proprioceptive-only updates between visual frames).

The paper provides the components of latency — camera update at ~25 Hz, perception processing at ~116 ms mean latency (Appendix E), policy inference at 50 Hz — but never integrates these into an end-to-end measurement or compares the integrated latency against the rule-based baseline.

The consequence. Without quantitative reactivity measurements, the paper's central claim is supported qualitatively (videos, behavioral descriptions) but not quantitatively. This matters for several reasons:

1. Reproducibility and comparison: A practitioner evaluating whether this approach would work for their own dynamic task (e.g., catching a thrown object, intercepting a moving target) cannot determine whether the achieved reactivity is sufficient for their timescale. The robot's soccer domain involves ball speeds of ~1–5 m/s and distances of ~1–10 m, giving characteristic timescales of 0.2–10 seconds for the ball to travel between the robot and the goal — relatively forgiving compared to tasks requiring sub-100 ms responses. The paper provides no latency budget analysis to indicate what timescales the approach can handle.

2. Bottleneck identification: Is the reactivity bottleneck the camera framerate (25 Hz), the perception processing latency (116 ms), the policy inference time (unknown, but presumably < 20 ms given the shallow MLP architecture), or the mechanical response of the robot's joints (PD controller bandwidth)? Without a breakdown, it is unclear where future engineering effort should focus to further improve reactivity.

3. Comparison to rule-based approach: The paper shows that the learned policy achieves faster kicking times (Figure 6A) and higher angular velocities (Figure 6B), but these are outcome metrics, not latency metrics. It is possible that the rule-based strategy's slower kicking time is due to its sequential behavioral architecture (rotate → approach → align → kick) rather than any difference in raw perception-action latency. A direct latency comparison would isolate the "tight coupling" advantage from other factors (different foot placement strategies, different kick execution speeds).

What evidence exists in the paper. The perceptual behaviors in Figure 4C (ball maintained within FOV for majority of time) and Figure 4D (head and torso rotation tracking a ball leaving the FOV) provide qualitative evidence of reactive perception-action coordination, but these are measured as angular distributions and trajectory overlays, not as latency measurements. The 46% RMSE reduction in ball position estimation (Figure 4B, 0.344 m → 0.186 m) is a measure of estimation accuracy, not response speed.

Mitigation status. Not addressed. The paper does not frame the lack of latency measurement as a limitation, and the behavioral evidence (smooth transitions, faster kicking, adaptive gait) may be considered sufficient to support the reactivity claim for the robotics audience. However, for a reader interested in the generality of the approach to other dynamic tasks, the absence of quantitative latency characterization limits the ability to assess transferability.

---

Limitation 6: The Approach Assumes a Functional Object Detector and Odometry Module That Are Trained and Tuned Independently

The assumption or constraint. The policy's observation space (Appendix A, Table 2) assumes that a YOLOv8 object detector [59] reliably outputs ball position (with modeled noise, latency, and detection failures) and that an odometry module (Appendix F) reliably outputs goal position and direction. These components are trained and tuned independently of the RL policy: YOLOv8 is "fine-tuned on a self-collected dataset from the robot's onboard camera" (Section 4.3), and the odometry module combines an MLP trained on simulation proprioceptive data with a particle filter for visual landmark correction (Appendix F).

The RL policy is trained with the virtual perception system modeling the statistical characteristics of these components' outputs, but the policy never sees raw sensor data and cannot adapt to changes in the detection or odometry modules. If YOLOv8 is updated (e.g., a new model version with different detection probability characteristics), if the camera configuration changes (affecting noise parameters), or if the odometry module's drift characteristics differ from those modeled in simulation, the policy's training distribution no longer matches the deployment distribution.

The consequence. The policy's robustness is bounded by the accuracy of the virtual perception system's modeling of the specific perception pipeline it was trained alongside. This creates a coupling between the perception frontend and the control policy that is not captured by the standard "modularity" argument — the paper argues that abstracting perception into structured outputs "promotes generalization" (Section 4.3), but this generalization is contingent on the statistical signature of those outputs matching the training distribution. If the perception pipeline changes, the virtual perception system must be re-modeled (re-collecting data with motion capture, re-fitting noise/latency/detection distributions) and the policy must be re-trained.

This coupling is more subtle than the traditional "modular systems are brittle" critique. The paper's system is modular — perception and control are separate components — but the control module has been trained to expect a specific statistical relationship between perception outputs and ground truth. Change the perception module, and that relationship changes. The policy may exhibit degraded performance not because of architectural incompatibility, but because the noise distribution, latency characteristics, or detection failure patterns it learned to compensate for no longer apply.

Concrete failure modes include:

• Detection model upgrade: If a newer YOLO version achieves higher recall but different spatial error characteristics (e.g., better at detecting distant balls but with a different noise-vs-distance curve), the policy's learned noise filter (the encoder-decoder) is miscalibrated — it expects a certain noise level at a given distance, and may over-smooth or under-smooth the new detection outputs.
• Camera hardware change: If the camera is upgraded to a higher resolution or different lens, the relationship between distance and positional noise changes (the slope and intercept of the 0.124d + 0.149 model in Appendix E would shift). The policy would need to be re-trained with a re-modeled virtual perception system.
• Odometry drift in novel environments: The proprioceptive odometry MLP (Appendix F) is trained on simulation data with the learned policy's gait distribution. If deployed in an environment where the ground surface differs substantially from simulation (e.g., very soft grass causing foot sinkage not modeled in Isaac Gym), the odometry drift characteristics may change, affecting goal position estimates and thus kicking accuracy.

What evidence exists in the paper. The paper demonstrates robustness across diverse real-world environments — Figure 2 (J–L) shows operation on grass, slabstone, soil, asphalt, and rubber — but this tests robustness to visual variation (which the YOLOv8 detector handles, having been trained on diverse data) and terrain variation (which the policy handles, having been trained with terrain randomization), not robustness to changes in the perception pipeline itself. The virtual perception system is modeled once from a specific hardware/software configuration and never re-calibrated.

The paper also notes that the behavioral data for modeling the virtual perception system was collected with a different gait than the learned policy produces (Section 4.3). This within-deployment distribution shift is acknowledged but not measured. The fact that sim-to-real transfer succeeds despite this shift is evidence that the modeling is "simple enough to generalize," but the limits of this generalization are unknown.

Mitigation status. Partially addressed through design choices. The compact, structured observation space (ball XY, goal XY, ball mask, goal direction) means the perception module's interface is narrow and well-defined — replacing YOLOv8 with another detector that outputs ball positions in the same coordinate frame is architecturally straightforward, even if it requires re-modeling the virtual perception system. The paper does not discuss this limitation explicitly, treating the perception pipeline as a fixed component of the deployed system. A more robust approach would be to train the policy with a wider distribution of virtual perception parameters (domain randomization of the perception model itself), so that the policy learns to adapt to a range of detection characteristics rather than a single calibrated model. This would increase training difficulty but reduce coupling to the specific perception pipeline.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around vision-driven humanoid control from a perception-as-preprocessing paradigm to a perception-as-part-of-the-control-problem paradigm, with concrete engineering consequences. Prior to this work, the dominant approaches to getting humanoid robots to perform dynamic tasks with onboard vision fell into two camps: either treat perception as a separate module that outputs a cleaned-up state estimate which a controller then consumes (the modular pipeline), or attempt to train end-to-end from pixels in simulation and hope photorealism bridges the gap (the NeRF-based approach of Tirumala et al., 2024 [14]). Both suffer from the same underlying problem — they assume the perception system and the control system can be optimized independently, with the interface between them being a "good enough" state estimate. This paper demonstrates that this assumption is not just suboptimal but actively harmful for reactive tasks: the policy needs to experience the statistical characteristics of imperfect perception (noise that grows with distance, intermittent detection failures, ~116 ms latency, frequency jitter) during training to develop the internal state estimation and active sensing behaviors that make tight perception-action coupling possible.

The conceptual reframing is this: sim-to-real transfer for visually grounded control is not about making simulation look like reality — it is about making the information content of simulation match reality. The virtual perception system is the embodiment of this reframing. It models not what the camera sees (pixels, textures, lighting) but what the policy can know about the world given a specific perception pipeline (a noisy, delayed, intermittently available ball position estimate). This is a fundamentally different axis of domain randomization — randomizing over perception quality rather than visual appearance — and it is validated not by photorealism metrics (FID, SSIM) but by the statistical alignment between simulated and real hardware success rates (Figure 3A).

The magnitude of this shift is incremental rather than revolutionary. The individual components — asymmetric actor-critic, encoder-decoder architectures with reconstruction loss, AMP-style imitation, multi-critic RL — are all established techniques. The contribution is the specific integration and the demonstration that this combination solves a previously open problem (vision-driven reactive soccer that transfers zero-shot to hardware under competition constraints). This is a systems contribution more than an algorithmic one, and its impact on the field will be measured by whether the virtual perception system methodology — model the statistical signature of your perception pipeline, not its visual appearance — is adopted as a standard tool in the sim-to-real transfer toolkit. The paper makes this adoption plausible by showing that the modeling is tractable (1 hour of data collection, simple parametric distributions) and that the resulting transfer is robust across diverse real-world conditions (5 surface types, varied lighting, Figure 2).

The paper also reconciles a tension in the AMP literature that was largely unarticulated but latently present: could GAN-based motion priors, which had only been demonstrated in proprioceptive-only, static-environment settings, extend to tasks where exteroceptive perception drives behavioral decisions? The concern was that noisy visual inputs would cause the policy to produce abrupt, non-kinematic state transitions (sudden direction changes in response to ball motion) that the discriminator would reject as implausible, creating destructive interference between the style reward and the task rewards. The paper demonstrates that this interference does not occur — the discriminator accepts the policy's reactive adjustments because they lie within the learned manifold of human-like motion, and the policy autonomously composes behaviors (walking, turning, kicking) that the discriminator recognizes without explicit temporal alignment. This result makes AMP (and GAN-based imitation more broadly) a viable candidate for any dynamic robotics task where motion quality matters and the agent must respond to external stimuli — a substantially larger domain than the character animation settings where AMP originated.

Two research directions become significantly more attractive after this paper:

1. Perception-characteristic-aware sim-to-real transfer for other sensor modalities. The methodology of fitting simple statistical models to a perception pipeline's noise, latency, detection probability, and update characteristics, then injecting those models into simulation training, is general — it applies equally to LiDAR point clouds, depth images, acoustic localization, or tactile sensing. The paper's success with a single RGB camera and YOLOv8 detector provides a template: collect ground-truth data with motion capture, log the perception pipeline's outputs, fit parametric distributions to the discrepancies, and train the policy against these distributions in simulation. This is far cheaper than building photorealistic simulators for each new sensor modality and is likely to become the default approach for tasks where structured perception outputs (detections, segmentations, pose estimates) exist.

2. Implicit world modeling through reconstruction objectives as a standard component of POMDP policy architectures. The decoder ablation — showing that the 46% RMSE reduction (0.344 m → 0.186 m) in ball position estimation disappears entirely when the decoder is trained post-hoc rather than jointly — provides strong evidence that auxiliary reconstruction objectives are not merely helpful but constitutive of the latent representation's properties. This finding is likely to hold broadly: in any POMDP where the policy must filter noisy observations over time, a jointly trained reconstruction head may be necessary to force the encoder to retain physically meaningful state information. Future POMDP policy architectures should treat reconstruction objectives as a default component, not an optional enhancement.

One research direction becomes less attractive after this paper: pure end-to-end pixel-to-action training for dynamic robotics tasks. The paper's deliberate choice to abstract away raw pixels and feed the policy structured detection outputs — combined with the success of this approach both in sim-to-real transfer (Figure 3A) and in competition (76 goals, RoboCup championship) — suggests that for tasks where reliable object detectors exist, the marginal benefit of training on raw pixels (presumably to handle edge cases the detector misses) is outweighed by the massive increase in training complexity and sim-to-real gap. The NeRF-based visual training approach of Tirumala et al. (2024) [14] required training a scene representation, rendering photorealistic images at scale, and then training a policy on those images — all to produce controllers that "exhibited reduced reactivity to the ball" (Section 1). The perceptual abstraction approach achieves better reactivity with dramatically less compute and better generalization. This does not mean pixel-level training is obsolete — for tasks without reliable object detectors (e.g., novel object manipulation, deformable object tracking, cluttered scenes with heavy occlusion), pixels remain necessary. But for structured tasks where perception frontends exist and are reliable (object detection, pose estimation, depth estimation), the paper makes a compelling case that abstracting perception into structured features is the pragmatically superior approach.

---

Follow-Up Research This Work Enables

Virtual perception system component ablation: which perceptual factors are necessary for sim-to-real transfer? The paper validates the virtual perception system as an integrated whole — all four factors (positional noise, detection probability, latency, update frequency) are modeled simultaneously, and the overall sim-to-real transfer succeeds (Figure 3A). A critical missing experiment is training separate policies with individual factors removed: (a) noise only (no latency, 100% detection, fixed 25 Hz), (b) latency only (no noise, 100% detection), (c) detection dropouts only (no noise, no latency), (d) frequency jitter only, and (e) all combinations. Each policy would be evaluated on hardware using the same 10-trial-per-position protocol (Figure 3A), measuring which factors cause the largest degradation when removed and whether any single factor is sufficient to achieve most of the transfer benefit. This experiment would transform the virtual perception system from a validated monolith into an understood mechanism, telling practitioners exactly which perception characteristics they must model for their own hardware and which they can ignore. The paper already has the infrastructure (Isaac Gym training pipeline, hardware deployment pipeline, motion capture system for ground truth); the ablation requires only re-running training with modified virtual perception configurations, which is straightforward.

Training without AMP: isolating the contribution of motion priors to task performance. The paper claims AMP enables "versatile and adaptive behaviors from human demonstration" (Section 3) and that the discriminator is essential for producing "human-like" motion. However, the claim that AMP is necessary rather than helpful is untested. A direct ablation would train a policy with all task and regularization rewards (Appendix B, Table 3) but without the AMP style reward — identical architecture, identical virtual perception system, identical training scale. The evaluation would compare: (a) kicking success rates (Figure 3A protocol), (b) kicking time and angular velocity (Figure 6 protocol), (c) motion naturalness (UMAP visualization analogous to Figure 5, plus potentially a user study comparing video clips), and (d) sim-to-real transfer success (do non-AMP gaits transfer to hardware or do they exhibit unrealistic dynamics that the simulator accepts but the real robot cannot execute?). A negative result — the task-reward-only policy achieves comparable success rates but produces visually unnatural motion — would clarify AMP's role as a style mechanism, not a performance enabler. A positive result — the task-reward-only policy fails to learn stable locomotion at all — would reveal that the discriminator is providing essential exploration guidance, not just aesthetics. Either outcome refines our understanding of when and why imitation learning matters for dynamic control.

Kalman filter baseline: learned state estimation vs. classical filtering. The paper demonstrates that the encoder-decoder architecture achieves a 46% reduction in ball position estimation RMSE (0.344 m → 0.186 m, Figure 4B), and that this reduction disappears when the decoder is trained post-hoc. An important open question is whether a classical state estimator — specifically, an Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF) with a constant-velocity motion model for the ball, using the same noisy detection inputs — could achieve comparable or better filtering performance. The experiment would: (a) implement an EKF/UKF that processes the same noisy ball detections the policy receives, (b) feed the filtered ball position estimates into a policy trained with ground-truth state (no encoder-decoder, no reconstruction loss), and (c) compare kicking success rates, estimation RMSE, and sim-to-real transfer against the learned filtering approach. This comparison is practically important because a classical filter requires no learned encoder, no reconstruction loss, and no decoder — simplifying the architecture, reducing training complexity, and avoiding the "decoder must be jointly trained" constraint. If the EKF achieves similar filtering performance, the paper's architectural contribution narrows from "encoder-decoder is necessary for noise filtering" to "any temporal integration mechanism works, and the encoder-decoder is one valid implementation." If the EKF performs worse, it would suggest that the learned filter is capturing non-linear, task-specific dynamics (e.g., correlation between robot ego-motion and perceptual noise during kicks) that a generic motion model misses — an interesting finding about the nature of perceptual noise in dynamic tasks.

Closed-loop latency measurement and comparison. The paper claims "reactive soccer skills" and "tight integration of visual perception and motion control" but never measures closed-loop latency. A rigorous follow-up would instrument the hardware system to record: (a) timestamp of image capture (camera hardware trigger), (b) timestamp of YOLOv8 detection output, (c) timestamp of policy inference completion, (d) timestamp of joint command reaching the motor controller, and (e) timestamp of measurable motor response (e.g., joint velocity change exceeding a threshold). From these, compute an end-to-end latency distribution: the delay between a visual event (ball position change) and the first measurable motor response. Compare this against: (i) the rule-based baseline's latency (Figure 6), (ii) the theoretical minimum latency given the hardware components (25 Hz camera + 20 ms policy period + motor response time), and (iii) the latency reported in prior vision-based robot soccer systems (Tirumala et al., 2024 [14]). The key measurement is not mean latency but reactivity bandwidth — the frequency at which the robot can meaningfully respond to ball motion. If the learned policy achieves, say, a 180 ms median end-to-end latency vs. 350 ms for the rule-based strategy, this would quantitatively validate the "reactive" claim and provide a transferable metric for other dynamic tasks. The measurement could be done with the existing hardware — it requires only logging timestamps across the software pipeline and a test protocol with sudden ball perturbations (e.g., a second robot or human kicking the ball mid-trial) to trigger measurable responses.

Multi-agent extension with opponent modeling. The paper's most clearly stated limitation (Section 3) is the lack of multi-agent capability. A natural follow-up would extend the training environment to include an opponent robot (controlled by a separate, possibly pre-trained or scripted policy) and expand the policy's observation space to include opponent position and velocity (detected by the same YOLOv8 pipeline, with the virtual perception system extended to model multi-object detection characteristics — correlated dropouts when objects are close, occlusion by the robot's own body, false positives from field markings). The training reward would be modified to include: (a) a penalty for opponent ball possession, (b) a reward for maintaining possession under opponent pressure, and (c) a reward for goals scored against an actively defending opponent. The evaluation would measure: (i) goal differential in 1v1 matches against the rule-based baseline, (ii) whether emergent defensive behaviors appear (positioning between opponent and goal, blocking shooting lanes), and (iii) whether the policy learns to modulate its kicking strategy based on opponent position (power shots when the goalkeeper is out of position, placed shots when the near post is covered). This extension directly tests whether the "tight perception-action coupling" the paper demonstrates for single-agent skills scales to the interactive setting, and whether the virtual perception system methodology generalizes to multi-object tracking characteristics. The RoboCup results (76 goals, 11 conceded against presumably less-capable opponents) provide a baseline; 1v1 matches against a competitive opponent would reveal whether the policy's current advantage is primarily athletic (faster, more agile) or also strategic (capable of adapting to defensive pressure).

Cross-platform generalization: does the methodology transfer to different humanoid hardware? All results are on the Booster T1 platform. The key open question for generalization is whether the approach is platform-specific (the virtual perception system parameters, reward weights, AMP reference motions, and encoder-decoder architecture are tuned to the T1's kinematics, mass distribution, and perception hardware) or platform-general (the methodology — model your perception pipeline's statistics, train with AMP, use encoder-decoder with reconstruction loss — works across platforms with appropriate retargeting). A strong follow-up would replicate the pipeline on a second humanoid platform (e.g., the Unitree H1, Tesla Optimus, or a RoboCup Kid-size robot) with the following protocol: (a) retarget the same reference motion dataset to the new platform's kinematics, (b) collect new virtual perception system parameters using the same 1-hour data collection protocol with motion capture, (c) train with identical hyperparameters (Appendix A, Table 1) except for platform-specific joint limits and PD gains, (d) evaluate using the same Figure 3A protocol. Success would mean: the new platform achieves comparable sim-to-real alignment (hardware success rates matching simulation predictions) and qualitatively similar behavioral clusters (UMAP visualization analogous to Figure 5). This experiment tests the paper's implicit claim that the methodology, not the specific tuning, is the contribution. It also addresses a practical question: can RoboCup teams adopt this approach without needing the exact hardware the authors used?

---

Practical Applications and Downstream Use Cases

Competition robot soccer with reduced engineering effort. The most immediate practical application — already demonstrated — is in RoboCup and similar robot competitions. The paper's approach replaces hundreds of hours of hand-engineering (behavior tree design, gait parameter tuning, kick trajectory optimization, perception-to-control interface debugging) with a single training pipeline that produces a controller capable of autonomous play after approximately 1 day of GPU training and 1 hour of perception data collection. The quantitative advantage over the state-of-the-art is clear: the learned policy achieves faster kicking times (Figure 6A, consistently below 2 seconds vs. up to 5 seconds for the rule-based runner-up team's strategy) and higher agility (Figure 6B, higher angular velocities across all approach angles). For competition teams, this translates directly to competitive advantage — reaching the ball first in contested situations, executing kicks before defenders can close distance, and maintaining performance under the time pressure of tournament adaptation (the RoboCup constraint of "adaptation time to the actual competition venue was highly limited," Section 2.2). The 76-goal, 11-conceded RoboCup 2025 championship performance provides a real-world benchmark: a team adopting this methodology can expect to dominate possession and scoring opportunities against teams using modular, hand-engineered control strategies. The practical barrier to adoption is the need for motion capture ground truth during the 1-hour perception data collection, which may not be available at all competition venues — a lower-cost calibration procedure (using fiducial markers or structure-from-motion to estimate ground-truth ball position) would significantly broaden accessibility.

Dynamic object interception for logistics and manufacturing. Beyond soccer, the core capability this paper demonstrates — vision-driven reactive locomotion to intercept and interact with a moving object — applies directly to warehouse and factory settings where mobile robots must pick, place, or manipulate items on conveyor belts, moving carts, or dynamic shelves. A humanoid or wheeled-legged robot equipped with a similar perception-to-control pipeline could: (a) visually track items moving at ~1–3 m/s on a conveyor, (b) adjust its walking or rolling gait to intercept the item at a specific location, and (c) execute a manipulation action (grasp, push, scan) with timing synchronized to the item's arrival. The paper's specific contributions translate to this domain: the virtual perception system methodology enables training entirely in simulation (no real-world conveyor belt data needed, just a statistical model of the perception pipeline's noise and latency when observing moving items), the encoder-decoder architecture provides the temporal filtering needed to estimate item velocity from noisy position detections (analogous to the 46% RMSE reduction for ball position), and the AMP framework ensures the resulting locomotion is stable and human-safe (important for human-robot collaborative workspaces). The key adaptation would be replacing the soccer-specific reward structure (ball approach, goal progress) with task-specific shaping (item approach, grasp success) and the kicking motion dataset with manipulation demonstrations. The 1-day training time on 8 V100 GPUs is practical for industrial deployment where retraining for new item types or conveyor configurations may be needed periodically.

Assistive robotics: fetching and delivering objects in dynamic home environments. A humanoid or mobile manipulator assisting an elderly or disabled person in a home setting must track and retrieve objects that may be moving (a rolling medication bottle, a sliding remote control, a pet pushing a toy) while navigating around furniture and maintaining balance. The paper's approach provides a blueprint: (a) a perception pipeline (YOLOv8 or similar) detects the target object and outputs its position in the robot's coordinate frame, (b) a virtual perception system models the specific noise and latency characteristics of this detection pipeline in the home environment (which may differ substantially from the controlled soccer field — worse lighting, more clutter, more occlusions), (c) the policy is trained in simulation with randomized home layouts and object dynamics, using the virtual perception system to inject realistic perceptual degradation, and (d) deployed zero-shot to the physical home. The key advantage over current assistive robotics approaches (which typically use SLAM for navigation and separate grasp planning for manipulation) is the seamless transition from navigation to interaction — the robot does not stop, localize, plan a grasp, and execute; it adjusts its foot placement during approach so that the final step positions its end-effector for a natural reach. This is exactly the "tight perception-action coupling" the paper demonstrates for soccer, translated to a different task domain. The 0.186 m RMSE ball position estimation after filtering (Figure 4B) — approximately the length of the robot's foot arch — provides a relevant benchmark: for assistive grasping, position estimation accuracy must be within the gripper's tolerance, which for typical parallel-jaw grippers is 1–5 cm. The paper's filtering approach achieves ~18 cm accuracy for a fast-moving soccer ball at distances of 1–7 meters; for a slower-moving household object at closer range (0.5–2 meters), the accuracy would likely improve due to lower perceptual noise at close distances (the 0.124d + 0.149 noise model predicts 0.21 m noise standard deviation at 0.5 m, vs. 1.02 m at 7 m).

---

When to Prefer This Method

The paper does not present a systematic comparison against named alternative approaches with explicit tradeoff conditions. It compares against one specific rule-based strategy (Section 2.5, Figure 6) but does not position its method within a broader taxonomy of vision-driven control approaches (e.g., "use our method vs. end-to-end pixel training vs. modular perception + model predictive control vs. behavior cloning from human teleoperation"). The decision criteria are implicit in the design choices and results but are not articulated as an explicit tradeoff framework by the authors. A forced "prefer A when X, prefer B when Y" matrix using only the paper's direct comparisons would reduce to a single underpowered comparison against the RoboCup runner-up's behavior-tree approach, which is not a representative baseline for the broader field. The reader should note this as a limitation of the paper's positioning rather than as a gap in the summary.