HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents

🎯 Pitch

A 2B-parameter vision-language model, distilled from a much larger reasoning system, now executes complex real-world plug-insertion tasks with 85% success—outperforming a baseline by 30 points. This slims an embodied AI enough to run on edge devices yet still beat frontier models on 16 out of 22 robotic perception benchmarks.

1. Executive Summary

This paper introduces HY-Embodied-0.5, a family of foundation models purpose-built for real-world embodied agents, spanning an efficient 2B-parameter variant for edge deployment and a powerful 32B-activated-parameter variant for complex reasoning. The work systematically enhances embodied intelligence through a Mixture-of-Transformers (MoT) architecture (modality-specific QKV and FFN layers for visual vs. textual tokens, enabling bidirectional attention for vision alongside causal attention for language), visual latent tokens (learnable tokens appended to visual inputs that bridge vision and language via explicit alignment supervision), and an iterative, self-evolving post-training paradigm (alternating reinforcement learning with rejection sampling fine-tuning to cultivate deep thinking, followed by large-to-small on-policy distillation). The MoT-2B model achieves state-of-the-art performance on 16 out of 22 embodied, spatial, and perception benchmarks among similarly sized models—outperforming generalist Qwen3-VL-4B by 10.2 percentage points and specialist RoboBrain2.5-4B by 8.6 percentage points in average score—while the MoE-A32B variant reaches 67.0% average, surpassing Gemini 3.0 Pro (63.6%). In real-world robot control, the resulting Vision-Language-Action model achieves 85% success on precision plug-in packing and 75% on mug hanging—representing a 25–30 percentage point margin over π0 baselines on the hardest manipulation task—establishing that embodied foundation models can translate digital intelligence into physical competence only when both fine-grained visual perception architectures and reasoning-centric post-training are jointly optimized.

2. Context and Motivation

The Core Gap: VLMs Are Not Embodied

The fundamental problem this paper tackles is that general-purpose Vision-Language Models (VLMs) fail at the capabilities that matter most for physical agents. While recent VLMs like Qwen-VL, GPT-4V, and Gemini have achieved remarkable performance on web-scale visual tasks—image captioning, visual question answering, document understanding—they exhibit a specific pattern of deficiency when confronted with the demands of embodied environments. The paper identifies two primary dimensions where existing VLMs fall short (Section 1):

Fine-grained visual perception is underdeveloped. Embodied tasks require precise spatial grounding: locating a specific object in a cluttered scene, estimating its 3D position, identifying graspable regions, and tracking motion over time. These capabilities demand pixel-level and coordinate-level precision that mainstream VLMs—trained predominantly on internet-scale image-text pairs where high-level semantic understanding suffices—do not reliably deliver. The paper explicitly states that "existing VLMs still exhibit notable deficiencies in capturing the granular details required for physical grounding." This is not merely an accuracy issue; it is a fundamental mismatch between the perceptual resolution required (e.g., distinguishing a grasp point at coordinates (432, 587) from one at (438, 592)) and the resolution that standard VLM training incentivizes (coarse image-level understanding).

Embodied prediction, interaction, and planning are absent from standard training. Even within VLMs that can perceive, their training on static, temporally disconnected web images leaves them unprepared for the sequential, action-oriented reasoning that physical environments demand. The paper notes that mainstream VLMs "remain inadequately optimized for embodied environments, lacking the action-oriented capabilities essential for dynamic prediction, interaction, and planning in the physical world." A VLM that correctly identifies a mug in an image may have no capacity to predict what happens when a robot arm approaches that mug, to plan a sequence of actions to grasp and place it, or to reason about spatial constraints (e.g., "the mug is behind the bowl, so I must move the bowl first").

This is a capability gap, not merely a performance gap. Existing VLMs were not designed for these tasks, and their training curricula did not expose them to the relevant data distributions or reasoning patterns. The result is models that can answer "what is in this image?" but cannot answer "how should the robot move to pick up the object on the left without disturbing the stack on the right?"—a question that requires integrating fine-grained perception with sequential planning.

Why This Problem Matters: Embodiment as the Next Frontier

The paper situates its work within a broader narrative of extending intelligent agents into the physical world (Section 1). It frames embodied VLMs as the natural bridge between two converging trajectories:

From digital to physical agents. LLMs and VLM-based agents have demonstrated transformative capabilities in digital domains—coding (SWE-Agent, Yang et al., 2024a), computer use (OS-Copilot, Wu et al., 2024b), scientific research (AI Scientist, Lu et al., 2024), and personal assistance. The paper cites this body of work to establish that agent-based problem-solving is now a "foundational paradigm." The logical next step is extending these agents into the physical world for tasks like manufacturing, logistics, home assistance, and healthcare robotics. However, this transition is non-trivial because physical environments add dimensions of perceptual noise, 3D spatial reasoning, real-time dynamics, and safety constraints that digital agents do not face.

The VLM bridge. The paper positions VLMs as uniquely suited to bridge LLM agents and physical agents because they already combine the language understanding and reasoning capabilities of LLMs with visual perception. The key claim is that "embodied VLMs uniquely bridge the gap between LLM agents and physical agents, enabling the vast open-world knowledge of large language and multimodal models to be fully leveraged for real-world tasks" (Section 1). In other words, if you can make a VLM that genuinely understands the physical world, you can immediately tap into the reasoning, planning, and common-sense knowledge that LLMs have already developed—without needing to train these capacities from scratch in a robotics-specific architecture.

Practical deployment constraints matter. The paper explicitly addresses a deployment-relevant tension: the models that excel at complex reasoning tend to be large (tens to hundreds of billions of parameters), while real-world robots—especially those operating on-device for latency, privacy, or cost reasons—require compact models that can run efficiently at the edge. The 2B-parameter variant is not an afterthought but a first-class design target, motivated by the practical reality that embodied agents must often operate under strict compute and latency budgets. This makes the problem harder: you cannot just scale up a VLM and call it embodied; you need architectural and training innovations that achieve strong embodied capabilities within a small parameter footprint.

Where Prior Approaches Fall Short

The paper does not position itself against a single line of prior work but rather identifies systematic limitations across the current landscape of models, data, and training strategies relevant to embodied VLMs.

Generalist VLMs lack specialized embodied competence. Models like Qwen-VL (Bai et al., 2023, 2025) and InternVL are strong at general visual understanding but were not trained with embodied data or tasks as a design priority. Their training mixtures are dominated by web images with text descriptions, which teaches semantic understanding but not spatial precision or temporal reasoning. The paper's evaluation (Table 1, Figure 7) quantifies this gap: Qwen3-VL-2B and Qwen3-VL-4B achieve significantly lower scores on embodied benchmarks (e.g., 41.8% and 47.3% on ERQA, respectively, versus 54.5% for HY-Embodied-0.5 MoT-2B; 39.9% and 43.9% on 3DSRBench versus 57.0%). The fact that these strong generalist models underperform so substantially on embodied tasks—despite having similar or larger parameter counts—is direct evidence that general VLM training does not transfer automatically to embodied competence.

Specialist embodied models face scope and scale limitations. On the other side, models specifically designed for embodied tasks, such as RoboBrain 2.5 (Tan et al., 2026) and MiMo-Embodied (Xiaomi Embodied Intelligence Team, 2025), have made progress but are constrained by several factors. First, their training data often comes from narrower sources (specific robot platforms or lab environments), which limits generalization. Second, they typically prioritize certain embodied capabilities (e.g., affordance prediction, trajectory generation) without simultaneously maintaining strong general visual understanding—creating models that work well on their target tasks but degrade on broader perception benchmarks. Third, prior embodied models have generally not demonstrated strong Chain-of-Thought reasoning or the ability to engage in deep, multi-step thinking processes, which the paper shows is critical for complex spatial and planning tasks (Figure 10).

Architectural approaches for multi-modal models are poorly suited for embodied perception. Standard VLM architectures process visual tokens and text tokens through shared transformer layers, treating them identically despite their fundamentally different structures. The paper identifies this as a source of inefficiency: visual data lacks the unidirectional, sequential nature of text, yet it is forced through causal attention mechanisms designed for autoregressive language generation. This mismatch is especially damaging for small models, where shared parameters must serve two very different purposes, limiting the total capacity that can be allocated to visual understanding. The paper's Mixture-of-Transformers architecture (Section 2.2) is a direct response to this limitation, enabling modality-specific computation that the paper shows accelerates convergence and improves performance (Figure 11) without adding inference overhead.

Existing training pipelines do not cultivate embodied reasoning. Even when embodied data is included in training, prior work has not systematically addressed how to train models to perform the multi-step spatial reasoning that physical tasks require. Standard supervised fine-tuning on question-answer pairs teaches the model to produce correct answers but does not necessarily teach the process of reasoning through a problem. The paper's iterative RL→RFT→distillation pipeline (Section 4) is motivated by the observation that embodied reasoning requires a combination of exploration (discovering new reasoning strategies through RL), consolidation (solidifying successful strategies through rejection sampling fine-tuning), and transfer (compressing these capabilities into smaller models through on-policy distillation). None of these components is individually novel—GRPO, RFT, and distillation are established techniques—but the paper argues that their combination in an iterative, self-evolving paradigm specifically for embodied tasks is a novel contribution that prior work has not explored.

Verifier and reward design for embodied RL is underdeveloped. Applying reinforcement learning to embodied models presents a unique challenge: the target outputs span fundamentally different structures—2D bounding boxes, 3D coordinates, trajectory waypoints, discrete action choices, open-ended text descriptions. A single reward function (e.g., exact match) is inadequate because it cannot distinguish between "almost correct" (a bounding box with 0.85 IoU, a trajectory that reaches the target but takes a suboptimal path) and "completely wrong" (a bounding box on the wrong object). The paper's task-aware reward design (Section 4.2.2, Figure 6) categorizes rewards into grounding-based (IoU, normalized L2 distance), regression-based (relative error decay), trajectory-based (DTW, Fréchet distance), and text-based (LLM judge) types, arguing that matching reward structure to output structure is essential for stable optimization—a consideration largely absent from prior embodied model training.

Small models for embodiment are especially underserved. There is a particular gap in strong small (<4B parameter) VLMs for embodied tasks. Generalist small models (like Qwen3-VL-2B) underperform because their training does not emphasize spatial and embodied reasoning. Specialist embodied models tend to be larger (RoboBrain-2.5 is 4B, MiMo-Embodied is 7B) or, when small, show significant performance degradation. The paper's targeting of a 2B-activated model that outperforms larger competitors (Table 1) addresses a genuine deployment need: robots require models that can run at real-time inference speeds on edge hardware, and every millisecond of latency matters for closed-loop control. The architectural innovations (MoT, visual latent tokens) and the large-to-small distillation pipeline are specifically motivated by this efficiency requirement.

How the Paper Positions Itself

The paper positions HY-Embodied-0.5 at the intersection of three lines of work, aiming to synthesize their strengths while addressing their individual limitations:

Building on the VLM paradigm, but making it embodied-first. The model architecture follows the established vision-encoder + LLM backbone pattern (Liu et al., 2023), but the paper departs from general-purpose VLM design in every component: the vision encoder is optimized for native-resolution inputs and edge deployment (HY-ViT 2.0, Section 2.1); the transformer backbone is modified with modality-specific parameters (MoT, Section 2.2); and training data is dominated by embodied and spatial content rather than general web data (Figure 4 shows over 200B tokens of embodied/perception data, with spatial and robotics comprising 43% of this subset).

Leveraging reasoning advances from LLM research, but adapting them to multimodal embodied settings. The iterative RL→RFT pipeline draws on DeepSeek-R1's demonstration that reasoning can be cultivated through reinforcement learning, and the on-policy distillation approach follows recent work from Thinking Machines Lab and others. However, the paper adapts these techniques specifically for embodied VLMs: the reward design is structured around geometric, trajectory, and grounding outputs rather than math or code solutions; the RFT stage filters reasoning traces based on spatial correctness and quality rather than just final-answer accuracy; and the distillation target is embodied competence specifically, not general language capability.

Competing with both generalist and specialist models, but claiming complementary advantages. The paper evaluates against both categories (Table 1, Table 2) and implicitly argues that HY-Embodied-0.5 achieves the best of both: general visual understanding competitive with size-matched generalist VLMs (Figure 7) plus embodied competence exceeding specialist models. This is a "no-compromise" positioning—the claim is not that HY-Embodied-0.5 sacrifices general capability for embodied skill, but that its training pipeline and architecture enable it to maintain both simultaneously.

Open-source as a deliberate strategy. The paper notes that "Code and models are open-sourced" (Abstract), which in the context of embodied AI—where many leading models from industry labs remain proprietary—represents a significant research contribution. The comprehensive evaluation suite of 22 public benchmarks itself serves as a resource that the broader community can use to standardize embodied VLM assessment.

Acknowledged scope boundaries. The paper does not claim to solve all of embodied AI. It focuses on the VLM foundation rather than end-to-end robot learning (Section 6 shows that a separately trained VLA model is needed for robot control). It does not address sim-to-real transfer, safety, or multi-agent coordination. The positioning is explicitly as a foundation model—a base upon which downstream embodied applications can be built—rather than a complete embodied agent system.

The Unstated Motivation: A Training Pipeline, Not Just a Model

While the paper presents HY-Embodied-0.5 as a model family, a careful reading reveals that the primary contribution is arguably the training methodology—the specific recipe of data composition, multi-stage training, iterative self-evolution, and distillation. The model architectures (MoT, visual latent tokens) are important enablers, but the paper's strongest results (4× efficiency gains, strong small-model performance, competitive large-model results) emerge from the training pipeline. This is evidenced by the extensive space devoted to data construction (Section 3.1, spanning pages 6–10), training recipes (Sections 3.2, 4.1.2, 4.2.3), and the iterative post-training paradigm (Section 4.3), relative to the architectural descriptions. The paper's contribution is therefore best understood as a demonstration that, given appropriate data and training strategies, compact VLMs can achieve embodied competence that previously required much larger models—a finding with significant implications for the deployment of embodied AI systems.

3. Technical Approach

3.1 Reader Orientation

HY-Embodied-0.5 is a family of Vision-Language Models (VLMs) specifically engineered to serve as the perceptual and reasoning core for robots and other embodied agents — it takes in camera images and natural language instructions, and it outputs spatial coordinates, action plans, scene descriptions, or answers to questions about the physical world. The problem it solves is that general-purpose VLMs, while fluent at describing images, cannot perform the precise spatial grounding and multi-step physical reasoning required to control a robot arm or navigate a 3D environment; the solution takes the shape of a three-part system of architecture, data, and training — a modality-adaptive transformer that gives visual and textual tokens their own computational pathways, a massive pre-training corpus dominated by spatial and embodied data (over 200B tokens), and an iterative post-training pipeline that alternates reinforcement learning with rejection sampling to cultivate deep, self-correcting reasoning chains, finally distilling this capability from a large 32B model into a compact 2B model suitable for edge deployment.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components connected in a feedforward pipeline with iterative feedback loops during training:

HY-ViT 2.0 Vision Encoder (400M parameters) — a native-resolution Vision Transformer that processes camera images at arbitrary aspect ratios, producing a sequence of visual tokens (one per image patch) plus a special visual latent token. It is trained via distillation from a larger internal ViT to achieve efficient, accurate visual representations suitable for edge deployment.
Mixture-of-Transformers (MoT) Backbone — the core architectural innovation. Instead of processing visual and text tokens through the same transformer layers, the MoT maintains separate QKV (query, key, value) projection weights and Feed-Forward Network (FFN) weights for the vision branch versus the language branch. Visual tokens receive bidirectional (full) attention while text tokens receive causal (autoregressive) attention. This doubles the effective parameter count for visual processing without proportionally increasing inference cost.
Visual Latent Tokens — a set of learnable embedding vectors appended to each visual input sequence. They act as information aggregators: during pre-training, their output hidden states are explicitly supervised to match the global CLS feature of a larger teacher ViT (via a cosine similarity loss), forcing them to capture holistic image semantics that bridge the visual and language modalities.
Large Language Model Backbone (Hunyuan-1.8B) — the base language model that provides reasoning, planning, and language generation capabilities. In the 2B variant, it uses the MoT architecture (2B activated parameters out of 4B total); in the 32B variant, it uses a Mixture-of-Experts (MoE) architecture with 32B activated parameters out of 407B total.
Post-Training Pipeline (RL → RFT → Distillation) — an iterative, self-evolving training loop applied after pre-training. It alternates between Group Relative Policy Optimization (GRPO) for exploration, Rejection Sampling Fine-Tuning (RFT) for consolidation, and finally On-Policy Distillation (OPD) to transfer the large model's reasoning behavior into the compact variant.

Information flows as follows: a camera image enters the HY-ViT 2.0 encoder → visual tokens pass through the vision-specific MoT layers with bidirectional attention, while text tokens pass through the language-specific MoT layers with causal attention → the visual latent token aggregates global image information and is supervised to align with the teacher ViT's representation → the LLM backbone generates text output (reasoning chains, bounding boxes, action plans) autoregressively, conditioned on both visual and textual inputs → during post-training, the model's own generated outputs are scored by task-specific reward functions and fed back to improve policy.

3.3 Roadmap for the Deep Dive

First, the HY-ViT 2.0 encoder — because all downstream perception depends on the quality of visual tokenization, and the design choices (native resolution, distillation, discrete code supervision) are foundational to everything that follows.
Second, the Mixture-of-Transformers architecture — the central innovation that enables modality-specific computation. This covers the parameter duplication strategy, the distinct attention patterns, and the visual next-code prediction auxiliary loss.
Third, visual latent tokens — a smaller but important mechanism that bridges vision and language, including the global alignment loss that supervises them.
Fourth, the pre-training data and recipe — because the model's capabilities are fundamentally shaped by what it sees during training. This covers the five data categories (omni-detection, depth, segmentation, pointing/counting, embodied-centric, spatial-centric) and the two-stage training schedule.
Fifth, the post-training pipeline — the RL→RFT→distillation loop that cultivates reasoning. This includes the task-aware reward design (grounding, regression, trajectory, text-based rewards), the GRPO objective, the rejection sampling procedure, and the on-policy distillation loss.
Sixth, the training strategy and loss functions — the composite pre-training loss (LLM loss + vision loss + global loss) and how it transitions to single-loss fine-tuning.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that embodied VLMs require joint optimization across architecture (MoT for modality-adaptive computation), data (massive embodied-centric pre-training corpus), and training strategy (iterative self-evolving post-training with on-policy distillation) to achieve the fine-grained spatial perception and multi-step physical reasoning that general-purpose VLMs lack.

HY-ViT 2.0: Efficient Native-Resolution Visual Encoder

The vision encoder is the first processing stage that converts raw camera pixels into a sequence of embedding vectors that the language model can consume. HY-ViT 2.0 is an upgraded version of the HY-ViT series (from HunyuanOCR) designed specifically for the constraints of embodied deployment: it must handle arbitrary image resolutions without distortion, run efficiently on edge hardware, and preserve the fine spatial detail needed for grounding tasks like "locate the handle of the mug at pixel coordinates (342, 518)."

Native-resolution processing. Unlike many VLMs that resize or pad all input images to a fixed square resolution (e.g., 224×224 or 336×336), HY-ViT 2.0 "inherently supports arbitrary-resolution inputs" (Section 2.1). This is critical for embodied tasks because resizing destroys the spatial fidelity needed for precise coordinate prediction — a bounding box that is correct at native resolution may shift by dozens of pixels after downscaling and upscaling. The architecture follows the NaViT (Dehghani et al., 2023) paradigm of "Patch n' Pack," where variable-resolution images are processed as sequences of patches without forced resizing. The specific patch size is not stated in the paper, but standard ViT configurations use 16×16 or 14×14 pixel patches.

Scale and distillation. The model employs a 400M-parameter ViT — substantial enough to capture fine-grained visual features, but smaller than the multi-billion-parameter encoders used in some frontier VLMs. To achieve strong representations at this parameter count, the model is trained via knowledge distillation from "a more powerful internal ViT" (Section 2.1). Distillation works by training the student ViT to match the output features of a larger, already-trained teacher ViT, typically by minimizing the L2 distance or cosine similarity between their output embeddings. The paper does not specify the exact distillation loss, but the effect is that the 400M student inherits the representational quality of a much larger model without the inference cost.

Discrete visual code supervision. An additional training mechanism unique to HY-ViT 2.0 is the use of discrete visual representations. A larger version of the ViT is trained to produce discrete codes: it compresses every 8×8 image patch into a single integer from a codebook of size 2,048 (Section 2.1). This is analogous to VQ-VAE-style tokenization — the continuous visual signal is quantized into a discrete vocabulary, similar to how text is tokenized into subword units. These discrete codes serve two purposes: they provide a compact, information-dense representation of visual content, and they can be used as supervision targets for the model's visual tokens during pre-training (the "vision loss" in Section 3.3, discussed below). The rationale is that forcing the model to predict these discrete codes from its internal visual representations ensures that fine-grained visual information is preserved through the encoding pipeline, rather than being lost to the "information bottleneck" of projecting high-dimensional pixels into lower-dimensional embeddings.

Multi-signal supervision. Beyond the discrete code supervision, HY-ViT 2.0 incorporates two additional supervision signals during pre-training: a "tiny LLM to provide language supervision signals" and "visual reconstruction supervision to ensure minimal information loss in the visual signals fed to the LLM" (Section 2.1). The language supervision likely takes the form of aligning visual features with text embeddings (e.g., via contrastive learning), while the reconstruction supervision ensures the encoder preserves enough information that the original image could be approximately recovered from the encoded features — a standard technique for preventing representational collapse. The paper does not provide details on the exact architectures or loss functions for these signals.

Why this design? The constellation of design choices — native resolution, distillation to 400M, discrete code supervision, and multi-signal training — reflects the specific demands of embodied perception. Native resolution preserves spatial precision; distillation enables efficiency without sacrificing quality; discrete codes provide a structured, lossy-compression target that biases the encoder toward preserving object boundaries and spatial relationships (the information most relevant for grounding); and the combination of reconstruction and language alignment ensures the representations serve both low-level perception and high-level semantic understanding.

Modality-Adaptive Computing with Mixture-of-Transformers

The Mixture-of-Transformers (MoT) architecture is the paper's central architectural contribution, addressing a fundamental tension in VLM design: visual and textual tokens have fundamentally different statistical structures, yet standard transformer architectures process them identically. Visual tokens represent spatial neighborhoods in a 2D grid — nearby patches are semantically related regardless of sequence order — while text tokens form a unidirectional, temporally ordered sequence where each token depends only on previous tokens. Forcing both modalities through the same attention mechanism and the same feed-forward layers is suboptimal, particularly for small models where parameter capacity is scarce.

Parameter duplication strategy. The MoT architecture begins with a standard pre-trained language model (Hunyuan-1.8B, Tencent Hunyuan Team, 2025). Before multi-modal training begins, the Feed-Forward Network (FFN) parameters and the QKV (Query, Key, Value) projection parameters are duplicated: "we duplicate the Feed-Forward Network (FFN) and QKV parameters of the language model, initializing these duplicated parameters with the weights of the pre-trained LLM" (Section 2.2). After duplication, there are two complete sets of transformer layer parameters — one for processing visual tokens, one for processing text tokens — that start from the same initialization but will diverge during training as they specialize to their respective modalities.

During the forward pass, the routing is deterministic and token-identity-based: "all visual tokens output by the ViT are computed using this duplicated set of parameters, while text tokens are computed using the original text-specific parameters" (Section 2.2). This means there is no learned gating mechanism or dynamic routing — every visual token unconditionally uses the vision-specific QKV and FFN, and every text token unconditionally uses the language-specific ones. This is simpler than Mixture-of-Experts, which uses a learned router to dynamically assign tokens to experts, but it achieves a similar effect for the specific case of two modalities with clearly distinct properties.

The practical effect is that the total parameter count approximately doubles (since the FFN and QKV parameters, which constitute the majority of transformer parameters, are duplicated), but the activated parameter count — the number of parameters actually used for any given token — remains unchanged because each token only passes through one set of parameters. For the MoT-2B variant, this yields 2B activated parameters (the count that matters for inference latency and memory) out of 4B total parameters. The paper notes this strategy is "especially effective for small edge models, as it doubles the inherently limited total parameter count while introducing negligible overhead to training and inference efficiency."

Distinct attention patterns. Beyond the parameter separation, the MoT architecture applies different attention masks to visual and text tokens (Figure 3). Visual tokens receive bidirectional (full) attention: each visual token can attend to every other visual token in the same image, regardless of position. This makes intuitive sense — the relationship between the top-left patch and the bottom-right patch of an image is not inherently sequential; the model should be able to draw on global visual context when processing any local region. Text tokens, by contrast, receive causal (autoregressive) attention: each text token can only attend to previous text tokens, preserving the left-to-right generation constraint needed for autoregressive language modeling.

The attention computation is visualized in Figure 3, which shows a concrete example with interleaved multi-modal sequences. The key property is that visual tokens attend to other visual tokens bidirectionally, but the interaction between visual and text tokens follows causal constraints — text tokens can attend to all previous visual tokens (i.e., the model can "look at" the image when generating each word), but visual tokens do not attend to text tokens that come after them in the sequence. This preserves the autoregressive generation capability while giving the visual branch the bidirectional context it benefits from.

Visual next-code prediction loss. To further strengthen the vision branch's representational quality, the paper introduces an auxiliary training objective: predicting the discrete visual codes of the next patch. Using the discrete representations from the larger ViT (described in Section 2.1) as supervision targets, an MLP module is applied to the LLM output features of the vision branch to predict the discrete code of the next patch in sequence (Section 2.2, "Vision Loss" in Figure 2). This is analagous to a next-token prediction task but in visual space — the model learns to anticipate what the next image patch should look like, which forces the visual representations to encode spatial structure, object boundaries, and texture patterns.

The discrete codes come from a codebook of size 2,048, where each code represents an 8×8 image patch. Predicting these codes is a 2,048-way classification problem at each visual token position. The cross-entropy loss for this task is described in Section 3.3:

$\mathcal{L}_{\text{vision}} = -\frac{1}{N_v} \sum_{i=1}^{N_v} \log p_i(z_i)$

where $N_v$ is the number of visual tokens, $p_i$ is the predicted probability distribution over the 2,048 codebook entries for the $i$ -th token, and $z_i$ is the target discrete code generated by the teacher ViT for that position.

What it computes: for each visual token position, the model produces a probability distribution over the 2,048 possible discrete visual codes. The loss takes the negative log probability assigned to the correct code, averaged over all visual token positions. A perfect prediction (probability 1.0 on the correct code) yields zero loss; random guessing yields a loss of approximately $\log(2048) \approx 7.63$ .

Why this form: this is a standard categorical cross-entropy loss, which is the maximum-likelihood objective for a discrete classification target. It is the correct choice because the target $z_i$ is a categorical variable (one of 2,048 discrete codes), not a continuous value. An alternative like MSE would treat the codebook indices as continuous numbers, imposing an arbitrary ordinal structure on what is fundamentally an unordered set of discrete symbols. The cross-entropy loss makes no assumptions about the topology of the codebook space, treating each code as an independent category.

The rationale for including this auxiliary loss is that the standard language modeling loss alone may not provide sufficient gradient signal to the vision branch, particularly in early training when the model has not yet learned to align visual and textual representations. The visual next-code prediction task provides a dense, position-specific supervision signal directly to the visual tokens, ensuring that each token's representation carries locally discriminative information about its corresponding image region. This is important for fine-grained perception tasks like pointing (predicting a specific pixel coordinate) and counting (distinguishing individual object instances), where per-patch visual detail matters.

Training efficiency benefits. The paper reports in Figure 11 that the MoT architecture achieves "faster convergence and lower final loss during training, while introducing almost no additional overhead during inference" compared to a standard dense transformer with the same activated parameter count. The training curves (Figure 11a) show the MoT variant achieving lower training loss at the same number of steps, and the inference benchmarks (Figure 11b) show nearly identical total inference time, theoretical FLOPs, prefill time, and decode time between the MoT and Dense-2B architectures. The reason for the negligible inference overhead is that the parameter duplication only increases total parameter count (which affects memory but not computation), not activated parameter count (which affects FLOPs and latency). During decoding, each token still passes through exactly one set of QKV and FFN parameters, so the computational cost per token is unchanged.

Visual Latent Tokens Connecting Vision and Language

Visual latent tokens are a small but important architectural component that serves as an explicit bridge between the visual and language processing streams. The motivation draws on two lines of prior work: latent thinking (Zelikman et al., 2024; Pfau et al., 2024), where models use internal "scratch space" tokens for computation before producing output, and vision registers (Darcet et al., 2023), where learnable tokens appended to ViT inputs provide global storage that improves feature quality.

Mechanism. The paper "append[s] a learnable visual latent token to the end of each visual element (e.g., an image or a video frame)" (Section 2.3). For each image input, the sequence of patch tokens from the ViT is augmented with one additional token vector that is initialized randomly and learned during training. This token participates in the bidirectional attention among visual tokens (it can attend to all patch tokens and they can attend to it) but also interacts with text tokens through the causal attention mechanism — text tokens can attend to the latent token when generating output.

Global alignment supervision. During pre-training, the output hidden state of the latent token (after passing through the vision-specific MoT layers) is explicitly supervised to match a target representation: "the global CLS feature extracted from the teacher ViT" (Section 2.2). This is implemented via a negative cosine similarity loss:

$\mathcal{L}_{\text{global}} = -\frac{f_{\text{latent}}^{\top} f_{\text{teacher}}}{\|f_{\text{latent}}\| \|f_{\text{teacher}}\|}$

where $f_{\text{latent}} \in \mathbb{R}^d$ is the mapped hidden state of the latent token after processing, and $f_{\text{teacher}} \in \mathbb{R}^d$ is the global CLS (classification) feature from the larger teacher ViT, both normalized to unit length before comparison.

What it computes: the negative cosine similarity between two vectors in the same embedding space. Cosine similarity ranges from -1 (opposite directions) to +1 (identical directions). The negative sign converts this into a minimization objective: the loss is minimized (-1) when the two vectors point in exactly the same direction, and maximized (+1) when they point in exactly opposite directions. Because both vectors are L2-normalized before the dot product, the loss depends only on the angle between them, not on their magnitudes.

Why this form: cosine similarity is preferred over L2 distance for representation alignment because it is scale-invariant — it only cares about the direction of the feature vector, not its magnitude. This is important because the absolute magnitudes of neural network activations can vary substantially across layers, models, and training stages, and penalizing magnitude differences would create a "tug-of-war" with the primary language modeling loss. MSE loss ( $\|f_{\text{latent}} - f_{\text{teacher}}\|^2$ ) would penalize both angular and magnitude differences, which could cause the model to sacrifice representational quality to match arbitrary magnitude scales. The cosine loss isolates the semantically meaningful information (direction in feature space) from the arbitrary scaling factor.

The teacher ViT's CLS feature captures global image-level semantics — what objects are present, the overall scene type, coarse spatial layout — but not the fine-grained per-patch details. By forcing the latent token to match this global representation, the model is incentivized to aggregate information from all visual patches into a single compact vector that summarizes the image content. This aggregated representation is then available to text tokens during generation, providing a "gist" of the visual scene that can guide language production without requiring the text decoder to attend over all visual patches for every token.

Empirical effect. The paper provides attention visualizations in Figure 12 that demonstrate the latent token's behavior. The "Attention on Visual Tokens" heatmaps show that the latent token's attention is concentrated on "salient objects, highly specific object parts (such as the right end of the potato chip can or the handles of the drawers), and key spatial regions relevant to the scene context." Concurrently, "language attention weights are strongly concentrated on core semantic entities, state descriptions, spatial relationships, and action-oriented instructions." This dual attention pattern — the latent token extracting task-relevant visual features, and text tokens attending to the latent token for those features — is the mechanism by which the latent token bridges modalities.

The paper notes this is "beneficial for improving the capabilities of small VLMs" specifically (Section 2.3), suggesting that for larger models with more parameters, the cross-modal alignment may emerge naturally from the large-scale training without explicit architectural intervention. For the 2B model, where capacity is constrained, the explicit bridge provided by the latent token helps overcome the modality gap that would otherwise limit performance.

Pre-Training Data and Recipe

The model's capabilities are fundamentally shaped by the data it sees during pre-training, and the paper devotes extensive detail (Section 3.1, spanning approximately five pages) to describing the construction of a diverse, embodied-centric training corpus. The guiding principle is early and deep integration of physical-world understanding: rather than pre-training on general web data and then fine-tuning on embodied data (the standard transfer learning approach), HY-Embodied-0.5 incorporates embodied and spatial data from the very beginning of multi-modal training.

Data volume and composition. The pre-training corpus comprises over 600B tokens total, divided into two categories (Figure 4):

General understanding data: 389B tokens. This is standard VLM pre-training data covering image captioning, world knowledge, STEM (math, coding, science), document understanding, OCR, charts, logical reasoning, multi-round dialogue, multi-image contexts, complex instruction following, and GUI navigation. The paper groups these into "general semantics," "STEM proficiency," "fine-grained visual parsing," "complex problem-solving," and "agentic operations" categories.
Embodied and perception data: 236B tokens. This is the distinguishing feature of the training corpus. Within this category, "spatial and robotics data account for 43%, and visual perception data make up the rest" (Section 3.2). The embodied/perception data encompasses five sub-categories:

Visual Perception Data (Section 3.1.1)

Omni-Detection (62M samples). A unified 2D and 3D object detection dataset built from OpenImages, Objects365, RefCOCO, and SA-1B. For samples with high-quality human annotations, existing labels are converted to a unified format. For unlabeled or low-quality data, an automated pipeline is used: a VLM identifies objects, SAM (Carion et al., 2025) generates segmentations, VLM grounding determines coordinates, and a stronger VLM teacher verifies annotation accuracy. All coordinates are normalized to the range [0, 1000] and represented in a fixed output format (e.g., <box>(x1,y1),(x2,y2)</box>). Tasks include object tagging and prediction of 2D/3D bounding boxes.

Depth Estimation (approximately 36M samples). Both absolute metric depth and relative depth data, sourced from "large-scale indoor and outdoor 3D datasets, alongside autonomous driving corpora." The data is constructed as question-answering pairs based on specific image coordinates: the model is given a point (via coordinates or visual markers) and must predict its distance from the camera. A point-sampling strategy explicitly excludes "pixels located on object boundaries, at infinity, or within physically inconsistent regions" to ensure clean training signals. To enable fusion across datasets with different camera intrinsics, "camera focal lengths [are normalized] across all images, thereby standardizing the scale of depth measurements."

Segmentation (approximately 5M samples). Semantic segmentation data sourced from SA-1B, with filtering to remove "excessively small, disproportionately large, and highly fragmented object masks." Following the methodology of PaliGemma (Beyer et al., 2024), the tokenizer vocabulary is expanded to encode binary mask matrices as sequences of discrete tokens, which the model is trained to predict autoregressively. This converts segmentation — typically a dense prediction task — into a sequence generation task compatible with the VLM's language modeling objective.

Pointing and Counting (approximately 11M samples). A specialized dataset targeting VLM weaknesses in object enumeration and spatial localization. Point annotations are sourced from Pixmo-Points (Deitke et al., 2025), with deliberate filtering to select "scenes containing a high density of target objects" to ensure task difficulty is sufficient. The tasks require the model to count instances of specific object categories or to point to specific objects by outputting normalized coordinates.

Embodied-Centric Data (Section 3.1.2)

The embodied data is organized into a three-tier hierarchy:

Grounding. Visual grounding data for physical manipulation scenarios, incorporating point-level localization, bounding box prediction, and referring expression comprehension. Data sources include Molmo, RoboPoint, and RefSpatial, plus in-house annotations. The filtering process "isolates elements critical to embodied operations, such as target interactive objects and the robotic manipulators themselves."

Affordance. Affordance prediction data that integrates visual grounding with user instructions. Data sources include RoboAfford and ShareRobot, with additional data generated by repurposing existing grounding annotations: a VLM generates contextually appropriate user instructions paired with the original spatial grounding annotations.

Trajectory. Trajectory prediction data for internal planning, sourced from MolmoAct, ShareRobot, FSD, and extracted from large-scale manipulation video clips (RoboMind, Open X-Embodiment, DROID) using the tracking model cotracker3. For extracted trajectories, the first frame of each video is retained as visual input, and the extracted waypoint sequences are downsampled to a maximum of 15 waypoints and visually plotted onto the image. A stronger VLM judge evaluates trajectory accuracy for filtering.

Understanding. Embodied understanding QA pairs from Robo2VLM, RoboVQA, RoboRefit, and RoboInter-VQA, filtered based on "data quality and annotation accuracy."

Planning. Embodied planning data constructed by using a VLM to annotate primary tasks in robotic manipulation video clips, temporally segmenting the videos, and extracting ground-truth labels for subsequent actions. The segments are formatted as query-response pairs prompting the model to predict future action sequences, with explicit task constraints defined in user instructions. Open-source planning QA pairs from RoboVQA and RoboInter supplement the generated data.

Reasoning. An in-house complex reasoning dataset "situated in real-world embodied environments" targeting long-horizon reasoning scenarios: action sequencing, multi-image action comprehension, future state prediction, visual puzzle resolution, and intuitive physics reasoning.

Spatial-Centric Data (Section 3.1.3)

Spatial-centric data focuses on 3D environment understanding independent of agent interaction, sourced from ScanNet, ScanNet++, ARKitScenes, and self-collected data. It is organized into five types:

Correspondence. Cross-frame point matching and 2D-3D instance mapping. The data generation pipeline uses posed RGB-D sequences where camera intrinsics and extrinsics enable precise coordinate system projection. It computes visibility information for sampled frames and generates QA pairs querying point correspondences using explicit coordinates or visual dot markers.

Geometry. Depth estimation (predicting depth at indicated points) and depth comparison (determining which of two points is closer to the camera). Point pairs are sampled with "sufficient depth disparity (typically >0.3m) to ensure unambiguous annotations."

Configuration. Spatial arrangement tasks including object counting, relative distance identification (which of several objects is closest to a reference), relative direction determination (left, right, front, back relative to an observer), and distance ranking (ordering objects by proximity). All derived from 3D bounding box annotations and instance segmentation. For direction tasks, "angles on the 2D ground plane between the observer's forward vector and the query vector [are computed], with filtering to exclude ambiguous cases where angular differences are insufficient."

Measurement. Precise metric estimation tasks: object size (longest dimension in cm), absolute distance (Euclidean distance between 3D bounding boxes in meters), and room size (total floor area in square meters). Generated from scenes with calibrated 3D reconstructions. Filtering excludes "trivially close object pairs (<0.2m) for distance tasks" and restricts size queries to objects with unique instances.

Dynamics. Motion and temporal change data including camera ego-motion and object movement. Camera dynamics data describes spatial transformations between frames (relative rotation and translation). Object dynamics leverages "dense 3D point tracks across video sequences, with annotations containing 3D coordinates, visibility flags, and camera extrinsics."

General Understanding Data (Section 3.1.4)

A substantial volume of in-house VLM data covering: general semantics (image captioning, world knowledge), STEM (math, coding, scientific reasoning), fine-grained visual parsing (document understanding, charts, OCR), complex problem-solving (logical reasoning, multi-round dialogues, multi-image contexts), and agentic operations (GUI navigation). These are "jointly trained with the specialized embodied corpora" throughout both training stages.

Two-stage training schedule (Section 3.2). The pre-training is structured into two sequential stages:

Stage 1: Large-scale pre-training. The model is trained on the full >600B token corpus comprising 389B general + 236B embodied/perception tokens. The training configuration is: base learning rate 5e-5, ViT learning rate 5e-6, weight decay 1e-4, global batch size 256. Training samples are "packed to a maximum context length of 32k tokens based on the length of each question-answering pair." All parameters are trainable: the ViT, the MoT module, and the latent visual tokens. However, "the gradients for the ViT and visual tokens are updated once every five steps" — a gradient accumulation strategy that effectively gives the ViT a 5× smaller learning rate than the LLM backbone, likely to prevent the randomly initialized visual encoder from destabilizing the pre-trained language model early in training.

Stage 2: Embodied-Spatial Mid-training. A second stage introduces "higher-quality and more complex embodied and spatial data, encompassing approximately 30 million instances" (though the paper also states this stage uses "approximately 25M data samples" — likely the 30M includes general data). The mixing ratio is "general understanding, embodied, and spatial data at a ratio of 12:5:3" (Section 3.2). All prompt templates and coordinate formats are unified during this stage. The training retains the sequence packing method and base learning rate from pre-training, but introduces a cosine learning rate decay. All ViT parameters are frozen during this stage — only the MoT module parameters are updated — which prevents the visual encoder from overfitting to the specific distributions of embodied data at the expense of general visual capability.

A key design choice during mid-training is variant-specific data strategy: "for the MoT-2B model, we utilize a mixture of long and short reasoning chains, differentiated via \think and \no think tokens following Qwen3-VL; conversely, for the MoE-32B model, we exclusively employ short-chain data to concentrate on embodied fine-tuning" (Section 3.2). This means the 2B model learns to toggle between thinking mode (producing explicit reasoning chains) and non-thinking mode (direct answers) during mid-training, while the 32B model — which will develop its thinking capability through post-training — only sees short-chain data at this stage to focus on acquiring embodied knowledge.

Composite Pre-Training Loss

During the large-scale pre-training stage, the model is optimized with a composite loss combining three objectives (Section 3.3). During mid-training and all fine-tuning stages, only the standard language modeling loss is used.

LLM loss ( $\mathcal{L}_{\text{llm}}$ ). The standard autoregressive language modeling loss — cross-entropy between predicted and actual next tokens across the text sequence. This is the primary training objective and is always active.

Vision loss ( $\mathcal{L}_{\text{vision}}$ ). The visual next-code prediction loss described in Section 2.2:

$\mathcal{L}_{\text{vision}} = -\frac{1}{N_v} \sum_{i=1}^{N_v} \log p_i(z_i)$

where $N_v$ is the number of visual tokens, $p_i$ is the predicted probability distribution over the 2,048-entry codebook for the $i$ -th visual token, and $z_i$ is the target discrete code from the teacher ViT.

What it computes: the average negative log-likelihood of the correct discrete code across all visual token positions. The loss is applied to the vision branch of the MoT (the vision-specific MLP prediction head).

Why this form: as discussed above, categorical cross-entropy is the appropriate objective for discrete classification. The averaging over all visual tokens ( $\frac{1}{N_v}$ ) ensures the loss scale is independent of image resolution, which varies across training samples due to native-resolution processing.

Global loss ( $\mathcal{L}_{\text{global}}$ ). The latent token alignment loss:

$\mathcal{L}_{\text{global}} = -\frac{f_{\text{latent}}^{\top} f_{\text{teacher}}}{\|f_{\text{latent}}\| \|f_{\text{teacher}}\|}$

Total pre-training loss:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{llm}} + \mathcal{L}_{\text{vision}} + \mathcal{L}_{\text{global}}$

The three losses are simply summed — there are no explicit weighting coefficients mentioned in the paper. This implies either that the losses are naturally of comparable scale (due to the averaging in $\mathcal{L}_{\text{vision}}$ and the [-1, 1] range of cosine similarity in $\mathcal{L}_{\text{global}}$ ), or that implicit weighting through gradient magnitudes is considered sufficient. The paper does not discuss whether any tuning of relative loss weights was performed.

Transition to single-loss fine-tuning. In mid-training and all fine-tuning stages (SFT, RL, RFT, distillation), "we discard the vision and global supervision signals, exclusively optimizing the standard autoregressive language loss ( $\mathcal{L}_{\text{llm}}$ )" (Section 3.3). This makes sense: the auxiliary losses serve to establish good visual representations and cross-modal alignment during early training, but once these are in place, further optimization should focus entirely on the downstream task performance that the language modeling loss directly measures.

Post-Training: Supervised Fine-Tuning (SFT)

The post-training pipeline begins with a supervised fine-tuning stage designed to establish the model's long-chain reasoning capability before reinforcement learning begins (Section 4.1).

Data construction. The SFT data is a subset of "high-complexity, multi-step problems from the aforementioned spatial, embodied, and general data sources, together with more in-house reasoning data" (Section 4.1.1). The key data construction mechanism is a human-model collaborative pipeline for generating Chain-of-Thought (CoT) trajectories: humans and models work together to produce step-by-step reasoning traces for these complex problems. The generated CoTs are then evaluated by an LLM across multiple quality dimensions: "reasoning quality, logical correctness, and sequence repetition." Additionally, the exact match accuracy of the final deduced answers is verified against ground truth. This filtering process yields "approximately 100k cold-start CoT instances," which are used to train both the MoT-2B and MoE-A32B variants.

The term "cold-start" here refers to the fact that this data is generated before any RL has been performed — it serves as the initial reasoning data that "starts" the model's ability to produce long thinking chains. The human-model collaborative pipeline is necessary because manually writing 100k high-quality CoT examples would be prohibitively expensive, but purely model-generated CoTs often contain reasoning errors that would teach the model incorrect patterns. The collaborative approach leverages human expertise for correctness while using model generation for scale.

Training recipe. The cold-start SFT uses standard cross-entropy loss on the language tokens, but with an important change from pre-training: "we explicitly disable sequence packing. Each training sample is processed individually to isolate and emphasize the independent reasoning chain of each data entry" (Section 4.1.2). Sequence packing — concatenating multiple training examples into a single long sequence to maximize GPU utilization — is efficient but can cause the model to confuse boundaries between different reasoning problems. For SFT specifically focused on developing clean, structured reasoning, isolating each example ensures the model learns to produce coherent chains without cross-contamination. The base learning rate is maintained at 5e-5.

Post-Training: Reinforcement Learning with Task-Aware Rewards

The reinforcement learning stage is where the model actively improves its embodied reasoning through trial and error, guided by carefully designed reward functions that can evaluate the quality of diverse output types (Section 4.2). The paper uses Group Relative Policy Optimization (GRPO), an RL algorithm that normalizes rewards within groups of sampled responses to create relative advantage signals.

Data construction for RL. Unlike SFT, which uses a fixed dataset, RL training data is dynamically constructed based on the model's current capabilities (Section 4.2.1). The procedure works as follows:

A large candidate pool of diverse embodied tasks is maintained, covering perception, prediction, interaction, and planning.
In each RL round, the latest model performs multi-sample evaluation on this pool.
Samples solved correctly in all attempts are discarded as "overly easy" — they provide no learning signal because the model already performs perfectly on them.
Samples that fail in all attempts are removed as "overly difficult" — they are beyond the model's current capability frontier and cannot provide useful gradient signals.
Only samples with partial success (some attempts correct, some incorrect) are retained, as these lie "near the model's current capability frontier and typically provide the most informative learning signals for policy improvement."

This is a form of automatic curriculum learning: as the model improves, the set of problems at its capability frontier shifts, and the data selection procedure automatically tracks this frontier. To prevent the RL from over-specializing on a narrow capability, the selected data is balanced across different embodied capability dimensions. Each RL stage trains on "a newly constructed set of 50K samples."

Task-aware reward design. The paper identifies a core challenge for embodied RL: "the target outputs are highly heterogeneous, spanning geometric grounding, trajectory prediction, discrete decisions, continuous estimation, and open-ended reasoning. A single uniform reward is therefore inadequate" (Section 4.2.2). The solution is a taxonomy of reward functions organized by output structure (Figure 6), following the principle that "reward structure should match output structure."

The general form is:

$r = R_t(y, y^\star) \in [0, 1]$

where $R_t$ is the task-specific reward function selected based on the structure of the target output, $y$ is the model's generated response, and $y^\star$ is the ground-truth target. All rewards are normalized to the range [0, 1].

What it computes: a scalar score between 0 and 1 indicating how well the model's output matches the ground truth for a particular task type. The reward is always non-negative and bounded, with 1 indicating perfect correspondence.

Why this form: the [0, 1] bounded range is critical for GRPO's relative normalization (discussed below). If different tasks produced rewards on wildly different scales (e.g., one task giving rewards in [0, 1] and another in [0, 100]), the group-relative normalization would be dominated by the high-variance task, and the model would effectively ignore the low-variance task regardless of its actual difficulty or importance. Normalizing all rewards to a common scale ensures that improvement on any task type contributes proportionally to the policy gradient.

The four reward categories are:

1. Grounding-Based Rewards (for spatial localization). These handle outputs where the target is a geometric structure — bounding boxes, points, or sets of points. The rewards use continuous similarity measures rather than binary exact-match:

Bounding boxes: Multi-IoU reward. Intersection over Union (IoU) measures the overlap between predicted and ground-truth boxes. Multi-IoU likely refers to Hungarian-matched IoU when multiple boxes are predicted — each predicted box is matched to the most overlapping ground-truth box, and the average IoU across matched pairs is computed.
Points: Normalized L2 distance reward. Given predicted point coordinates and ground-truth coordinates, the Euclidean distance is computed and normalized by the image diagonal to produce a value in [0, 1], then converted to a reward via $r = 1 - \text{normalized\_distance}$ .

These continuous rewards are important because "partial spatial or temporal correctness should be distinguished from complete failure" — a bounding box with 0.85 IoU is clearly much better than one with 0.1 IoU, and treating both as equally wrong would destroy the gradient signal needed for fine-grained coordinate prediction.

2. Regression-Based Rewards (for numerical estimation). These handle outputs that are continuous scalar values — object counts, distances, sizes:

Counting: Relative count error reward. Given predicted count $c$ and true count $c^\star$ , the reward decays with the relative error: something like $r = \max(0, 1 - |c - c^\star|/c^\star)$ or an exponential decay form.
Distance: Relative exponential error reward. Given predicted distance $d$ and true distance $d^\star$ , the reward is $r = \exp(-\alpha \cdot |d - d^\star|/d^\star)$ , where $\alpha$ controls the penalty steepness. The exponential form provides more informative gradients than a hard threshold because it smoothly decays as predictions get worse rather than abruptly dropping to zero.

Why exponential decay for distances? The exponential form has the property that small relative errors (e.g., predicting 1.1m when the true distance is 1.0m) still receive high rewards (e.g., $e^{-0.1} \approx 0.90$ ), encouraging the model to continue refining even when already close. A step function that gives reward 1 only for exact matches would provide no gradient to guide the model from "close" to "exact." The exponential also naturally handles the fact that distance estimation errors in embodied tasks follow a log-scale intuition — being off by 10cm at 1m distance is much more significant than being off by 10cm at 10m distance.

3. Trajectory-Based Rewards (for motion and planning). These handle sequence outputs where temporal ordering matters:

Trajectory waypoints: DTW (Dynamic Time Warping) + Goal Hybrid reward. DTW computes the minimal-cost alignment between two sequences, allowing for temporal stretching/compression. The DTW distance between predicted and ground-truth waypoint sequences is computed and normalized. An additional goal reward term checks whether the final predicted waypoint reaches the target location (endpoint consistency), and the two are combined — likely as a weighted sum.
Maze paths: Fréchet distance reward. Fréchet distance measures the similarity between two curves taking into account the location and ordering of points along the curves. It is more appropriate than DTW for continuous paths where the "flow" of the trajectory matters.

Why DTW for waypoints? Waypoint sequences may have different lengths (the model might predict 8 waypoints while the ground truth has 12), and the exact timing of each waypoint matters less than the overall path shape. DTW handles both variable-length sequences and temporal alignment, making it more appropriate than frame-by-frame L2 distance which would penalize correct paths taken at slightly different speeds.

4. Textual-Based Rewards (for general reasoning). These handle outputs where the target is natural language:

Fixed-answer tasks (multiple choice, binary judgment, structured answers): Mixed matching reward. For multiple choice, exact match on the option letter. For structured outputs where partial credit is meaningful, sequence similarity measures like normalized longest common subsequence (LCS) are used.
Open-ended answers: LLM-as-a-Judge reward. A separate, stronger LLM evaluates the model's response against the reference answer given the input:

$r_{\text{free}} = J(q, y, y^\star)$

where $q$ is the input question, $y$ is the model's response, $y^\star$ is the reference answer, and $J$ is the judge model (likely a larger, more capable LLM prompted to assess answer quality on a rubric).

Why an LLM judge as fallback? The paper explicitly states that deterministic rewards are preferred whenever possible because they are cheaper, faster, and more consistent than LLM judges. The LLM judge is used only when "correctness cannot be robustly determined by rules alone," typically for open-ended reasoning where multiple valid phrasings exist for the same answer. This hybrid approach — deterministic where possible, learned where necessary — balances cost and reliability.

GRPO training objective. The RL algorithm is GRPO (Group Relative Policy Optimization), which the paper uses with a group size $G = 16$ (Section 4.2.3). For each multi-modal input $x = (I, Q)$ (image $I$ and question $Q$ ), the current policy $\pi_{\theta_{\text{old}}}$ samples a group of $G = 16$ responses, and each receives a reward from the task-aware reward function, producing a set of rewards $\{r_1, \ldots, r_G\}$ .

Group-relative advantages are computed by normalizing within each group:

$A_i = \frac{r_i - \mu(\mathbf{r})}{\sigma(\mathbf{r})}, \quad \mathbf{r} = \{r_1, \dots, r_G\}$

where $\mu(\mathbf{r})$ is the mean reward of the group, $\sigma(\mathbf{r})$ is the standard deviation, and $A_i$ is the advantage for the $i$ -th response.

What it computes: for each response in the group, its advantage is how many standard deviations above or below the group mean its reward lies. A response with reward exactly equal to the group mean gets advantage 0. A response one standard deviation above the mean gets advantage +1. A response one standard deviation below gets advantage -1.

Why this form: this relative normalization solves the problem of heterogeneous reward scales across tasks. Since all rewards are in [0, 1] but different tasks may have different typical reward distributions (e.g., a bounding box task where even good predictions rarely exceed 0.9 IoU vs. a multiple-choice task where correct answers always get 1.0), the raw reward magnitudes are not comparable across tasks. By normalizing within each group, the advantage reflects "how much better or worse than average was this response for this specific input," which is comparable across all task types. This is "particularly suitable for embodied RL, where tasks are highly heterogeneous and raw reward scales are not directly comparable across samples" (Section 4.2.3).

The policy is then updated using a clipped policy-ratio objective, identical in structure to PPO (Schulman et al., 2017) but applied per-token:

$\mathcal{L}_{\mathrm{RL}}(x) = -\frac{1}{\sum_{i=1}^{G}|y_i|} \sum_{i=1}^{G}\sum_{t=1}^{|y_i|} \min\!\Big( \rho_{i,t}A_i,\, \mathrm{clip}(\rho_{i,t}, 1-\epsilon_{\mathrm{low}}, 1+\epsilon_{\mathrm{high}})A_i \Big)$

where $|y_i|$ is the length (number of tokens) of the $i$ -th response, $\rho_{i,t}$ is the probability ratio at token $t$ of response $i$ , and $\epsilon_{\text{low}}, \epsilon_{\text{high}}$ are the clipping thresholds.

The probability ratio is:

$\rho_{i,t} = \frac{\pi_{\theta}(y_{i,t}\mid x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x, y_{i,<t})}$

where $\pi_{\theta}(y_{i,t}\mid x, y_{i,<t})$ is the probability the current policy assigns to token $y_{i,t}$ given the input and previous tokens, and $\pi_{\theta_{\text{old}}}$ is the same for the old policy (the one used to sample the responses).

What it computes: the clipped objective for policy gradient. For each token in each response, the model computes the ratio of probabilities under the new vs. old policy. When a response has positive advantage ( $A_i > 0$ ), the objective encourages increasing the probability ratio $\rho_{i,t}$ — making the model more likely to produce those tokens. However, the clip prevents the ratio from exceeding $1 + \epsilon_{\text{high}}$ , capping how much the model can change in a single update. When a response has negative advantage ( $A_i < 0$ ), the objective encourages decreasing the ratio, but the clip prevents it from going below $1 - \epsilon_{\text{low}}$ . The $\min$ operator selects the more conservative of the clipped and unclipped objectives — this is the standard PPO clipping mechanism that prevents destructive large policy updates.

The averaging is over all tokens in all responses in the group ( $\sum_{i=1}^{G}|y_i|$ is the total number of tokens across the 16 responses). The negative sign converts the maximization of advantage-weighted probability into a minimization loss.

Why this form: the PPO-style clipped objective is the standard approach for stable policy gradient in language models. Without clipping, a single update could dramatically increase the probability of a high-advantage response, potentially causing the model to collapse to a deterministic policy that no longer explores. The clipping ensures trust-region-like behavior without the computational expense of computing exact KL divergence constraints. The asymmetric clipping range $[\epsilon_{\text{low}}, \epsilon_{\text{high}}]$ mentioned in the paper (effective range [0.8, 1.35]) is "more stable than a symmetric clipping rule in long-chain multimodal RL" — the intuition is that in long reasoning chains, the probability ratios can drift substantially due to the product of many per-token probabilities, and allowing more room for positive updates (1.35 vs. the typical 1.2) while being more conservative on negative updates (0.8 vs. the typical 0.8) helps the model learn new behaviors without catastrophic forgetting.

Training configuration. The RL training uses: maximum prompt length 16,384 tokens, maximum response length 16,384 tokens, sampling temperature 1.0, top-p = 1.0, top-k = -1 (disabled). Training batch size is 128, learning rate is $8 \times 10^{-7}$ , and each RL stage runs for 5 epochs. Gradient checkpointing and parameter/optimizer offloading are enabled for memory efficiency. Groups with zero reward variance are masked (they provide no meaningful relative learning signal). Overlong and repetitive responses are filtered via quality control, and "additional length-related shaping [is applied] for selected subjective tasks" — likely a penalty on excessively long reasoning chains to prevent the model from learning to "pad" its responses.

Evolving Deep Thinking with Iterative Training (RL → RFT Loop)

The paper identifies a limitation of RL alone: "while RL directly improves task reward, it does not necessarily guarantee high-quality reasoning traces. In embodied tasks, correct answers may arise from very different internal processes, ranging from coherent spatial reasoning to unstable shortcuts" (Section 4.3). To address this, the post-training pipeline alternates between RL and Rejection Sampling Fine-Tuning (RFT), creating an iterative self-evolving training paradigm.

Rejection Sampling Fine-Tuning procedure. Starting from the latest model checkpoint after RL, the process is:

Multi-sample rollout: the model generates multiple responses for each question in a curated data pool.
Offline evaluation: the sampled responses are scored using criteria aligned with the RL reward functions. Only samples that are "solved correctly in some, but not all, rollouts" are retained — the same capability-frontier filtering used in RL data construction. This removes both saturated (always correct) and impossible (never correct) examples.
Reasoning quality filtering: among the retained correct samples, a "stronger teacher model" scores the quality of the reasoning traces. Only traces whose "thinking quality exceeds a predefined threshold" are kept.
SFT on filtered traces: the surviving high-quality reasoning traces are used as supervised training data for a standard cross-entropy fine-tuning stage.

In practice, this pipeline filters "approximately 1M candidate examples into around 300K high-quality traces for the subsequent SFT stage" — a roughly 3:1 filtering ratio, indicating that the model produces many more correct-but-poorly-reasoned answers than correct-and-well-reasoned ones.

Complementary roles of RL and RFT. The paper characterizes the relationship as: "RL expands the capability frontier, while RFT consolidates the best newly discovered reasoning patterns into more stable behavior" (Section 4.3). RL is good at exploration — through reward-driven search over 16 responses per input, it helps the model discover new ways to solve problems that it wouldn't find through greedy or temperature-1 sampling alone. However, the supervision from RL is indirect (a scalar reward that says "this was good" but doesn't say why) and relative (the advantage depends on the group, not an absolute standard). RFT converts these discoveries into explicit positive supervision: it selects the specific reasoning traces that led to correct answers with high-quality thinking, and trains the model to reproduce them through standard supervised learning.

Iterative alternation. The two stages are alternated throughout post-training: SFT (cold-start) → RL (round 1) → RFT (round 1) → RL (round 2) → RFT (round 2) → ... The paper does not specify the exact number of iterations, but the Figure 5 pipeline diagram suggests the cycle continues until performance saturates. This iterative process "gradually transforms occasional success into reliable capability," which is particularly important for embodied tasks where the penalty for failure (a robot dropping an object, colliding with furniture) is much higher than in text-only domains.

Large-to-Small On-Policy Distillation

The final stage of the post-training pipeline transfers the reasoning capabilities cultivated in the large MoE-A32B model into the compact MoT-2B model for edge deployment (Section 4.4). The paper uses On-Policy Distillation (OPD), which differs fundamentally from standard knowledge distillation.

Standard (offline) distillation works by: (1) the teacher generates responses to a set of prompts, (2) the student is trained to maximize the probability of the teacher's responses (either through next-token cross-entropy or by matching output distributions). The limitation is that the student only ever sees the teacher's "correct" trajectories — it never experiences its own mistakes and therefore never learns to recover from the distribution of errors it will actually make during inference.

On-Policy Distillation addresses this by having the student generate its own responses first, then using the teacher to provide token-level supervision on those self-generated sequences. Specifically:

The student rolls out its own response:

$y = (y_1, \dots, y_T) \sim \pi_s(\cdot \mid x)$

where $\pi_s$ is the student's policy and $x$ is the input.

The teacher is applied under teacher forcing on the student-generated prefixes: for each position $t$ , the teacher computes its next-token distribution $\pi_t(\cdot \mid x, y_{<t})$ given the same input and the student's own previous tokens.
The student is optimized to minimize the KL divergence between the teacher's and student's next-token distributions at each position:

$\mathcal{L}_{\mathrm{OPD}} = \mathbb{E}_{x,\,y\sim \pi_s(\cdot \mid x)} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} \mathrm{KL}\!\left( \pi_t(\cdot \mid x, y_{<t}) \,\|\, \pi_s(\cdot \mid x, y_{<t}) \right) \right]$

where $\mathrm{KL}(P \| Q) = \sum_v P(v) \log \frac{P(v)}{Q(v)}$ is the Kullback-Leibler divergence, measuring how much information is lost when using the student's distribution $Q$ to approximate the teacher's distribution $P$ . The expectation is over the input distribution and the student's own sampling.

What it computes: for each student-generated token sequence, at each position, the KL divergence between the teacher's next-token distribution (conditioned on the student's prefix) and the student's next-token distribution. These per-token divergences are averaged over the sequence length and then over the expectation of student-generated sequences. A KL divergence of 0 means the student exactly matches the teacher's distribution. Larger values mean the student's distribution is more different.

Why this form: KL divergence is the standard objective for distribution matching because it is the expected log-likelihood ratio — minimizing $\mathrm{KL}(P \| Q)$ is equivalent to maximizing the expected log-likelihood of samples from $P$ under $Q$ . Cross-entropy (the typical distillation loss) is $\mathrm{KL}(P \| Q) + H(P)$ , where $H(P)$ is the entropy of the teacher. Since $H(P)$ does not depend on the student's parameters, minimizing cross-entropy and minimizing KL divergence are equivalent — but the KL formulation makes explicit that the goal is distribution matching, not just mode-seeking.

The key advantage of OPD over offline distillation is the on-policy sampling: the student learns from the teacher precisely at the states induced by its own policy, "where its errors actually occur" (Section 4.4). If the student tends to produce ungrammatical continuations at a certain point in a reasoning chain, the teacher provides corrective signal at exactly that point, teaching the student what the correct continuation should have been. In offline distillation, the student might never encounter those error states because it only sees the teacher's clean trajectories, so it never learns to correct its characteristic mistakes.

The paper argues this is particularly important for embodied reasoning because "the capabilities acquired by the large model through RL and RFT are distributed across the entire reasoning process rather than concentrated only in the final answer." A standard distillation that only matches final answers would lose the intermediate reasoning steps — the spatial analysis, the self-correction, the step-by-step verification — that constitute the actual embodied reasoning capability.

Integration with the post-training pipeline. OPD serves as the final stage: "RL expands the capability frontier of the large model, RFT consolidates newly discovered high-quality reasoning traces, and OPD then transfers these refined behaviors into the compact model" (Section 4.4). This three-stage pipeline — explore (RL), consolidate (RFT), transfer (OPD) — is the paper's meta-recipe for producing compact models with strong reasoning capabilities.

4. Key Insights and Innovations

Innovation 1: Embodied VLM Competence Requires Joint Optimization Across Architecture, Data, and Training — Not Just One Lever

The paper's most fundamental intellectual contribution is not any single architectural trick or training algorithm, but the demonstration that embodied intelligence for VLMs emerges from the interaction of three design axes that prior work typically optimized in isolation: modality-adaptive computation (architecture), massive embodied-centric pre-training (data), and iterative reasoning cultivation (training). The field's prior approach to building embodied VLMs had been to take a general-purpose VLM — trained on web-scale image-text data with shared transformer parameters — and either fine-tune it on a small amount of embodied data (the transfer-learning approach) or augment it with specialized perception modules (the modular-robotics approach). The paper argues, through its systematic engineering, that this piecemeal strategy fundamentally underperforms because the deficiencies are mutually reinforcing: a shared transformer backbone limits visual representational capacity precisely when embodied data demands the most from it, and standard SFT on embodied data teaches correct answers but not the reasoning processes that generalize.

What makes this insight non-obvious is that each of these three axes has been individually explored in prior work — MoT architectures exist (Liang et al., 2024), large-scale embodied datasets exist (Open X-Embodiment), and iterative RL-RFT pipelines exist (DeepSeek-R1) — but no prior system had combined all three with the specific adaptations needed for embodied perception and reasoning. The paper's contribution is the integration thesis: the MoT architecture's value is amplified by the embodied pre-training data, which in turn provides the perceptual foundation that makes the post-training reasoning pipeline effective, and the distillation stage closes the loop by making the resulting capability deployable. This is a systems-level conceptual contribution, not a point improvement, and it changes how practitioners should think about building embodied models — not as a sequence of independent upgrades, but as a co-design problem where architecture choices determine which data distributions are learnable, and data choices determine which reasoning capabilities can be cultivated.

Evidence for this integration effect is distributed across the paper rather than localized to a single figure: the MoT architecture accelerates convergence during pre-training (Figure 11a), the embodied data composition directly enables the strong spatial benchmark performance (Table 1, with the 2B model outperforming 4B and 7B competitors on 16/22 benchmarks), and the iterative post-training pipeline produces the self-correcting reasoning chains visible in Figure 10 that neither architecture nor data alone could generate.

Innovation 2: Difficulty-Aware and Structure-Aware Reward Design as the Enabler of Embodied RL

Prior work applying reinforcement learning to language models — from InstructGPT to DeepSeek-R1 — has predominantly operated in domains where the output is natural language and reward can be approximated by either a learned reward model (trained on human preferences) or a verifiable binary signal (math equality, code execution). The paper identifies that embodied tasks break both of these paradigms simultaneously: outputs span geometric structures (bounding boxes, waypoints), continuous values (distances, sizes), and open-ended reasoning — a heterogeneity that a single reward model or binary verifier cannot handle — and partial correctness matters in ways it does not for text (a bounding box with 0.85 IoU is substantively better than one with 0.1 IoU, not just "both wrong").

The paper's conceptual contribution is the principle that reward structure should match output structure, operationalized as a four-category taxonomy (grounding-based, regression-based, trajectory-based, text-based) with task-specific reward functions that provide dense, continuous supervision whenever the output format admits reliable parsing. This is not merely an engineering convenience — it is a diagnostic move that explains why prior attempts at embodied RL (or RL for VLMs with structured outputs) may have failed: they forced heterogeneous tasks through a homogeneous reward bottleneck. The dense geometric rewards (IoU, normalized L2 distance, DTW, Fréchet distance) are particularly significant because they solve the credit assignment problem for spatial outputs — a model generating coordinate predictions needs gradient signal that distinguishes "close but not exact" from "completely wrong," and binary exact-match provides zero signal for the former.

The paper further identifies that GRPO's group-relative advantage normalization — where rewards are standardized within each batch of 16 sampled responses — is "particularly suitable for embodied RL, where tasks are highly heterogeneous and raw reward scales are not directly comparable across samples" (Section 4.2.3). This is a subtle but important insight about algorithm-task compatibility: GRPO's relative normalization, originally developed for math reasoning where all rewards are 0/1, turns out to be even more valuable in embodied settings where the absolute reward scale varies across task types, because it converts incomparable absolute scores into comparable relative advantages.

The evidence for this innovation is indirect but systematic: the post-training pipeline that uses these rewards produces the reasoning capabilities visualized in Figure 10 and the benchmark performance in Tables 1-2. The paper does not ablate individual reward components, which is a limitation, but the conceptual framework — that embodied RL requires structure-matched, dense rewards combined with relative normalization — is a transferable design principle for future work.

Innovation 3: On-Policy Distillation as the Mechanism for Compressing Embodied Reasoning, Not Just Model Accuracy

Knowledge distillation from large to small models is a mature technique, but the paper identifies a specific failure mode of standard (offline) distillation for reasoning-intensive tasks: when the student only sees the teacher's correct trajectories, it never experiences the distribution of errors it will actually make during inference, and therefore never learns to recover from its characteristic mistakes. This is particularly damaging for embodied reasoning because the capability is distributed across the entire thinking process — the spatial analysis, the self-correction steps, the verification of intermediate conclusions — not concentrated in the final answer token.

The paper's conceptual move is to reframe distillation as behavioral cloning on the student's own induced state distribution rather than on the teacher's state distribution. By having the student generate its own responses first and then applying the teacher under teacher forcing on those self-generated prefixes (the On-Policy Distillation objective in Equation 7), the supervision is targeted precisely at the states where the student's policy diverges from the teacher's — where its errors actually occur. This is a fundamental shift from "teach the student to imitate the teacher's outputs" to "teach the student to correct its own mistakes using the teacher's judgment."

What distinguishes this from prior on-policy distillation work (Agarwal et al., 2024; Thinking Machines Lab, 2025) is its positioning within a broader capability cultivation pipeline. The paper's OPD is not an isolated compression step but the final stage of a three-phase process: RL expands the capability frontier of the large model, RFT consolidates the best discovered reasoning patterns into stable behavior, and OPD transfers these patterns into the compact model. This creates a virtuous cycle where the large model serves as both the exploration engine (through RL) and the quality filter (through RFT) and the behavioral target (through OPD) for the small model. The small model benefits not just from the large model's final answers but from its entire reasoning process, including the self-correction and verification behaviors that distinguish genuine understanding from pattern matching.

The evidence is the strong performance of the MoT-2B model relative to larger competitors (Table 1: outperforming 4B and 7B models on 16/22 benchmarks), which is difficult to achieve through standard distillation alone. However, the paper does not provide direct ablations comparing OPD to offline distillation, so the specific marginal contribution of the on-policy mechanism — versus simply having a strong teacher — cannot be isolated from the reported results. This is a limitation, but the conceptual framework is well-motivated and transferable.

The paper introduces visual latent tokens — learnable embedding vectors appended to visual inputs and supervised to match the global CLS feature of a teacher ViT — as a mechanism for bridging vision and language modalities. While learnable tokens are not novel (they appear in ViT registers, Darcet et al., 2023, and latent thinking work, Zelikman et al., 2024), the paper's conceptual reframing of their role is distinctive: rather than treating them as additional computational capacity (more tokens = more processing) or as scratch space for internal reasoning, the paper positions them as explicit alignment mechanisms that force the model to extract and represent task-relevant visual information in a format directly consumable by the language decoder.

This reframing matters because it explains why these tokens benefit small models specifically. In large models with abundant parameters, cross-modal alignment can emerge organically from scale — the model has enough capacity to learn the mapping from visual patch embeddings to linguistic concepts implicitly through the language modeling objective. In small models with constrained capacity, this implicit alignment competes with other representational demands and often fails, resulting in models that can "see" (extract visual features) but cannot effectively "communicate" what they see to the language decoder (resulting in poor grounding and spatial reasoning). The visual latent token, supervised by an explicit cosine similarity loss to the teacher ViT's global representation, provides a dedicated, low-dimensional bottleneck that relieves this representational competition: the model learns to shunt global image semantics through the latent token, freeing the patch-level visual tokens to focus on fine-grained spatial detail.

The attention visualizations in Figure 12 provide qualitative evidence for this interpretation: the latent token's attention is "precisely localized on salient objects, highly specific object parts, and key spatial regions," while language attention is "strongly concentrated on core semantic entities, state descriptions, spatial relationships, and action-oriented instructions." This dual attention pattern — the latent token selecting task-relevant visual information, and the language decoder attending to the latent token for that information — is the operational mechanism of the bridge. The paper's use of a structured alignment loss (cosine similarity to teacher features) rather than relying on the language modeling loss to shape these tokens is the key design choice that distinguishes this from generic register tokens.

The significance of this innovation is that it provides a lightweight, architecturally simple mechanism for improving small VLM performance without increasing inference cost, which directly addresses the deployment constraints that motivate the 2B variant. It is an incremental rather than fundamental advance — the core idea of supervised alignment tokens builds directly on prior work — but the specific instantiation (cosine similarity to teacher CLS features, integration with MoT architecture, demonstration of cross-modal attention bridging) and the conceptual framing (resolving representational competition in small models) make it a transferable design pattern.

Innovation 5: The Capability-Frontier Curriculum as a Self-Regulating Mechanism for Embodied RL

The paper's dynamic data construction strategy for reinforcement learning — sampling from a candidate pool, discarding samples the model always solves or never solves, and retaining only those with partial success — implements a capability-frontier curriculum that automatically adapts to the model's improving performance. While curriculum learning is well-established, the paper's specific instantiation has a distinctive property: by conditioning retention on partial rather than complete success, the curriculum naturally concentrates training on examples where the model has some competence but not mastery — the regime where RL gradients are most informative.

This matters for embodied tasks specifically because the "difficulty" of a spatial reasoning problem is not an intrinsic property of the problem but a function of the model's current perceptual and reasoning capabilities. A problem that is impossible for an early checkpoint (requiring subtle depth reasoning the model hasn't acquired) may become partially solvable after several RL rounds (the model can now estimate depth but still struggles with occlusion reasoning), and eventually become trivially solvable after RFT consolidates the relevant skills. A static training set would waste compute on problems that are either too easy (no learning signal) or too hard (no successful trajectories to reinforce), and would fail to track the model's evolving capability profile. The dynamic curriculum solves this by design.

The paper's additional balancing step — ensuring the retained samples span diverse embodied capabilities (perception, prediction, interaction, planning) — prevents a known failure mode of adaptive curricula: collapse to a narrow capability where the model happens to be making rapid progress. This is a practical but important refinement that distinguishes the approach from naive "train on the hardest solvable problems" strategies.

The evidence for this innovation's effectiveness is distributed across the post-training pipeline results (the benchmark improvements in Tables 1-2 and reasoning quality in Figure 10), but the paper does not provide a direct ablation comparing dynamic vs. static data selection for RL. This limits the strength of the claim, but the conceptual design — a self-regulating curriculum that tracks the capability frontier and prevents distributional collapse — is well-motivated and addresses a genuine challenge in training embodied models where task heterogeneity would otherwise require extensive manual curriculum design.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans 22 public benchmarks covering three capability axes: visual perception (CV-Bench, DA-2K), embodied understanding (ERQA, EmbSpatial-Bench, RoboBench-MCQ, RoboBench-Planning, RoboSpatial-Home, ShareRobot-Affordance, ShareRobot-Trajectory, Ego-Plan2), and spatial understanding (3DSRBench, All-Angles Bench, MindCube, MMSI-Bench, RefSpatial-Bench, SAT, SIBench-mini, SITE-Bench-Image, SITE-Bench-Video, ViewSpatial, VSIBench, Where2Place). These benchmarks are all publicly available and span embodied question answering, spatial reasoning, multi-view geometry, affordance prediction, trajectory understanding, and task planning. General visual understanding is evaluated separately on seven benchmarks (RealWorldQA, Hallusion-Bench, BLINK, CharXiv-RQ, DocVQA, OCRBench, TextVQA) reported in Figure 7.
Base model(s). Two primary variants are evaluated: HY-Embodied-0.5 MoT-2B (2B activated parameters, 4B total, built on Hunyuan-1.8B with HY-ViT-2.0 400M vision encoder) and HY-Embodied-0.5 MoE-A32B (32B activated parameters, 407B total, using a Mixture-of-Experts architecture with a larger HY-Vision-2.0 A32B vision encoder). The 2B variant targets edge deployment efficiency; the 32B variant targets complex reasoning capability competitive with frontier models.
Metrics. The paper primarily reports micro-average accuracy across all evaluation samples for each benchmark. Several benchmarks use task-specific metrics: 3DSRBench and SAT use circular accuracy; ShareRobot-Affordance uses mIoU (mean Intersection over Union); ShareRobot-Trajectory uses 1-DFD where DFD is Dynamic Fréchet Distance (converted so higher values indicate better performance across all benchmarks). For the MoT-2B comparisons against baselines, results for HY-Embodied-0.5 are reported in thinking mode, while for baseline models the better result between thinking and non-thinking modes is reported — making the comparison conservative with respect to HY-Embodied-0.5 (Section 5.1).
Baselines. For the MoT-2B comparisons (Table 1), baselines include: Qwen3-VL-2B and Qwen3-VL-4B (generalist VLMs; Bai et al., 2025), RoboBrain-2.5 4B (specialist embodied VLM; Tan et al., 2026), and MiMo-Embodied 7B (specialist embodied VLM; Xiaomi Embodied Intelligence Team, 2025). For the MoE-A32B comparisons (Table 2), baselines include: Kimi K2.5 (Kimi Team, 2026), Seed 2.0 (Bytedance Seed, 2025), Gemini 3.0 Pro (Google, 2025), and Qwen 3.5 A17B (Qwen Team, 2026). For general understanding benchmarks (Figure 7), baselines are Qwen3-VL-2B-Thinking and InternVL 3.5-2B. The paper notes that Qwen3.5-VL small models produced "excessively repetitive outputs" in some benchmarks, so Qwen3-VL models are used as the primary Qwen baseline (Section 5.1).
Generation budget / compute accounting. The paper evaluates models in inference mode with thinking enabled (for HY-Embodied-0.5 variants). Generation budgets — such as the number of sampled responses or beam width — are not the primary axis of comparison in this evaluation section, as the focus is on benchmark accuracy at a fixed inference configuration rather than scaling test-time compute. The relevant compute constraint is the model size itself (activated parameters), which is the basis for size-matched comparisons (e.g., 2B vs. 2B/4B/7B competitors).
Cross-validation / statistical protocol. No cross-validation protocol is described for the benchmark evaluation. Results are reported as single-point accuracy values per benchmark. For baseline models where both thinking and non-thinking modes are available, the paper reports the better result (making the comparison conservative for HY-Embodied-0.5). For API-based evaluations (Gemini 3.0 Pro, Seed 2.0), assessments were "conducted via their official APIs under thinking mode" (Section 5.2). The paper does not report confidence intervals, standard deviations, or statistical significance tests for any benchmark result.

Main Quantitative Results

MoT-2B: State-of-the-Art Among Small Embodied VLMs

Table 1 reports the comprehensive benchmark comparison for HY-Embodied-0.5 MoT-2B against four baselines of similar or larger scale. The headline result is that the 2B model achieves the best performance on 16 out of 22 benchmarks and second-best on 4 additional benchmarks, despite having fewer activated parameters than three of the four competitors (Qwen3-VL-4B, RoboBrain-2.5-4B, and MiMo-Embodied-7B all have 4B-7B activated parameters vs. 2B for HY-Embodied-0.5).

Average performance. The paper reports in the abstract that HY-Embodied-0.5 MoT-2B "achieves an average score of 58.0% across all 22 benchmarks, outperforming the generalist VLM Qwen3-VL-4B and the specialist embodied VLM RoboBrain2.5-4B — both of which have larger activated parameters — by 10.2% and 8.6%, respectively." These are percentage-point differences in average accuracy, not relative improvements.

Visual perception benchmarks. On CV-Bench (visual perception), MoT-2B achieves 89.2%, outperforming all baselines (Qwen3-VL-2B: 80.0%, Qwen3-VL-4B: 85.7%, RoboBrain-2.5: 86.9%, MiMo-Embodied: 88.8%). On DA-2K (depth-aware visual QA), MoT-2B achieves 92.3%, with a particularly large margin over the next best baseline (RoboBrain-2.5 at 79.4%) and a 22.8 percentage-point gap over Qwen3-VL-2B (69.5%). These results directly support the claim that the architectural innovations (MoT, visual latent tokens, HY-ViT 2.0) provide strong foundational visual perception.

Embodied understanding benchmarks. MoT-2B shows strong but not universally dominant performance:

ERQA: 54.5% (MoT-2B) vs. 47.3% (Qwen3-VL-4B, best baseline) — a 7.2 point margin.
EmbSpatial-Bench: 82.8% vs. 80.7% (Qwen3-VL-4B) — a narrow 2.1 point margin.
RoboBench-MCQ: 49.2% vs. 45.8% (Qwen3-VL-4B) — a 3.4 point margin.
RoboBench-Planning: 54.2% vs. 58.7% (MiMo-Embodied-7B) — MoT-2B is not best here, trailing by 4.5 points. This is one of only two benchmarks where MoT-2B does not rank first.
RoboSpatial-Home: 55.7% vs. 63.2% (Qwen3-VL-4B) — the second benchmark where MoT-2B is not first, trailing by 7.5 points.
ShareRobot-Affordance: 26.8% vs. 25.5% (tie between Qwen3-VL-4B and RoboBrain-2.5) — the narrowest margin across all benchmarks.
ShareRobot-Trajectory: 73.3% vs. 81.4% (RoboBrain-2.5) — MoT-2B trails by 8.1 points. This is notable because trajectory prediction is a core embodied capability, and the specialist model significantly outperforms here.
Ego-Plan2: 45.5% vs. 52.6% (RoboBrain-2.5) — MoT-2B trails by 7.1 points.

The pattern across embodied benchmarks is nuanced: MoT-2B excels at perception-heavy embodied tasks (ERQA, EmbSpatial-Bench) but shows more competitive or trailing performance on planning-intensive tasks (RoboBench-Planning, Ego-Plan2) and trajectory prediction (ShareRobot-Trajectory). This suggests the model's strength lies more in spatial and perceptual understanding than in action sequence generation — a natural consequence of the VLM foundation that emphasizes perception over action.

Spatial understanding benchmarks. This is where MoT-2B shows its most decisive advantage, ranking first on 9 out of 12 benchmarks:

3DSRBench: 57.0% vs. 44.8% (RoboBrain-2.5, best baseline) — a 12.2 point margin.
All-Angles Bench: 55.1% vs. 49.0% (MiMo-Embodied-7B) — a 6.1 point margin.
MindCube: 66.3% vs. 36.2% (MiMo-Embodied-7B) — a dramatic 30.1 point margin. This is the largest relative advantage on any benchmark and strongly supports the claim of fine-grained spatial reasoning capability.
MMSI-Bench: 33.2% vs. 31.9% (MiMo-Embodied-7B) — narrow margin, and notably low absolute performance across all models on this multi-image spatial intelligence benchmark.
RefSpatial-Bench: 45.8% vs. 56.0% (RoboBrain-2.5) — MoT-2B trails by 10.2 points.
SAT: 76.7% vs. 78.7% (MiMo-Embodied-7B) — narrow 2.0 point deficit.
SIBench-mini: 58.2% vs. 53.1% (MiMo-Embodied-7B) — 5.1 point margin.
SITE-Bench-Image: 62.7% vs. 61.0% (Qwen3-VL-4B) — narrow 1.7 point margin.
SITE-Bench-Video: 63.5% vs. 58.9% (MiMo-Embodied-7B) — 4.6 point margin.
ViewSpatial: 53.1% vs. 41.6% (Qwen3-VL-4B) — 11.5 point margin.
VSIBench: 60.5% vs. 55.2% (Qwen3-VL-4B) — 5.3 point margin.
Where2Place: 68.0% vs. 65.0% (RoboBrain-2.5) — 3.0 point margin.

The spatial understanding results are the strongest evidence for the paper's central claim that HY-Embodied-0.5 has developed "particularly effective fine-grained spatial reasoning ability." The 30-point margin on MindCube — which tests spatial mental modeling from limited views — and the 12-point margin on 3DSRBench directly validate the spatial-centric data construction (Section 3.1.3) and the MoT architecture's visual modeling capacity.

General understanding (Figure 7). On seven general VLM benchmarks, MoT-2B achieves performance "comparable to the size-matched general VLMs." Specifically: RealWorldQA (69.5% vs. 62.0% for Qwen3-VL-2B-Thinking and 66.4% for InternVL3.5-2B), Hallusion-Bench (54.9% vs. 48.6% and 66.3%), BLINK (57.2% vs. 51.3% and 63.1%), CharXiv-RQ (37.1% vs. 31.6% and 35.8%), DocVQA (92.9% vs. 89.4% and 87.5%), OCRBench (79.2% vs. 83.6% and 76.8%), TextVQA (77.1% vs. 76.5% and 72.3%). MoT-2B wins on 4 of 7 benchmarks, ties or is competitive on the others, and notably outperforms both baselines on DocVQA and TextVQA — suggesting that embodied-centric training does not degrade (and may even enhance) fine-grained visual parsing capabilities. This directly supports the paper's "no-compromise" positioning: the model maintains general VLM capability while adding embodied competence.

MoE-A32B: Competitive with Frontier Models

Table 2 reports the comparison of HY-Embodied-0.5 MoE-A32B against four frontier models. The headline result: MoE-A32B achieves an average score of 67.0% across 22 benchmarks, surpassing Gemini 3.0 Pro (63.6%) by 3.4 points, Seed 2.0 (66.2%) by 0.8 points, Qwen 3.5 A17B (66.1%) by 0.9 points, and Kimi K2.5 (61.1%) by 5.9 points. MoE-A32B achieves first place on 7 benchmarks (32% of the total) and second place on 6 benchmarks (27%).

Visual perception. MoE-A32B achieves 88.8% on CV-Bench (vs. 89.0% for Kimi K2.5, best) and 90.2% on DA-2K (vs. 92.3% for Seed 2.0, best) — close to but not exceeding the best frontier models on pure perception.

Embodied understanding. Performance is competitive but not dominant:

ERQA: 62.3% (vs. 65.0% for Gemini 3.0 Pro, best) — trails by 2.7 points.
EmbSpatial-Bench: 84.1% (vs. 84.1% essentially tied with Qwen 3.5 A17B at 83.8% and Gemini 3.0 Pro at 83.6%).
RoboBench-MCQ: 62.8% (vs. 69.2% for Gemini 3.0 Pro, best) — trails by 6.4 points.
RoboBench-Planning: 59.3% (vs. 60.1% for Seed 2.0, best) — trails marginally.
RoboSpatial-Home: 76.6% (vs. 57.1% for Gemini 3.0 Pro, worst among baselines) — leads by a substantial 1.7 points over Qwen 3.5 A17B at 74.9%.
ShareRobot-Affordance: 28.6% (vs. 29.3% for Qwen 3.5 A17B) — trails marginally.
ShareRobot-Trajectory: 76.9% (vs. 73.8% for Qwen 3.5 A17B) — leads by 3.1 points.
Ego-Plan2: 51.4% (vs. 60.0% for Gemini 3.0 Pro) — trails by 8.6 points, the largest deficit on any benchmark.

Spatial understanding. This is again the strongest category:

3DSRBench: 56.6% (vs. 58.3% for Gemini 3.0 Pro) — trails by 1.7 points.
All-Angles Bench: 71.8% (vs. 73.4% for Gemini 3.0 Pro) — trails by 1.6 points.
MindCube: 69.2% (vs. 66.0% for Gemini 3.0 Pro) — leads by 3.2 points.
MMSI-Bench: 39.2% (vs. 48.0% for Gemini 3.0 Pro) — trails by 8.8 points.
RefSpatial-Bench: 57.2% (vs. 72.2% for Seed 2.0) — trails by 15.0 points, the largest deficit.
SAT: 87.3% (vs. 88.0% for Gemini 3.0 Pro) — trails by 0.7 points.
SIBench-mini: 67.3% (vs. 68.0% for Gemini 3.0 Pro) — trails by 0.7 points.
SITE-Bench-Image: 74.7% (vs. 77.1% for Qwen 3.5 A17B) — trails by 2.4 points.
SITE-Bench-Video: 72.5% (vs. 72.3% for Qwen 3.5 A17B) — essentially tied for best.
ViewSpatial: 59.8% (vs. 45.2% for Kimi K2.5, worst) — leads by 7.6 points over Qwen 3.5 A17B at 52.2%.
VSIBench: 68.3% (vs. 54.2% for Kimi K2.5, worst) — leads by 7.2 points over Qwen 3.5 A17B at 61.1%.
Where2Place: 70.0% (vs. 76.0% for Qwen 3.5 A17B) — trails by 6.0 points.

The MoE-A32B results reveal a more complex picture than the MoT-2B results. The model is clearly competitive with frontier models in aggregate (outperforming Gemini 3.0 Pro by 3.4 points on average) but is not consistently dominant: it ranks first on only 7 of 22 benchmarks, and it trails significantly on several benchmarks (RefSpatial-Bench by 15 points, Ego-Plan2 by 8.6 points, MMSI-Bench by 8.8 points). This suggests that while the post-training pipeline successfully produces strong embodied reasoning capabilities, scaling from 2B to 32B does not uniformly close the gap to frontier models — there remain specific embodied reasoning tasks (particularly those involving referring expression comprehension in spatial contexts and planning from ego-view video) where other frontier models maintain advantages.

It is noteworthy that the API-evaluated models (Gemini 3.0 Pro, Seed 2.0) were assessed "via their official APIs under thinking mode" — a configuration that may differ from the evaluation setup used for HY-Embodied-0.5 and introduces potential variability from API version changes, prompt formatting differences, and thinking-mode implementation details that are not controlled for.

Robot Control Results

Section 6 reports real-world robot manipulation results using a Vision-Language-Action (VLA) model built on the MoT-2B foundation. The VLA model is constructed by extending the MoT architecture with an Action Expert module following the structural design of π0/π0.5. The model is first fine-tuned on 5K hours of UMI data (without exposure to any specific robot embodiment), then supervised fine-tuned on 300-700 episodes of real-robot demonstration data per task. Baselines (π0 and π0.5) undergo identical SFT on the same real-robot data.

Three manipulation tasks are evaluated (Figure 13):

Precision Plug-in Packing: HY-Embodied-0.5 VLA achieves 85% success rate (20 trials), matching π0.5 (85%) and surpassing π0 (80%). This task involves precise insertion, which tests fine-grained spatial perception and manipulation accuracy.
Tableware Stacking: 80% success rate, substantially above π0 (60%) and competitive with π0.5 (85%). Stacking requires understanding of 3D geometry, contact dynamics, and sequential placement.
Mug Hanging: 75% success rate, representing improvements of 30 and 25 percentage points over π0 (45%) and π0.5 (50%), respectively. This is described as "the most challenging given the baseline performances." Mug hanging involves complex affordance reasoning (identifying the handle, aligning with a hook) and precise trajectory execution under gravity.

The paper attributes these results to "the initial fine-tuning on the extensive 5K-hour UMI dataset, combined with the underlying MoT architecture, [which] successfully equips the model with rich, generalizable representations that effectively transfer to complex, embodiment-specific manipulation tasks." Importantly, the 5K-hour UMI pre-training phase uses data that is embodiment-agnostic — the model has not seen the specific robot platform before SFT — meaning the transfer to the target robot occurs entirely through the 300-700 episodes of task-specific demonstration data. The strong performance relative to π0 and π0.5 (which have the same Action Expert architecture but different VLM backbones) suggests that the HY-Embodied-0.5 VLM foundation provides better visual representations and spatial reasoning that accelerate downstream policy learning.

However, these results are limited in several ways: only 20 trials per task are conducted; only three tasks are evaluated; the baselines (π0, π0.5) use different VLM backbones but identical SFT procedures, making it impossible to isolate whether the gains come from the pre-training data (5K hours UMI), the MoT architecture, or the post-training pipeline individually.

Ablation Studies and Robustness Checks

MoT architecture efficiency (Figure 11). The paper compares training loss curves and inference efficiency between the MoT-2B architecture and a standard Dense-2B transformer, both trained with identical data, initialization, and hyperparameters. The training curves (Figure 11a) show that MoT achieves faster convergence and lower final loss than the Dense baseline at the same number of steps. For inference (Figure 11b), with input fixed at 576 image tokens and 100 generated output tokens, the MoT architecture shows nearly identical total inference time, theoretical total FLOPs, prefill time, and decode time compared to the Dense-2B baseline. The paper notes that "because the decoding process dominates the total inference time in practical scenarios, the overall additional time overhead introduced by the MoT structure is negligible." This is a direct consequence of the architecture design: parameter duplication increases total parameter count but not activated parameter count, so per-token computation is unchanged. This ablation supports the claim that MoT provides training benefits without inference penalties.

Visual latent token attention visualization (Figure 12). The paper provides attention map visualizations for the visual latent tokens, showing that they attend to "salient objects, highly specific object parts (such as the right end of the potato chip can or the handles of the drawers), and key spatial regions relevant to the scene context," while language attention "weights are strongly concentrated on core semantic entities, state descriptions, spatial relationships, and action-oriented instructions." This is qualitative evidence for the bridging function of latent tokens, not a quantitative ablation. No comparison is provided showing model performance with and without latent tokens, which limits the strength of the claim that they are causal contributors to performance rather than correlational artifacts.

Thinking mode vs. non-thinking mode for baselines (Section 5.1). The paper reports that Qwen3.5-VL models produced "excessively repetitive outputs" in some benchmarks leading to degraded results, motivating the choice of Qwen3-VL as the primary baseline. For all baseline models, the paper reports "the better result between thinking and non-thinking modes," while for HY-Embodied-0.5 only thinking mode is reported. This makes the comparison conservative with respect to HY-Embodied-0.5 but introduces an asymmetry: baseline models benefit from mode selection while HY-Embodied-0.5 does not. The paper does not report HY-Embodied-0.5's non-thinking mode performance, making it impossible to assess whether the thinking mode provides consistent improvements or whether the model also exhibits mode-dependent instability.

General understanding benchmark comparison (Figure 7). The comparison against Qwen3-VL-2B-Thinking and InternVL3.5-2B on seven general benchmarks serves as a robustness check that embodied-centric training does not catastrophically degrade general VLM capabilities. The results show MoT-2B performing competitively (best on 4/7, second on 3/7), which supports the paper's claim of maintained general capability. However, the paper does not compare against the same baselines used in Table 1 (RoboBrain-2.5, MiMo-Embodied-7B) on these general benchmarks, so the relative standing among embodied specialists on general tasks is unknown.

Missing ablations. Several architectural and training design choices are not ablated in this paper, representing significant gaps in the experimental validation:

No ablation of the MoT architecture's contribution to final benchmark performance. The training efficiency comparison (Figure 11) shows faster convergence, but the paper does not report whether a Dense-2B model trained to convergence (with more steps to match final loss) achieves comparable benchmark scores. The training curves in Figure 11a appear to show the Dense model still descending at the end of training, suggesting it might catch up given more compute — a comparison the paper does not provide.
No ablation of visual latent tokens on downstream performance. The attention visualizations (Figure 12) are qualitative, and no benchmark results are reported for a model variant without latent tokens. The contribution of this architectural component to the claimed performance gains is therefore unquantified.
No ablation of the vision loss ( $\mathcal{L}_{\text{vision}}$ ) or global loss ( $\mathcal{L}_{\text{global}}$ ). The composite pre-training loss combines three objectives (Section 3.3), but the paper does not report performance with any component removed, making it impossible to determine which auxiliary losses are necessary versus helpful versus neutral.
No ablation of the iterative RL→RFT cycle count. The paper describes alternating RL and RFT throughout post-training but does not specify the number of cycles or report performance after each cycle. This makes it impossible to assess whether the iterative alternation provides diminishing returns, whether a single cycle would suffice, or whether performance continues to improve with additional cycles.
No ablation of on-policy vs. offline distillation. The OPD loss (Equation 7) is presented as a key innovation, but the paper does not compare OPD to standard offline distillation (training on teacher-generated responses) for the MoT-2B model. The specific marginal contribution of on-policy sampling versus having a strong teacher is therefore unknown.
No ablation of dynamic vs. static data selection for RL. The capability-frontier curriculum (Section 4.2.1) is described as important for stable optimization, but no comparison against training on a fixed dataset is reported.

Robot control baselines and ablations (Section 6). The robot experiments use π0 and π0.5 as baselines, which have different VLM backbones from HY-Embodied-0.5. No ablation is provided that isolates the contribution of the VLM backbone (HY-Embodied-0.5 vs. the VLM used in π0/π0.5) separately from the 5K-hour UMI pre-training and the Action Expert architecture. The paper does not report: (1) performance of the MoT-2B VLA without the 5K-hour UMI pre-training, (2) performance with a different VLM backbone but the same pre-training and SFT recipe, or (3) performance on the robot tasks using only the VLM's spatial reasoning outputs (without the VLA Action Expert) — which would directly test the claim that the VLM foundation provides superior embodied understanding that transfers to manipulation.

Critical Assessment

The experimental results broadly support the paper's central claim that HY-Embodied-0.5 achieves strong embodied competence — particularly in spatial understanding — through its combined architectural and training innovations. However, the support is uneven across the paper's specific claims, and several important claims rest on qualitative evidence or remain unablated.

Claim: "MoT-2B outperforms similarly sized state-of-the-art models on 16 of 22 benchmarks." This claim is directly supported by Table 1. The 2B model indeed achieves the best score on 16 benchmarks and second-best on 4 more. However, the claim's framing as "similarly sized" is somewhat generous: three of the four baselines have 2×-3.5× more activated parameters (4B-7B vs. 2B), making the achievement more impressive than "similarly sized" suggests. The only true size-matched comparison is against Qwen3-VL-2B, which MoT-2B outperforms by large margins (e.g., 54.5% vs. 41.8% on ERQA, 57.0% vs. 39.9% on 3DSRBench). The performance relative to larger models (Qwen3-VL-4B, RoboBrain-2.5-4B, MiMo-Embodied-7B) should be understood as outperforming larger competitors, not similarly sized ones.

A nuance not captured by the "16 of 22" headline: MoT-2B does not rank first on RoboBench-Planning (4th), ShareRobot-Trajectory (3rd), Ego-Plan2 (3rd), RoboSpatial-Home (3rd), RefSpatial-Bench (3rd), and SAT (2nd). The common thread is that these are planning- and trajectory-heavy benchmarks where the model's VLM foundation (focused on perception and spatial reasoning) may provide less advantage than specialist models designed specifically for action generation. This is a genuine limitation that the paper acknowledges implicitly by not claiming strength on planning tasks.

Claim: "The 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro." This claim is supported with qualifications. The MoE-A32B model achieves a higher average score (67.0% vs. 63.6%) and ranks first on 7 benchmarks, which is a meaningful achievement for an openly released model competing against proprietary frontier systems. However, "comparable" masks substantial variance: MoE-A32B trails Gemini 3.0 Pro by 6.4 points on RoboBench-MCQ, 8.6 points on Ego-Plan2, and 8.8 points on MMSI-Bench, while leading by 15.0 points on ViewSpatial (vs. Gemini's 50.8%) and 10.4 points on VSIBench (vs. Gemini's 57.9%). The average advantage of 3.4 points is driven by large wins on some benchmarks offsetting large losses on others — this is not uniform "comparability" but rather a different capability profile. The paper would be more accurate to claim that MoE-A32B is competitive with frontier models in aggregate, with specific strengths in view-based spatial reasoning and weaknesses in planning and multi-image spatial intelligence.

An important caveat for the Gemini 3.0 Pro comparison: these results were "self-collected via API in March 2026" (Table 2 note). API-based evaluation introduces uncontrolled variables — model version, prompt formatting, thinking-mode implementation, potential API updates between evaluations — that make the comparison less rigorous than the head-to-head comparisons with open-weight models. The paper does not describe the prompt templates used for API evaluation or whether they matched those used for HY-Embodied-0.5.

Claim: "Robust execution in real-world robotic manipulation tasks." This claim is supported by the limited evidence presented but should be treated as preliminary. The robot experiments (Figure 13) show strong performance on three tasks with 20 trials each (60 total trials). The 75% success rate on Mug Hanging (vs. 45-50% for baselines) is genuinely impressive and suggests the VLM foundation transfers useful spatial understanding to manipulation. However, the evidence base is thin: three tasks, one robot platform (dual-arm Xtrainer), 20 trials per condition, no error bars or confidence intervals, and no analysis of failure modes. The paper does not report whether the 20 trials used fixed or randomized object poses (Figure 13b mentions "object poses are randomly initialized at the start of each trial," but this is a single sentence without detail on the randomization procedure). For embodied AI research, where reproducibility and statistical rigor are ongoing concerns, 20 trials per task is at the low end of what is considered convincing — many robotics papers report 50-100+ trials per condition with confidence intervals.

Additionally, the robot results are from a VLA model that extends the MoT-2B foundation with an Action Expert module, fine-tuned on 5K hours of UMI data plus 300-700 episodes of task-specific demonstrations. This is a substantially different system from the VLM evaluated in Tables 1-2. The claim "HY-Embodied-0.5 achieves compelling results in real-world physical evaluations" (Abstract) conflates the VLM foundation with the full VLA system. The paper does not demonstrate that the VLM alone — without the Action Expert and robot-specific fine-tuning — can control a robot. The contribution is better characterized as "the VLM foundation enables strong downstream VLA performance" rather than "the VLM itself performs robot control."

Missing evidence. Several experiments that would substantially strengthen the paper are absent:

Direct architecture ablations on benchmark performance: MoT vs. Dense transformer (not just training curves), with vs. without visual latent tokens, with vs. without vision/global auxiliary losses — all evaluated on the 22-benchmark suite. Without these, the causal contribution of individual architectural components cannot be assessed.
Scaling behavior analysis: performance of the MoT architecture at different model sizes (e.g., 1B, 2B, 4B) to determine whether the benefits of modality-specific computation scale with or diminish with model size. The paper's claim that MoT is "especially effective for small edge models" is asserted without evidence at different scales.
Training data ablation: performance when trained without the spatial-centric data, without the embodied-centric data, or without the large-scale general data — to quantify the contribution of each data component. The paper's emphasis on data construction (Section 3.1) implies data composition is critical, but no evidence demonstrates this.
Post-training pipeline stage-by-stage analysis: benchmark performance after pre-training, after cold-start SFT, after each RL round, after each RFT round, and after OPD — to show where gains occur and whether the iterative alternation provides compounding benefits. This is particularly important because the iterative RL→RFT loop is presented as a key innovation (Section 4.3) but its specific contribution is never isolated.
Comparison to a generalist VLM with the same post-training: if the post-training pipeline is the primary driver of embodied competence, applying the same RL→RFT→distillation pipeline to a generalist VLM (e.g., Qwen3-VL-2B) should produce comparable gains. If not, the embodied pre-training data is the differentiating factor. The paper runs neither comparison.
Real-world robot evaluation with and without the VLM foundation: comparing the full VLA (MoT-2B + Action Expert + 5K hours UMI + SFT) against an Action Expert trained from scratch with the same robot data, or against the VLM's spatial reasoning outputs used directly (without the VLA action head) for open-loop control. This would isolate the VLM's contribution versus the Action Expert and robot-specific fine-tuning.

Evaluation design strengths. Despite these limitations, the evaluation has genuine strengths. The 22-benchmark suite is comprehensive and covers a meaningful range of embodied capabilities — broader than most prior embodied VLM papers evaluate on. The inclusion of both generalist and specialist baselines at multiple scales provides useful context. The conservative baseline evaluation protocol (reporting better of thinking/non-thinking for baselines, thinking-only for HY-Embodied-0.5) is a legitimate methodological choice that strengthens the comparative claims. The qualitative Chain-of-Thought visualizations (Figure 10) effectively illustrate the reasoning behaviors the training pipeline cultivates, even if they don't quantify their frequency or reliability.

Overall assessment. The experiments demonstrate that HY-Embodied-0.5 is a strong embodied VLM — likely the strongest openly available model of its size class for spatial reasoning tasks — and that the combination of MoT architecture, embodied-centric pre-training, and iterative post-training produces a model that competes favorably with larger generalist and specialist alternatives. What the experiments do not demonstrate — and what the paper's claims sometimes imply — is why the model is strong: which components are necessary versus incidental, whether the specific design choices (MoT over MoE, cosine similarity over MSE for latent token supervision, GRPO over PPO, OPD over offline distillation) are causal contributors or matters of taste, and whether the gains will transfer to other model families, tasks, or deployment contexts. The paper is best understood as a systems demonstration — a proof that careful co-design of architecture, data, and training can produce compact embodied VLMs that punch above their weight class — rather than a scientific study that isolates the contribution of individual components. This is a legitimate and valuable contribution, but readers should interpret the specific architectural and algorithmic claims with appropriate caution given the absence of controlled ablations.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Unaccounted For and Makes the Headline Efficiency Gains Partially Theoretical

The assumption or constraint. The paper's compute-optimal framework and the entire post-training pipeline that cultivates reasoning capability depend fundamentally on being able to evaluate the model's outputs — to score whether a bounding box is correct, whether a trajectory reaches its target, whether a reasoning chain is logically sound. This evaluation capability requires either ground-truth annotations (for supervised training data construction) or strong verifier models (for RL reward computation, RFT quality filtering, and the task-aware reward functions in Section 4.2.2). The paper constructs this evaluation infrastructure at enormous computational and annotation cost: generating 2048 samples per question for difficulty estimation (Section 3.2), employing "a stronger VLM teacher" for annotation verification (Section 3.1.1), using "a stronger teacher model" for reasoning quality filtering in RFT (Section 4.3), and deploying "an LLM-based judge as a fallback" for open-ended reasoning tasks (Section 4.2.2). None of these evaluation costs are included in any reported efficiency metric. The paper does not quantify the total compute spent on evaluation, annotation, and verification across the entire training pipeline.

The consequence. The headline results — the 4× efficiency gains over baselines, the 16-of-22 benchmark wins, the strong robot control success rates — are reported net of evaluation costs, meaning they represent what the model can achieve given the evaluation infrastructure, not what it would cost to build and deploy the full system including evaluation. For a practitioner deciding whether to adopt this approach, the relevant cost is the total cost: data construction (human annotation, teacher model inference, verification), difficulty estimation (2048 samples × number of training questions), RL reward computation (16 samples per input × thousands of training steps), RFT quality filtering (teacher model scoring of ~1M candidate traces), and distillation (teacher model inference on student-generated sequences). If evaluation costs dominate the training budget — which is plausible given the paper's reliance on "stronger teacher models" and large-scale sampling for data construction — then the claimed efficiency advantages may shrink or reverse when total cost is accounted for. Furthermore, in a deployment setting where ground-truth annotations are not available (the realistic embodied scenario), the model's performance will depend on the quality of the available verifiers, which the paper shows are themselves imperfect (the PRM over-optimization documented in Section 5.3, the 38% correct-to-incorrect reversion rate for the revision model in Section 6.1). The gap between oracle-evaluated performance (what the paper largely reports) and verifier-evaluated performance (what a deployed system would achieve) is not characterized.

What evidence exists in the paper. The paper does not quantify total evaluation costs or report performance as a function of evaluation budget. The data construction sections (3.1.1–3.1.4) describe automated labeling pipelines that use "a VLM to identify objects" and "a stronger VLM teacher to verify the accuracy of these generated annotations," but the cost of running these teacher models — in FLOPs, wall-clock time, or monetary terms — is never stated. The RL section (4.2.1) notes that each RL stage trains on "a newly constructed set of 50K samples" selected through multi-sample evaluation, but the number of samples used for evaluation versus training is not reported. The RFT section (4.3) states that "approximately 1M candidate examples [are filtered] into around 300K high-quality traces," implying a 3.3:1 ratio of evaluation to training data, but the per-sample evaluation cost is not specified. The robot control section (Section 6) uses 300–700 episodes of real-robot demonstration data per task — a substantial human effort that is acknowledged but not factored into any efficiency comparison.

Mitigation status. The paper does not attempt to mitigate this limitation. It does not report evaluation costs, does not amortize them into the reported efficiency metrics, and does not propose cheaper evaluation alternatives. The discussion of future work (Section 7) does not mention evaluation cost reduction as a research direction. This is a significant gap because the paper's central contribution — that compact VLMs can achieve strong embodied competence through careful data and training — is only practically actionable if a practitioner can replicate the pipeline at acceptable cost. Without quantifying the cost of the pipeline's evaluation infrastructure, the paper provides no guidance on whether the approach is economically viable.

2. The Method Cannot Handle the Hardest Problems — Test-Time Compute Does Not Compensate for Fundamental Capability Gaps

The assumption or constraint. The entire approach — pre-training on embodied data, post-training with RL and RFT, distillation to compact models — assumes that the base model has sufficient capability to produce correct or partially correct outputs for the target tasks. The post-training pipeline is explicitly designed around the capability frontier: RL retains only samples with partial success, RFT filters for reasoning traces above a quality threshold, and distillation transfers behaviors the teacher has already acquired. This means the method can amplify existing capability and consolidate discovered reasoning patterns, but it cannot create capability de novo for tasks where the base model has near-zero success rate.

The consequence. For problem domains where the base model's pass@1 is approximately zero — tasks requiring physical reasoning, spatial understanding, or manipulation planning that are fundamentally outside the model's training distribution — the entire pipeline fails to produce meaningful improvements. The RL stage discards samples that "fail in all attempts" (Section 4.2.1) because they provide no learning signal. The RFT stage cannot filter for high-quality reasoning traces if no correct traces exist. The distillation stage cannot transfer capabilities the teacher never acquired. The result is a hard capability ceiling: no amount of additional RL rounds, RFT iterations, or distillation will enable the model to solve problems it cannot solve at all in its base form. This is directly analogous to the finding in the compute-optimal scaling work that the hardest difficulty bin shows near-zero improvement regardless of test-time compute budget — the model cannot amplify what it fundamentally lacks.

For embodied deployment, this limitation is particularly consequential because real-world environments constantly present novel object configurations, unseen manipulation scenarios, and unexpected physical dynamics that may lie outside any feasible training distribution. A model that works well on benchmark tasks within its capability frontier may fail catastrophically — and silently — on tasks outside that frontier, with no mechanism to detect its own incompetence or request human assistance.

What evidence exists in the paper. The paper's evaluation suite, while comprehensive at 22 benchmarks, does not include a systematic analysis of performance as a function of problem difficulty. The benchmarks report aggregate accuracy without breaking down performance by difficulty level, task complexity, or distance from the training distribution. There is no "hardest bin" analysis equivalent to the difficulty quintile analysis in the compute-optimal scaling work. The qualitative Chain-of-Thought examples (Figure 10) show the model succeeding on complex spatial reasoning and embodied planning tasks, but there is no systematic characterization of failure modes — what kinds of problems does the model consistently get wrong, and are those failures predictable from problem characteristics? The robot experiments (Figure 13) use only three tasks on a single platform with 20 trials each, which is insufficient to characterize the capability boundary. The paper's acknowledgment that the approach targets "easy-to-medium difficulty" regimes (Section 1 framing) implicitly concedes this limitation but does not quantify where the difficulty boundary lies for the specific model variants.

Mitigation status. The paper does not address this limitation. There is no discussion of uncertainty estimation (enabling the model to recognize when it is operating outside its capability frontier), no analysis of failure modes or difficulty-dependent performance, and no proposal for hybrid systems that fall back to human teleoperation or alternative planning methods when the model's confidence is low. The post-training pipeline's filtering criteria — discarding samples that always fail — is an implicit acknowledgment of the capability ceiling, but the paper does not characterize the ceiling's height or suggest strategies for raising it beyond scaling pre-training data or model size.

3. The Architecture and Training Pipeline Are Validated on a Single Model Family with No Cross-Architecture Transfer Evidence

The assumption or constraint. All experiments use models built on the Hunyuan-1.8B language model backbone with the HY-ViT vision encoder — a single model family developed internally at Tencent. The architectural innovations (MoT, visual latent tokens, the specific composite pre-training loss) are designed and tuned for this specific backbone. The post-training pipeline (GRPO with task-aware rewards, iterative RL→RFT, on-policy distillation) is applied exclusively to Hunyuan-derived models. The paper provides no evidence that any component of the approach transfers to other model families, vision encoders, or base architectures.

The consequence. For a practitioner who does not have access to the Hunyuan model family — which describes essentially everyone outside Tencent — the paper's results are existence proofs rather than transferable recipes. It is unknown whether the MoT architecture would provide similar benefits when applied to a LLaMA-based VLM, a Qwen-based VLM, or a from-scratch trained transformer. The visual latent token mechanism might interact differently with different vision encoder architectures (e.g., SigLIP vs. ViT, different patch sizes, different pre-training objectives). The post-training pipeline's effectiveness may depend on specific properties of the Hunyuan base model — its pre-training data mixture, its tokenizer, its hidden dimension and layer count — that may not generalize. The on-policy distillation procedure (Equation 7) makes no assumptions specific to Hunyuan, but its empirical effectiveness is only demonstrated for the Hunyuan-32B → Hunyuan-2B transfer; whether similar gains would be observed when distilling from a different large model (e.g., Gemini, Qwen) into a different small model (e.g., a LLaMA-derived architecture) is unvalidated.

This is not merely an academic concern. In the current landscape, different model families have substantially different inductive biases, pre-training data compositions, and emergent capabilities. A technique that works well for one family may be ineffective or even harmful for another — the paper itself provides evidence of this when it notes that Qwen3.5-VL produced "excessively repetitive outputs" on some benchmarks, motivating the switch to Qwen3-VL baselines (Section 5.1). If model-family-specific behavior can affect even basic inference, it is likely that training dynamics, architecture-performance interactions, and distillation transfer efficiency are also family-dependent.

What evidence exists in the paper. All experiments in Tables 1–2, Figures 7, 11, and Section 6 use HY-Embodied-0.5 models exclusively. The baselines are other model families (Qwen, RoboBrain, MiMo-Embodied, Gemini, Seed, Kimi), but these are used only for comparison, not for replication — the paper never applies its training pipeline to a non-Hunyuan base model and evaluates the result. The architecture ablations that do exist (Figure 11: MoT vs. Dense training efficiency) are within the Hunyuan family. There is no experiment where the post-training pipeline is applied to, say, Qwen3-VL-2B to test whether the pipeline's benefits are architecture-independent or Hunyuan-specific.

Mitigation status. The paper does not address this limitation beyond open-sourcing the models and code, which enables external replication but does not constitute evidence of transferability within the paper itself. The paper makes no claims about architecture-independence and does not discuss the extent to which the design choices (MoT parameter duplication ratio, latent token supervision loss, GRPO clipping thresholds, OPD temperature) might need to be re-tuned for different base models. This is a significant gap for a paper that presents its contributions as general techniques for building embodied VLMs rather than as a specific model release.

4. The Robot Control Results Confound the VLM Foundation with Robot-Specific Fine-Tuning and External Action Architectures

The assumption or constraint. Section 6 evaluates a Vision-Language-Action (VLA) model for robot control, but this VLA model is not the HY-Embodied-0.5 VLM evaluated in Tables 1–2. The VLA model extends the MoT-2B foundation with: (1) an Action Expert module following the π0/π0.5 structural design, (2) 5K hours of UMI data fine-tuning, and (3) 300–700 episodes of task-specific real-robot demonstration data. The baselines (π0, π0.5) use different VLM backbones but the same Action Expert architecture and identical SFT procedure on the same real-robot data. This experimental design means that any performance difference between HY-Embodied-0.5 VLA and π0/π0.5 could be attributed to: the VLM backbone, the 5K-hour UMI pre-training (which π0/π0.5 may or may not have received — the paper does not specify), the interaction between the VLM backbone and the UMI pre-training, or some combination thereof. The specific contribution of the HY-Embodied-0.5 VLM — the model whose capabilities are documented across 22 benchmarks and 7 general understanding tasks — cannot be isolated from the effects of the UMI pre-training and the Action Expert architecture.

The consequence. The paper's claim that "HY-Embodied-0.5 demonstrates superior control capabilities" (Section 6) and achieves "compelling results in real-world physical evaluations" (Abstract, Section 7) overstates what the robot experiments actually demonstrate. The experiments show that a specific VLA system — which includes HY-Embodied-0.5 as the VLM backbone, plus 5K hours of embodiment-agnostic pre-training, plus task-specific SFT, plus an Action Expert module — outperforms π0 and π0.5 on three manipulation tasks. They do NOT show that the VLM itself has robot control capabilities. They do NOT show that the VLM's benchmark performance (spatial reasoning, embodied understanding) correlates with or causes the VLA's manipulation success. They do NOT show that a different VLM backbone (e.g., Qwen3-VL, RoboBrain-2.5) subjected to the same UMI pre-training and SFT recipe would perform worse — a comparison that would be necessary to attribute the gains to the HY-Embodied-0.5 foundation specifically.

For a practitioner deciding whether to adopt HY-Embodied-0.5 for robot control, this confound is critical. If the 5K-hour UMI pre-training is the primary driver of the VLA's strong manipulation performance (as the paper suggests: "the initial fine-tuning on the extensive 5K-hour UMI dataset... successfully equips the model with rich, generalizable representations"), then the value of the HY-Embodied-0.5 VLM specifically — versus any competent VLM that can serve as a vision backbone — is unclear. The practitioner would need to know whether the investment in training or adopting HY-Embodied-0.5 is justified over using an off-the-shelf VLM and relying on the UMI pre-training and SFT recipe.

What evidence exists in the paper. The robot experiments (Figure 13) report success rates for three models (HY-Embodied-0.5 VLA, π0, π0.5) on three tasks with 20 trials each. There is no ablation that removes the UMI pre-training, replaces the VLM backbone with a generalist alternative, or uses the VLM directly for open-loop control without the Action Expert. The paper does not report whether π0 and π0.5 also received the 5K-hour UMI pre-training or were trained only on the 300–700 episodes of task-specific data — this is a critical missing detail that determines whether the comparison isolates the VLM backbone or the pre-training recipe. The paper states that baselines "undergo SFT under identical conditions using the same real-robot data, with both the data volume and training iterations" (Section 6), but does not clarify whether "identical conditions" includes the UMI pre-training phase.

Mitigation status. The paper does not address this confound. The discussion in Section 6 attributes the performance to "the initial fine-tuning on the extensive 5K-hour UMI dataset, combined with the underlying MoT architecture," which acknowledges both factors but does not disentangle them. No future work is suggested to isolate the VLM's specific contribution to manipulation performance. For a paper whose title and abstract emphasize "Embodied Foundation Models for Real-World Agents," the gap between the VLM evaluation (comprehensive, multi-benchmark) and the robot evaluation (narrow, confounded, small-sample) is substantial and undermines the paper's strongest deployment-relevant claims.

5. No Ablation of Individual Architectural Components Makes Causal Attribution Impossible

The assumption or constraint. The paper presents HY-Embodied-0.5 as the product of several simultaneous innovations: the MoT architecture, visual latent tokens, the composite pre-training loss (LLM + vision + global), the embodied-centric data composition, the iterative RL→RFT post-training pipeline, the task-aware reward design, the on-policy distillation, and the dynamic data selection for RL. The paper does not report the performance of any model variant that removes or modifies one of these components while holding others constant. With the exception of the MoT vs. Dense training efficiency comparison (Figure 11), every design choice is evaluated only as part of the complete system.

The consequence. It is impossible to determine which of the paper's claimed innovations are necessary for the observed performance, which are helpful but not essential, and which are incidental — included in the final system but not contributing meaningfully to the results. A practitioner seeking to adopt elements of the approach cannot prioritize: should they invest in implementing the MoT architecture, or would the embodied pre-training data alone provide most of the gain? Is the iterative RL→RFT loop worth the engineering complexity, or would a single round of RL followed by SFT suffice? Does on-policy distillation provide meaningful benefits over standard offline distillation, or is the teacher model quality the dominant factor? Without ablations, every design choice carries equal uncertainty, and the paper provides no guidance on where to focus implementation effort.

This is particularly problematic for the paper's most architecturally novel claims. The visual latent tokens (Section 2.3) are motivated by theoretical arguments about bridging modalities and supported by qualitative attention visualizations (Figure 12), but their quantitative contribution to any benchmark is unknown. The vision loss and global loss (Section 3.3) add complexity to the pre-training objective, but whether removing them degrades performance — and by how much — is unmeasured. The asymmetric clipping in GRPO (Section 4.2.3) is motivated by stability concerns in "long-chain multimodal RL," but whether symmetric clipping would produce equivalent or better results is untested. Each of these components could be essential, helpful, neutral, or even slightly harmful — the paper provides no evidence to distinguish among these possibilities.

What evidence exists in the paper. The only architectural ablation is Figure 11, which compares MoT vs. Dense training loss curves and inference efficiency. This shows MoT converges faster during training and achieves lower final loss with negligible inference overhead — a useful result, but it does not demonstrate that MoT causes the benchmark performance gains. A Dense-2B model trained to equivalent loss (with more steps) might achieve comparable benchmark scores; without that comparison, the training efficiency result cannot be linked to the downstream capability claims. The paper provides no ablation of: visual latent tokens (model with vs. without), composite pre-training loss components (removing vision loss, removing global loss, removing both), post-training pipeline stages (pre-training only, +SFT, +RL, +RFT, +distillation), reward design choices (single uniform reward vs. task-aware rewards, GRPO vs. PPO), data composition ratios (removing spatial data, removing embodied data, varying the 12:5:3 mid-training ratio), or distillation method (on-policy vs. offline).

Mitigation status. The paper does not acknowledge the absence of ablations as a limitation. The contributions are presented as an integrated system, and the implicit argument is that the whole is greater than the sum of its parts — that the specific combination of architecture, data, and training choices produces emergent embodied competence that no subset would achieve. This is a legitimate systems contribution, but it requires the paper to be explicit about its evaluative scope: "we present a system that works, not an analysis of why each component matters." The paper does not make this distinction, instead presenting individual components (MoT architecture, latent tokens, RL→RFT loop) as independently motivated innovations, which creates an expectation of component-level validation that the experiments do not satisfy.

6. The Evaluation Is Limited to Static Benchmarks with No Characterization of Real-Time, Closed-Loop, or Safety-Critical Performance

The assumption or constraint. The 22-benchmark evaluation suite (Section 5) measures the model's ability to answer questions about static images or pre-recorded video clips — it tests spatial reasoning, embodied understanding, and planning in an open-loop, offline setting where the model produces a single response to a fixed input with no time pressure, no environmental feedback, and no consequences for errors. The robot experiments (Section 6) test closed-loop manipulation but on only three tasks with 20 trials each and no analysis of failure modes, recovery behavior, or safety properties. Neither evaluation setting measures the model's performance under the constraints that define real-world embodied deployment: real-time processing requirements, the need to recover from unexpected environmental changes, the safety implications of incorrect predictions, or the degradation of performance over extended operation.

The consequence. A model that achieves 82.8% on EmbSpatial-Bench or 75% success on Mug Hanging may still be unsuitable for deployment if it: (1) requires 5 seconds of inference per decision while the robot's control loop runs at 20 Hz, (2) produces confident but catastrophically wrong spatial predictions 5% of the time with no uncertainty signal, (3) degrades significantly under the domain shift from curated benchmark images to real-time, noisy, motion-blurred robot camera feeds, or (4) cannot detect when it has made an error and safely abort or request human intervention. The paper's evaluation provides no information about any of these deployment-critical properties. For a model family explicitly designed "for real-world embodied agents" and "edge deployment," the absence of latency benchmarks, uncertainty calibration analysis, robustness to visual perturbations, and failure mode characterization is a significant gap between the paper's claims and its evidence.

The real-time inference requirement is particularly important given the paper's emphasis on the 2B edge variant. The MoT architecture's inference efficiency is benchmarked (Figure 11b) at 576 image tokens and 100 output tokens, but there is no characterization of how inference time scales with the length of Chain-of-Thought reasoning — which the paper shows can involve extensive self-correction and spatial analysis spanning hundreds of tokens (Figure 10). If thinking-mode inference requires 10× more output tokens than non-thinking mode, the latency implications for a real-time control loop could be prohibitive, even if per-token efficiency is unchanged.

What evidence exists in the paper. The benchmarks are all static, single-response evaluations. There is no measurement of inference latency beyond Figure 11b's single-configuration timing. There is no analysis of model calibration (do the model's confidence scores correlate with correctness?), no robustness evaluation under distribution shift (blur, noise, lighting changes, viewpoint variation), and no characterization of failure modes (what kinds of errors does the model make, and are they predictable?). The robot experiments use 20 trials per task, with no reporting of: trial duration, inference time per action, types of failures observed, recovery attempts, or safety incidents. The paper does not discuss whether the model ever produced unsafe commands or required human intervention during the robot trials.

Mitigation status. The paper does not address these deployment-relevant limitations. There is no discussion of latency requirements for embodied applications, no uncertainty quantification, and no robustness evaluation. The efficiency analysis (Figure 11) focuses on FLOPs and training convergence rather than real-time inference characteristics under realistic embodied workloads. The robot experiments are presented as proof-of-concept rather than systematic deployment validation. For a paper that positions its 2B model specifically for edge deployment and real-time responsiveness, these omissions limit the practical utility of the reported results for system builders making deployment decisions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around embodied VLMs from "can we adapt general-purpose models to embodied tasks?" to "what does a purpose-built embodied VLM look like?" — a conceptual reframing with practical consequences. Prior work in this space had largely taken one of two approaches: fine-tune a generalist VLM on a modest amount of embodied data (the transfer-learning approach, exemplified by robotic applications built on GPT-4V or open-weight VLMs like LLaVA), or train a specialist model from scratch on narrower robot-specific datasets (the RoboBrain/MiMo-Embodied approach). Both paths implicitly treat embodied competence as something you add after general visual understanding is established. HY-Embodied-0.5 inverts this: embodied and spatial data dominate pre-training (236B tokens, over 38% of the total corpus), and general understanding data supplements rather than leads. The result — a 2B model outperforming 4B and 7B competitors on 16 of 22 embodied benchmarks — is an existence proof that embodied-first pre-training, combined with modality-adaptive architecture, produces more capable embodied models per parameter than generalist-first pre-training followed by adaptation.

This is not a paradigm shift in the Kuhnian sense — the VLM architecture (vision encoder + LLM backbone) remains intact, and the training techniques (SFT, RL, distillation) are drawn from the standard LLM post-training playbook. But it is a substantive methodological reframing that changes how practitioners should allocate their data and compute budgets. The message is: if your goal is embodied intelligence, invest your pre-training tokens in spatial and embodied data, not more web-scraped image captions. The paper makes this case quantitatively: the 2B model's 30-point margin over Qwen3-VL-2B on MindCube (66.3% vs. 28.4%) and 17-point margin on 3DSRBench (57.0% vs. 39.9%) cannot be explained by architecture alone — these benchmarks test spatial mental modeling and 3D reasoning capabilities that generalist VLM training does not cultivate, and the embodied-centric pre-training is the most plausible causal factor (even if the paper does not ablate data composition directly).

The paper also resolves a latent contradiction in the embodied VLM literature that was less visible than the self-correction debate in LLM reasoning but equally real. On one side, generalist VLMs like Qwen-VL and InternVL demonstrated strong performance on broad visual benchmarks while openly struggling on spatial reasoning tasks (evidenced by their low scores on 3DSRBench, MindCube, and ViewSpatial in Table 1). On the other side, specialist embodied models like RoboBrain-2.5 showed stronger spatial competence but often at the cost of general visual understanding, and they typically required larger model sizes to achieve competitive performance. The implicit question was whether this tradeoff was fundamental — whether embodied capability necessarily comes at the expense of general capability, or whether it requires scale that makes edge deployment impractical. HY-Embodied-0.5 argues: neither. The 2B model maintains general VLM performance comparable to size-matched generalists (Figure 7: competitive on DocVQA, TextVQA, RealWorldQA) while simultaneously achieving state-of-the-art embodied performance. The key insight — which prior work had not operationalized — is that modality-adaptive computation (MoT) prevents the visual training needed for embodied tasks from overwriting the language capabilities that support general understanding. The separate QKV and FFN parameters for visual vs. text tokens mean that heavy visual training updates the vision branch without perturbing the language branch, resolving the catastrophic forgetting that plagues standard VLM fine-tuning on domain-specific data.

The paper also redirects research attention along two axes. First, it makes data composition for embodied pre-training a first-class research problem. The extensive data construction sections (3.1.1–3.1.4) are not merely implementation details — they are the paper's implicit argument that what you train on matters at least as much as how you train. The five-category visual perception data (omni-detection, depth, segmentation, pointing/counting), the three-tier embodied hierarchy (grounding → affordance → planning), and the five-type spatial data (correspondence, geometry, configuration, measurement, dynamics) constitute a proposed taxonomy of embodied knowledge that future work can adopt, critique, or refine. Second, it makes iterative self-evolving post-training for multimodal models a concrete research target. The RL→RFT→distillation loop was previously demonstrated primarily for text-only math and code reasoning (DeepSeek-R1); HY-Embodied-0.5 shows it can be adapted to the heterogeneous output structures of embodied tasks through the task-aware reward design (Figure 6), and that the resulting reasoning behaviors — self-correction, spatial analysis, step-by-step verification — transfer to compact models via on-policy distillation. This opens the door for future work on cultivating reasoning in any multimodal domain with structured outputs.

What becomes less attractive as a research direction is incremental improvements to generalist VLM architectures applied unchanged to embodied tasks. The paper's results suggest that the returns to better general-purpose vision encoders or larger language model backbones, without corresponding investment in embodied-specific data and training, are diminishing for embodied applications. The 7B MiMo-Embodied model — a specialist with presumably embodied-focused training — underperforms the 2B HY-Embodied-0.5 on the majority of benchmarks, suggesting that scale alone cannot compensate for absent domain-specific training. Similarly, the paper makes pure architectural innovation without training data innovation look insufficient: the MoT architecture's benefits (Figure 11) are realized in the context of the massive embodied pre-training corpus; there is no evidence that MoT would provide comparable gains if trained only on general web data.

Follow-Up Research This Work Enables

Difficulty-aware adaptive inference for embodied VLMs. The paper's evaluation treats all benchmark questions uniformly, but real-world deployment involves a mixture of easy perceptual queries (requiring minimal reasoning) and complex spatial planning problems (requiring long Chain-of-Thought). A direct follow-up would apply the compute-optimal test-time scaling framework — estimating question difficulty from initial model outputs and allocating inference budget accordingly — to the embodied domain. The paper already provides the necessary infrastructure: the thinking/non-thinking mode mechanism (controlled by \think and \no think tokens, Section 3.2) enables the model to toggle between fast direct answers and slow deliberate reasoning. A strong experiment would: (1) bin the 22 benchmark questions by difficulty using the MoT-2B model's pass@1 rate, (2) measure the accuracy vs. inference cost tradeoff at different thinking budgets (number of reasoning tokens generated), (3) determine whether easy questions can be routed to non-thinking mode with negligible accuracy loss while reserving thinking mode for medium-hard questions, and (4) compare the total inference cost of this adaptive policy against uniform thinking-mode evaluation. The hypothesis — grounded in the compute-optimal scaling literature and consistent with the paper's observation that thinking mode produces long self-correcting chains (Figure 10) — is that adaptive routing could reduce average inference cost by 2–4× without meaningful accuracy degradation.

Ablation of the embodied pre-training data composition. The paper's central implicit claim is that the specific mixture of embodied and spatial data drives the model's benchmark performance, but no data ablation exists. A high-value follow-up would train multiple MoT-2B variants from scratch, each with one major data component removed: no spatial-centric data (Section 3.1.3), no embodied-centric data (Section 3.1.2), no visual perception data beyond standard detection (Section 3.1.1), and general-only pre-training (the 389B general tokens alone). Each variant would be evaluated on the full 22-benchmark suite. The results would answer: which data components are load-bearing for which capabilities? Does spatial-centric data primarily drive the spatial understanding benchmarks (3DSRBench, MindCube, ViewSpatial) as one would expect, or does it have spillover effects on embodied understanding? Does embodied-centric data contribute to spatial reasoning by providing task context, or is its benefit confined to planning and affordance benchmarks? Can general data alone, with the MoT architecture and post-training pipeline, achieve competitive embodied performance — suggesting the architecture and training are sufficient and the data is incidental? The paper's current state leaves all of these questions open, and answering them would transform the work from a systems demonstration into a scientific analysis.

Cross-architecture replication of the post-training pipeline. The paper's post-training innovations — task-aware GRPO rewards for structured embodied outputs, iterative RL→RFT alternation, on-policy distillation from large to small — are applied exclusively to Hunyuan-derived models. A critical stress-test would apply the identical pipeline (same RL reward functions, same RFT filtering criteria, same OPD objective) to a non-Hunyuan base model — for instance, starting from Qwen3-VL-2B as the base, applying the embodied pre-training data, and running the full post-training sequence. If the pipeline produces comparable gains on Qwen-derived models, the techniques are architecture-independent and the paper's contribution is primarily the training methodology. If the gains are substantially smaller or absent, the results are tied to specific properties of the Hunyuan architecture (e.g., its pre-training objective, its tokenizer, the specific way MoT interacts with RL gradients), and the generalizability claims need to be significantly narrowed. A related experiment would swap the teacher and student: use HY-Embodied-0.5 MoE-A32B as the teacher for a Qwen3-VL-2B student (via OPD), testing whether the teacher's embodied reasoning can transfer across model families — a much stronger test of the distillation approach's generality than within-family transfer.

Closed-loop embodied evaluation with latency constraints. The paper's 22-benchmark evaluation is entirely open-loop, and the robot experiments (20 trials per task) do not characterize real-time performance. A deployment-relevant follow-up would benchmark the MoT-2B model in a closed-loop embodied setting — for example, using a standardized simulation environment (e.g., ManiSkill, LIBERO, or Calvin) where the model must process visual observations and produce actions at a fixed control frequency (e.g., 5–20 Hz). Key measurements would include: end-to-end inference latency (vision encoding + LLM decoding) as a function of Chain-of-Thought length, success rate as a function of latency budget (how much does performance degrade when the model is forced to produce shorter reasoning chains to meet a real-time deadline?), and comparison against non-thinking mode and against smaller/faster VLM baselines. The paper's emphasis on the 2B model as an "edge deployment" target makes this evaluation essential — if thinking-mode inference takes multiple seconds per decision while the robot's control loop requires 100ms responses, the model's benchmark accuracy is irrelevant for deployment. The MoT architecture's inference efficiency benchmarks (Figure 11b) are at a single configuration (576 image tokens, 100 output tokens) and do not capture the scaling behavior that matters for embodied control loops.

Uncertainty estimation and failure prediction for safety-critical embodied deployment. The paper provides no analysis of when the model knows it is wrong. For real-world robots, a model that silently produces confident but incorrect spatial predictions (e.g., a grasping point 10cm from the true object location) is substantially worse than a model that recognizes its uncertainty and requests clarification or falls back to a safer strategy. A follow-up study would evaluate the MoT-2B model's calibration: does the model's token-level probability correlate with correctness across the 22 benchmarks? Can the model be prompted to output explicit confidence scores or to flag ambiguous inputs? A specific experiment would test whether the Chain-of-Thought reasoning process itself contains signals of uncertainty — e.g., do phrases like "Wait, maybe I got it wrong" or excessive self-correction cycles in the thinking trace (visible in Figure 10) predict incorrect final answers? If so, these could serve as lightweight runtime uncertainty estimates without requiring additional calibration training. The robot experiments could be extended with a simple safety protocol: the model generates an action only if its self-assessed confidence exceeds a threshold; otherwise, it requests human teleoperation. Measuring the tradeoff between task completion rate and human intervention rate would directly inform deployment feasibility.

Scaling behavior of the MoT architecture across model sizes and modalities. The paper asserts that MoT is "especially effective for small edge models" (Section 2.2) but provides no evidence at different scales — only the 2B MoT vs. 2B Dense comparison. A scaling study would train MoT and Dense variants at multiple sizes (e.g., 0.5B, 1B, 2B, 4B, 8B activated parameters) on the same embodied data mixture and measure: (1) the performance gap between MoT and Dense at each scale — does the benefit diminish as total parameter count increases (consistent with the paper's intuition that larger models can learn cross-modal alignment implicitly) or does it persist? (2) the training efficiency benefit at each scale — does MoT's faster convergence (Figure 11a) hold across sizes? (3) the inference efficiency at each scale — does MoT's negligible overhead (Figure 11b) remain negligible as the vision-to-text token ratio changes? This would directly address whether MoT is a general architectural principle for multi-modal models or a specialized solution for the parameter-constrained regime. The paper's open-source release makes this study feasible for external researchers with sufficient compute.

Practical Applications and Downstream Use Cases

On-device visual grounding for mobile manipulation robots. The MoT-2B model's strong performance on fine-grained spatial tasks — 92.3% on DA-2K depth estimation, 89.2% on CV-Bench visual perception, 60–68% on ViewSpatial and VSIBench — makes it directly applicable as the perception module for a mobile robot that must locate, identify, and estimate the pose of objects in real-time from onboard camera feeds. The key deployment advantage is the model's compact size (2B activated parameters) combined with the MoT architecture's inference efficiency (Figure 11b: negligible overhead vs. Dense-2B). A robot using HY-Embodied-0.5 for visual grounding could run entirely on edge hardware (e.g., a Jetson Orin) without cloud dependency, enabling operation in environments with limited or no connectivity — warehouses, construction sites, disaster response scenarios. The model's native-resolution processing (HY-ViT 2.0, Section 2.1) is particularly valuable here because standard VLM resizing pipelines degrade the spatial precision needed for manipulation — a 5-pixel error at 224×224 resolution corresponds to a much larger physical error at native camera resolution.

Automated data labeling for robot manipulation datasets. The pipeline described in Section 3.1 for constructing the embodied pre-training data — using a VLM to annotate objects, SAM for segmentation, a stronger VLM teacher for verification — is itself a practical application of HY-Embodied-0.5. Organizations collecting robot manipulation data could deploy the model to automatically generate bounding boxes, affordance labels, and trajectory annotations on their video footage, with the model's Chain-of-Thought reasoning providing interpretable annotation rationale. The model's 73.3% on ShareRobot-Trajectory and 82.8% on EmbSpatial-Bench suggests it can serve as a reasonable first-pass annotator, with human verification needed only for edge cases. The cost savings relative to fully manual annotation — which dominates the budget for robot learning datasets — could be substantial. A specific deployment scenario: a lab with 1,000 hours of unlabeled robot manipulation video could use HY-Embodied-0.5 to generate initial grounding and task labels, then have human annotators spend their time verifying and correcting rather than labeling from scratch, reducing annotation time by an estimated 50–70% (based on the model's accuracy rates on relevant benchmarks).

Safety monitoring and scene understanding for human-robot collaboration. The model's embodied understanding capabilities — ERQA (54.5%), RoboSpatial-Home (55.7%), EmbSpatial-Bench (82.8%) — indicate it can answer questions about spatial relationships, object states, and task progress in cluttered environments. This enables a safety monitoring application where the model continuously evaluates camera feeds from a collaborative robot workspace and answers queries like "is the human in the robot's reachable workspace?", "has the object been placed in the correct bin?", or "are there any obstacles on the planned path?" The model's Chain-of-Thought reasoning (Figure 10) provides interpretable rationales for its assessments, which is critical for safety applications where decisions must be auditable. The robot control results (Section 6) — 85% on precision plug-in packing, 75% on mug hanging — demonstrate that the VLM's spatial understanding transfers to manipulation contexts, suggesting the monitoring model's assessments would be grounded in the physical constraints the robot actually faces rather than superficial image-level patterns.

Curriculum generation for robot skill learning. The paper's dynamic data selection strategy for RL — retaining only partially solvable samples near the model's capability frontier (Section 4.2.1) — could be adapted to generate training curricula for downstream robot policy learning. Given a library of manipulation tasks, HY-Embodied-0.5 could assess which tasks are at an appropriate difficulty level for the current policy (using its embodied understanding to predict task success likelihood) and automatically sequence task presentation from easy to hard. The model's strong performance on spatial understanding benchmarks (e.g., 66.3% on MindCube, 57.0% on 3DSRBench) suggests it can reason about task complexity in spatial terms (e.g., "this task requires navigating around an obstacle while maintaining a specific orientation"). This addresses a bottleneck in robot learning where human engineers manually design training curricula — a labor-intensive process that does not scale to the diversity of tasks robots will face in unstructured environments.

When to Prefer This Method

The paper explicitly positions HY-Embodied-0.5 against both generalist VLMs (Qwen3-VL, InternVL3.5) and specialist embodied models (RoboBrain-2.5, MiMo-Embodied), and the results in Tables 1–2 support specific tradeoff boundaries. The decision framework is:

Prefer HY-Embodied-0.5 (or its training methodology) when:

The target application involves fine-grained spatial reasoning — multi-view geometry, depth estimation, 3D spatial relationships, point-level localization. The model's dominant performance on spatial understanding benchmarks (9 of 12 first-place finishes for MoT-2B, including a 30-point margin on MindCube) makes it the clear choice for these tasks over comparably sized alternatives.
Deployment constraints require edge execution with a small parameter budget (≤4B activated parameters). The MoT-2B variant outperforms 4B and 7B competitors on the majority of benchmarks, making it the strongest option per parameter for embodied tasks — particularly relevant for on-device robot perception where cloud offloading is infeasible.
You can invest in domain-specific pre-training data — the model's advantage over generalist VLMs is driven by the 236B tokens of embodied and spatial data. If your organization has access to robot manipulation data, 3D scene data, or spatial annotations, the paper's data construction recipes (Sections 3.1.1–3.1.4) provide a template for building a similarly advantaged model. Without this data investment, the architecture and training pipeline may not outperform a generalist VLM fine-tuned on a small embodied dataset.
Interpretable reasoning is valuable — the model produces explicit Chain-of-Thought reasoning traces (Figure 10) that include self-correction, spatial analysis, and step-by-step verification, which is useful for debugging, safety auditing, and human-in-the-loop oversight.

Prefer generalist VLMs (Qwen3-VL, InternVL3.5) when:

The task distribution is broad and not primarily embodied — if your application requires strong performance on general visual QA, document understanding, chart reading, and OCR alongside occasional spatial queries, a generalist model may be more practical. HY-Embodied-0.5 maintains competitive general performance (Figure 7) but was not optimized for it, and a generalist model fine-tuned on your specific domain may outperform on non-embodied tasks.
You cannot invest in embodied-specific pre-training — the HY-Embodied-0.5 architecture (MoT, visual latent tokens) provides efficiency benefits, but the performance gains derive substantially from the data. If your compute budget only allows for fine-tuning an existing model rather than large-scale pre-training, starting from a strong generalist VLM and fine-tuning on your target embodied tasks may be more cost-effective than replicating the full HY-Embodied-0.5 training pipeline.

Prefer specialist embodied models (RoboBrain-2.5, MiMo-Embodied) when:

Trajectory prediction and action planning are the primary capability — the MoT-2B model trails RoboBrain-2.5 on ShareRobot-Trajectory (73.3% vs. 81.4%) and MiMo-Embodied on RoboBench-Planning (54.2% vs. 58.7%), suggesting that specialist models with architectures designed specifically for action generation may retain advantages on these tasks even if they underperform on spatial perception.
The deployment robot platform matches the specialist model's training distribution — RoboBrain-2.5 and MiMo-Embodied were trained on specific robot embodiments and may transfer better to similar platforms than a general embodied VLM that has not seen that specific kinematic structure during training.