LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment

ArXiv: 2604.11689

🎯 Pitch

General vision models like DINOv3, trained with zero action supervision, crush specialized robotic latent-action models on physical control—and latent space trumps pixel space every time. This benchmark shows that the best action representations for robots might already be hiding inside your favorite vision foundation model.

---

1. Executive Summary

This paper introduces the Latent Action Representation Yielding (LARY) Benchmark, a unified evaluation framework that decouples the assessment of latent action representations from downstream policy performance by probing representations along two complementary dimensions—high-level semantic action classification (what to do) and low-level physical control regression (how to do it)—across a curated dataset of over 1.2M videos, 620K image pairs, and 595K motion trajectories spanning 151 action categories and 11 robotic embodiments. Through systematic evaluation of 11 models across three paradigms—Embodied Latent Action Models (LAMs such as LAPA, UniVLA, and villa-X), General Vision Encoders (DINOv3, V-JEPA 2), and a newly proposed class of General LAMs that graft LAM training onto frozen vision backbones—the paper reveals that off-the-shelf general vision foundation models trained without any action supervision consistently outperform specialized embodied LAMs (e.g., V-JEPA 2 achieving 76.62% average classification accuracy versus LAPA's 20.17%), and that latent-based visual spaces are fundamentally better aligned to physical action space than pixel-based spaces (e.g., DINOv3 achieving 0.19 average regression MSE versus FLUX.2-dev's 0.35), establishing that action-relevant features emerge naturally from large-scale visual pretraining while purpose-built embodied training may actually constrain representation quality due to data scarcity.

2. Context and Motivation

The Core Problem: We Don't Know How to Evaluate Latent Action Representations

The fundamental question this paper tackles is deceptively simple: how do we know whether a latent action representation is any good? This matters because the dominant paradigm in contemporary robotics research attempts to solve the "data island" problem—the chronic scarcity of action-labeled robotic datasets—by extracting reusable control signals from unlabeled video data, particularly human demonstrations available at internet scale. The logic is appealing: if we can automatically discover a representation that captures what actions are being performed and how they're executed, purely from watching videos, we can then transfer that understanding to robots without expensive teleoperation data.

However, this logic contains a hidden assumption that is almost never challenged: that the latent action representation being extracted is actually capturing meaningful, transferable action structure rather than surface-level visual correlations, background cues, or degenerate collapsed representations. The paper identifies this as a critical missing piece in the VLA research pipeline—we have many methods for producing latent actions but no rigorous, standardized way to evaluate their quality independently of downstream task performance.

This gap is significant for several interconnected reasons the authors highlight across the introduction (Section 1) and related work (Section 2):

• Conflated evaluation: Current practice evaluates latent action models by plugging them into a complete policy-learning pipeline and measuring task success. This means a poor latent representation can be partially compensated for by a powerful policy head, and conversely, a good latent representation can be unfairly dismissed because the downstream policy was poorly tuned. The representation quality and the policy quality are indistinguishably entangled. The paper states this directly: existing approaches "fail to decouple the assessment of the VLA components from the latent action quality itself" (Section 1).

• No cross-embodiment generalization testing: Robot embodiments vary dramatically in morphology (single-arm Franka, bimanual AgileX, semi-humanoid AgiBot G1), sensor configuration (egocentric vs. exocentric cameras), and action parameterization (7-DoF, 12-DoF, 16-DoF). A latent action representation that works well for one robot may fail entirely for another, yet existing evaluations rarely test across embodiments systematically.

• No multi-granularity assessment: Actions in embodied intelligence span at least two fundamentally different levels: high-level semantic intent ("pick up the cup") and low-level physical execution (the precise joint velocities and end-effector trajectory to accomplish that pick-up). The paper notes that existing evaluations "fail to span different entities, tasks, and granularities, making it difficult to assess the generalization capabilities of action representations" (Section 1).

• Underexplored architecture impact: Latent action models use wildly different training architectures (VQ-VAE with discrete codebooks, continuous autoencoders, inverse dynamics models), different training data distributions (robot-only, human-only, mixed), and different inductive biases (forward dynamics, language conditioning, saliency constraints). Yet there is "a notable absence of evaluation methods that... span different entities, tasks, and granularities" (Section 1), meaning we cannot systematically compare which design choices matter.

The Data Island Problem and Why Latent Actions Matter

To appreciate why rigorous evaluation of latent actions is important, we need to understand the broader context that motivates extracting actions from video in the first place.

Robotics foundation models face a fundamental data scarcity problem. Unlike vision or language, where models can be trained on billions of internet images or trillions of text tokens, robotics data is painfully expensive to collect. Teleoperating a robot to perform a single manipulation task requires specialized hardware, trained human operators, and careful setup—producing perhaps a few hundred demonstrations for a few hundred dollars of operator time. The result is that even the largest robotic manipulation datasets are orders of magnitude smaller than their vision and language counterparts.

Human video data, in contrast, is abundant and diverse. Platforms like YouTube, ego-centric datasets (Ego4D, EPIC-KITCHENS), and instructional video corpora (Something-Something v2) provide millions of hours of humans performing a vast range of actions in natural environments with diverse objects. If we could extract useful control signals from these videos, we could potentially bootstrap robotic learning from a much richer data source.

This is where latent action models (LAMs) enter the picture. As described in Section 2, the core idea—pioneered by works like LAPA, Moto, and LAPO—is to train an unsupervised model that takes two frames (or a sequence of frames) from a video and outputs a compressed representation (a "latent action") that captures the visual change between them. This latent action is meant to be ontology-independent: it doesn't care whether the agent is a human hand, a Franka arm, or a mobile manipulator. It just captures the abstract idea of what motion occurred.

The analogy the field draws (alluded to in Section 1) is to NLP: just as word pieces provide a language-agnostic subword representation that enables cross-lingual transfer, latent actions are meant to provide an embodiment-agnostic motion representation that enables cross-embodiment transfer. A "pick up" latent action extracted from human videos should, in principle, contain useful structure for a robot learning to pick up objects, even if the robot's kinematics look completely different.

Where Existing Evaluation Approaches Fall Short

The paper identifies specific limitations in how prior work evaluates latent action representations. These limitations are not merely methodological nitpicks—they fundamentally undermine our ability to make progress on VLA design.

1. Downstream task performance as a proxy is unreliable. The primary evaluation method in the literature (used by LAPA, UniVLA, villa-X, GR00T, and others) is end-to-end: train a VLA policy using the latent actions as the action space, run the policy on a manipulation benchmark (typically LIBERO, CALVIN, or a custom task suite), and report the success rate. The paper points out that this "fails to decouple the assessment of the VLA components from the latent action quality itself" (Section 1). If the policy succeeds, we don't know whether it succeeded because the latent actions were informative, because the policy head was powerful enough to compensate for noisy latents, or because the task was simply easy. If the policy fails, we similarly cannot attribute the failure to the latent representation versus the policy architecture versus training instability.

2. Qualitative cluster visualizations are suggestive but not diagnostic. Many papers (LAPA, CoMo, AdaWorld) include t-SNE or PCA visualizations showing that latent actions for similar motions cluster together. While visually appealing, the paper implicitly argues that these are insufficient: they don't tell us whether the clusters correspond to the right semantic categories (or merely to correlated visual features like background color), they don't quantify separability, and they don't generalize to new embodiments or environments.

3. Reconstruction metrics conflate action and appearance. Standard autoencoder training losses measure how well the latent action can reconstruct the target frame. The paper notes (Section 2) that "standard reconstruction metrics often fail to distinguish action dynamics from environmental noise." If the model accurately reconstructs a static background but misses the subtle motion of a gripper closing, it achieves low reconstruction error while failing to capture the action. Reconstruction quality is dominated by the parts of the image that change least, which is precisely the opposite of what we need from an action representation.

4. No standardized cross-method comparison framework. Prior work evaluates each LAM in isolation, on different datasets, with different downstream tasks, using different metrics. There is no common ground truth for "what does a good latent action look like" that spans models. Two approaches to latent action extraction—discrete VQ-based (LAPA, Moto) versus continuous (CoMo)—have never been systematically compared on the same action categories, the same embodiments, or the same evaluation protocol. The field lacks a shared benchmark, analogous to how ImageNet standardized visual representation evaluation.

5. Existing benchmarks probe the wrong things. The paper mentions EWMBENCH (Yue et al., 2025) and LAWM (Tharwat et al., 2025), which use trajectory consistency or Canonical Correlation Analysis for alignment. These approaches evaluate whether the latent dynamics are self-consistent (does the same motion always produce the same latent?), but they don't evaluate whether the latent space captures semantically meaningful action categories or generalizes across embodiments. A latent space can be perfectly self-consistent while being entirely uninformative about what physical action is occurring.

6. The "emergent action from vision" hypothesis is untested. There is a tantalizing possibility that standard visual foundation models (DINO, V-JEPA, SigLIP) trained purely on internet images or videos—with no robotics-specific training, no inverse dynamics modeling, no action annotation—might already encode rich action-relevant structure in their feature spaces. After all, these models are trained on data containing countless examples of humans performing actions. However, no prior work has systematically tested this hypothesis. The paper positions itself as the first to ask: do we even need specialized LAM training, or can we just use off-the-shelf vision encoders?

Conflicting Evidence and Unresolved Questions

The paper is motivated by several tensions in the existing literature that a unified benchmark could help resolve:

Discrete versus continuous latent actions. Discrete approaches (LAPA, Moto) use vector quantization to produce a finite vocabulary of action tokens, enabling autoregressive behavior cloning (predict the next action token given the observation history). The paper notes that these "often come at the cost of fine-grained information loss." Continuous approaches (CoMo) preserve motion fidelity at the risk of "shortcut learning from background cues." Which one actually captures more useful action structure? The literature provides no apples-to-apples comparison.

Training data distribution. Embodied LAMs like LAPA are trained on relatively small, domain-specific datasets (primarily robot manipulation videos). General vision encoders like DINOv3 are trained on massive, diverse internet datasets. How does training data scale and diversity affect latent action quality? Is more diverse data always better, or does domain-specific training provide essential inductive biases?

Architectural inductive biases. Different LAMs incorporate different auxiliary objectives: UniVLA adds language-conditioned semantic constraints, IGOR focuses on goal-image representations, LatBot integrates robot trajectory priors. Do these additions help or hurt? Without a standardized evaluation, we cannot attribute performance differences to specific design choices.

The cross-embodiment transfer hypothesis. The entire premise of learning from human videos is that action representations transfer across embodiments. But how well does this actually work? Do latent actions extracted from human demonstrations contain useful structure for robotic control, or do the morphological differences between human hands and robot grippers create an insurmountable domain gap? The paper notes (Section 2) that recent diagnostic studies (Zhang et al., 2025a) reveal that "many models struggle with distractor robustness"—suggesting that putative cross-embodiment transfer may be shallower than hoped.

How This Paper Positions Itself

The paper positions itself not as proposing a new latent action model, but as providing the first standardized, decoupled evaluation framework for any latent action representation, regardless of its training methodology. The key conceptual move is to separate evaluation into two orthogonal dimensions that correspond to the two fundamental questions any action representation must answer:

Semantic (what to do): Can the latent representation distinguish between different action categories (pick, place, pour, stir)? This is operationalized as a probe-based classification task: freeze the latent action extractor, train a lightweight classifier on top, and measure classification accuracy across 151 action categories spanning both human and robot videos.

Control (how to do it): Does the latent representation preserve sufficient physical detail to reconstruct the actual motion trajectory? This is operationalized as a regression task: given a latent action extracted from two frames, predict the sequence of end-effector positions/rotations/gripper states that occurred between those frames, measured by mean squared error against ground-truth robot trajectories.

By testing both dimensions across diverse embodiments (11 robotic platforms plus human) and environments (simulated tabletops, real kitchens, industrial scenes), using a consistent evaluation protocol applied to all models, the framework enables direct comparisons that were previously impossible.

The paper explicitly frames this as analogous to how standardized benchmarks transformed other areas of machine learning. Just as ImageNet enabled systematic comparison of visual representations by decoupling feature extraction from downstream task-specific architectures, LARYBench aims to decouple latent action quality from downstream policy learning. This is a methodological contribution rather than an algorithmic one—but one that the authors argue is necessary for the field to make principled progress.

The paper also positions itself as testing a provocative hypothesis: that specialized LAM training may be unnecessary or even harmful. If general vision foundation models already encode rich action structure from large-scale pretraining, then the entire paradigm of training domain-specific latent action models on scarce robotic data may be misguided. The benchmark results in subsequent sections provide evidence for this hypothesis, but the framing in the introduction and related work is careful to present it as an open question that the benchmark is designed to answer, not a foregone conclusion.

3. Technical Approach

This is primarily a benchmark and systematic evaluation paper whose core idea is to decouple the assessment of latent action representations from downstream policy performance by probing them along two orthogonal axes—semantic discriminability (via probe-based classification) and physical-grounding fidelity (via regression of robot trajectories)—using a meticulously curated, multi-embodiment, multi-granularity dataset and a standardized evaluation protocol applied identically across all candidate models.

3.1 Reader Orientation

The authors are building a standardized test suite—not a new latent action model—that measures how well any given visual-to-action representation captures two things simultaneously: the identity of an action (what is happening: pick, place, pour) and the physics of an action (the precise trajectory the robot arm or hand follows). The problem this solves is that the field currently conflates representation quality with downstream policy performance, making it impossible to compare different latent action extraction methods or to diagnose what makes a representation good or bad. The solution takes the shape of two fixed, lightweight "probe" heads—an attentive classifier and an MLP regressor—that are trained on frozen latent representations, so the resulting accuracy and MSE depend only on the quality of the representation, not on the power of the probing architecture.

3.2 Big-Picture Architecture (Diagram in Words)

The LARYBench system has four major components, arranged in a pipeline that flows from raw video data through standardized evaluation tasks to a ranking of representation quality:

1. Data Curation Engine — an automated pipeline (built around the doubao-1.5-pro-vision VLM API) that takes raw, heterogeneous video datasets from multiple sources, performs temporal action segmentation, verifies semantic consistency between video clips and action labels, and produces a clean, uniformly annotated corpus of ~1.2M short video clips, ~620K image pairs, and ~595K motion trajectories organized into two evaluation tracks (classification and regression).

2. Latent Action Extractors (frozen) — the models under test. These are any function that maps visual observations $o_{1:T}$ to a latent action representation $z \in \mathcal{Z}$. The benchmark evaluates three families: Embodied LAMs (purpose-built VQ-VAE + inverse dynamics models like LAPA, UniVLA, villa-X), General Vision Encoders (off-the-shelf models like DINOv3, V-JEPA 2 trained without action supervision), and General LAMs (the LAPA VQ-VAE architecture re-trained with frozen DINOv2/DINOv3/SigLIP2/MAGVIT2 backbones replacing the pixel encoder). Each extractor is run once per test sample to produce a frozen latent vector; the extractor weights are never updated during evaluation.

3. Evaluation Protocol (two tracks) — a fixed, lightweight probing architecture applied identically to all extractors:

   • Semantic Action Classification Track: A 4-layer attentive probe (following V-JEPA's architecture) maps the latent $z$ to a predicted action category $\hat{c}$ from a set of 151 classes. The metric is Top-1 classification accuracy, averaged across action categories.
   • Low-Level Control Regression Track: An MLP-based Action Expert (2 residual blocks, hidden dimension 4096, residual connections) maps $z$ to a predicted trajectory chunk of size $s$ (7-DoF, 12-DoF, or 16-DoF per frame, depending on the robot embodiment). The metric is Mean Squared Error (MSE) between predicted and ground-truth trajectories.

4. Analysis and Ranking — the system outputs accuracy and MSE numbers per model per task, enabling direct, decoupled comparison. Additional diagnostic analyses (cross-attention visualization, stride ablation, long-tail performance breakdown) provide insight into why certain representations succeed or fail.

Information flows as follows: raw videos from diverse sources (Ego4D, SSv2, EPIC-KITCHENS, LIBERO, CALVIN, AgiBotWorld, etc.) → Data Curation Engine temporally segments and re-annotates them → standardized video clips, image pairs, and trajectories are organized into classification and regression task splits → each candidate latent action extractor processes the visual inputs and outputs a frozen latent vector $z$ → lightweight probes are trained on these frozen latents → accuracy and MSE are reported, compared across models, and decomposed by action frequency, embodiment, and temporal stride.

3.3 Roadmap for the Deep Dive

• First, the formal evaluation objective and the two-task decomposition, which defines what "good latent action representation" means and why the benchmark tests what it tests. Understanding this framing is essential before diving into implementation details.
• Second, the data curation engine—the automated pipeline for transforming raw heterogeneous videos into standardized evaluation samples—since the quality of the entire benchmark depends on clean, consistent annotations. This is a substantial engineering contribution that enables the benchmark's scale and diversity.
• Third, the classification track protocol (probe architecture, frame sampling, latent extraction specifics), because it is the simpler of the two evaluation axes and introduces the core idea of probing frozen representations.
• Fourth, the regression track protocol (Action Expert architecture, trajectory chunking, cross-embodiment handling), which extends the probing paradigm to continuous physical control and introduces the additional complexity of heterogeneous action spaces.
• Fifth, the General LAM construction procedure—how existing pre-trained vision encoders are integrated into the LAPA VQ-VAE framework—since this introduces a new model family that bridges the gap between general vision and embodied LAMs and requires careful design choices about what to freeze and what to learn from scratch.

3.4 Detailed, Sentence-Based Technical Breakdown

Formal Evaluation Objective and Task Decomposition

The paper defines the benchmark's evaluation objective in Section 3 with a concise formalism. Given a sequence of visual observations $o_{1:T}$ (the raw frames from a video clip), a latent action model (LAM) extracts the motion information as a latent representation:

$z \in \mathcal{Z}$

where $z$ is the latent action vector produced by the model under test, and $\mathcal{Z}$ is the model's latent space. The notation $o_{1:T}$ refers to a sequence of $T$ frames; in practice, $T$ varies by task.

What it computes: The LAM (any of the 11 candidate models) takes raw visual input—either a video clip for composite actions, a pair of frames for atomic actions, or a pair of frames with a specified temporal stride for regression—and outputs a compressed vector $z$ that is meant to capture the change between those frames. The LAM is treated as a black box; the benchmark cares only about the properties of $z$ as a representation, not about how $z$ was produced.

Why this form: This abstraction is deliberately minimal. By defining the evaluation target as $z \in \mathcal{Z}—a fixed-dimensional continuous vector irrespective of the underlying model architecture—the benchmark can accommodate any representation paradigm: continuous autoencoders, discrete VQ-VAE codebook indices (converted to continuous embeddings), semantic encoders (using penultimate layer features), or pixel-based generative encoders (using VAE latents). The only requirement is that the model can produce a vector representation of visual change. This avoids baking architectural assumptions into the evaluation framework, which is essential for fair cross-paradigm comparison.

The benchmark then evaluates the efficacy of $\mathcal{Z}$ through two complementary tasks:

$f_{\text{sem}} : \mathcal{Z} \rightarrow \mathcal{C} \quad \text{(semantic action decoding via classification)}$

$f_{\text{dyn}} : \mathcal{Z} \rightarrow \mathcal{A} \quad \text{(robotic control construction via regression)}$

where $\mathcal{C}$ is the set of 151 action categories, and $\mathcal{A}$ is the space of continuous end-effector trajectories (D-dimensional vectors repeated over $s$ future time steps, where $D \in \{7, 12, 16\}$ DoF depending on the robot embodiment, and $s$ is the temporal stride between the input frames).

What these compute: $f_{\text{sem}}$ takes a latent action from an arbitrary model and produces a predicted action category label. $f_{\text{dyn}}$ takes the same latent action (extracted from the same input, but in the regression track rather than the classification track) and produces a predicted sequence of robot joint/pose commands. Both $f_{\text{sem}}$ and $f_{\text{dyn}}$ are learned probes: lightweight, fixed-architecture neural networks trained specifically to map from $z$ to the target, with $z$ held frozen. The quality of $z$ is measured by how accurately these simple probes can decode the ground truth—better representations should make this mapping easier, regardless of the model that produced them.

Why this form: This two-axis decomposition directly operationalizes the intuition that actions span two complementary levels (Section 3, opening paragraph): "high-level semantic intent that specifies what to do, and low-level physical control that determines how to do it." A representation that excels at classification but fails at regression captures semantic identity but loses physical detail (plausible for discrete VQ models with small codebooks). A representation that excels at regression but fails at classification captures fine-grained motion but cannot distinguish action categories (plausible for continuous models that overfit to low-level dynamics). The benchmark demands that a good representation do both, and the two separate probes make this demand explicit and decoupled—a model cannot hide behind a powerful policy head or use one dimension to compensate for weakness in the other. Alternative formulations would combine these into a single multi-task probe, but that would make it harder to diagnose which aspect of action understanding a representation fails at.

Data Curation Engine: From Raw Videos to Standardized Evaluation Samples

The benchmark's data construction pipeline (described in Section 3 and detailed in Appendix B.2) is a substantial engineering contribution that deserves careful explication, because the quality and consistency of annotations directly determine the benchmark's reliability. The pipeline is organized around a central design choice: rather than relying on existing dataset annotations—which suffer from "imprecise temporal boundaries and inconsistent action annotations" (Section 1)—the authors build an automated engine that re-processes all source data uniformly using a single Vision-Language Model (the doubao-1.5-pro-vision API) as the core reasoning agent.

The pipeline operates differently for the two data types (video clips for classification, image pairs for regression), but shares a common philosophy: deterministic extraction where possible, VLM-based curation where human judgment is needed, and manual inspection as a final quality filter.

Atomic Classification Data Curation (LIBERO)

For the Kinematic-Level Atomic Primitives task, the data comes from the LIBERO suite and requires no VLM-based processing because the actions are simple, discrete, and mechanically well-defined. The procedure (described in Appendix B.2 under "Atomic Classification") works as follows:

1. Trajectory parsing: The authors track the cumulative positional offset of the end-effector in the LIBERO dataset. They extract "the precise start and end frames corresponding to the exact moments when the cumulative displacement surpasses a predefined threshold." This threshold-based detection segments the continuous robot motion into discrete chunks, each bounded by the start of motion and the end of motion.

2. Label assignment: Ground-truth action labels are assigned based on "the directional shifts along the axes." For example, if the cumulative displacement is predominantly along the positive x-axis, the action is labeled as a directional translation (e.g., move_forward). Binary gripper states are similarly detected from gripper open/close commands. The paper defines 28 discrete kinematic primitives comprising "finely resolved directional translations and binary gripper states" (Section 3.1). Examples visible in Figure 4 include move_left, move_right, move_forward, move_backward, move_up, move_down, and their diagonal combinations (e.g., move_top_left, move_forward_right_up), plus gripper_open and gripper_close.

3. Output: The result is 25,940 high-quality image pairs, each comprising a temporally aligned start-frame/end-frame pair coupled with a kinematic action annotation. No video understanding is needed because the action labels are mechanically determined from the robot's own proprioceptive data, making this track a clean test of whether latent representations can capture fine-grained spatial displacements.

Composite Classification Data Curation (Multi-Source Human and Robot Videos)

This is the far more challenging case. The source datasets (HoloAssist, Ego4D, SSv2, TACO, EPIC-KITCHENS, EgoDex, AgiBotWorld-Beta) come with heterogeneous temporal segmentation (some have frame-level action boundaries, some have coarse clip-level labels), inconsistent action vocabularies (different datasets use different verb taxonomies), and varying video quality (different frame rates, camera angles, lighting conditions). The curation pipeline (Section 3.1 and Appendix B.2) operates in four stages, all using the same VLM API for consistency.

Stage 1: Action Segmentation. The raw video is processed through the doubao-1.5-pro-vision API with a specific system prompt and action segmentation prompt. The API receives uniformly sampled frames—specifically, the sampling FPS is set to 5, and the minimum number of sampled frames is 16 (Appendix B.2, before the prompt listing)—and is asked: "Please watch this video and find out all key actions in this video. Ensure each action is processed independently of the others. List every single action separately with its start/end timestamps. Keep descriptions concise, objective, and in English."

The API returns a list of action segments, each with temporal boundaries (start time, end time) and a short English description. This stage transforms raw long videos into a massive corpus of short clips, each roughly corresponding to a single action.

Stage 2: Video-Description Matching. The initial segmentation is noisy—automated cropping may produce clips that are too short (fragments of an action), too long (multiple actions concatenated), or mismatched (the description doesn't match the visual content). This stage applies three filtering criteria using a second API call:

The prompt (Video Description Matching Prompt, Appendix B.2) instructs the API:

• "Strict Verification: Does the video content EXCLUSIVELY represent the description {old_desc}?" It must match only if the video "contains the described action and NOTHING else." It must mismatch if "the video is missing parts of the action, OR contains any additional, unrelated actions before, during, or after."
• "Identify perspective: '1st' (ego) or '3rd' (non-ego)."
• "Final Action: If and only if it's a STRICT match, provide ONE most precise English verb defining the movement... Otherwise, return 'None'."

The three criteria enforced are:

• Temporal validity: Clip duration must be within [0.5s, 20s]. Clips shorter than 0.5 seconds cannot capture complete action semantics; clips longer than 20 seconds likely contain multiple entangled actions.
• Semantic alignment: The visual content must perfectly match the description, capturing a single complete action.
• Perspective consistency: Only egocentric videos are retained; exocentric viewpoints from mixed datasets (like SSv2) are explicitly filtered out.

Additionally, the API extracts the core action verb from the descriptive sentence. The output of this stage is a refined set of video clips, each paired with a single verb label.

Stage 3: Video-Verb Consistency Check. To eliminate semantic ambiguity and ensure the isolated verb accurately encapsulates the visual content, a third API call is performed. The prompt (Video-Verb Consistency Check Prompt, Appendix B.2) asks: "Please watch this video and determine if the action {action} is performed. Output Yes only if the action is clearly identifiable and the video content does not contradict the label. Output No if the action is missing, incorrect, or represents a different verb entirely." This stage catches cases where the verb extracted in Stage 2 doesn't fully describe what's happening—e.g., a video labeled "put" that actually shows "place" plus "withdraw."

Stage 4: Manual Sampling Inspection. Finally, authors perform "a manual quality assurance review to exclude verbs that lack explicit or well-defined kinematic meanings." Action categories representing "overly abstract or kinematically ambiguous operations (e.g., apply, arrange, clean) are systematically purged from the taxonomy to maintain the physical and dynamic rigor of the benchmark" (Appendix B.2). This is a crucial filtering step: the benchmark wants to evaluate whether latent actions capture physical motion, not whether they capture high-level task intent (which would be more a test of scene understanding than action understanding).

The output of this four-stage pipeline is a clean, uniformly annotated corpus of 692,297 human clips and 538,423 robot clips under a unified taxonomy of 145 composite behavior classes. The paper notes (Section 3.1) that "beyond this initial curation, our automated engine holds the potential to continuously process future data streams, ensuring the ever-growing nature of the dataset."

Regression Data Curation

For the regression track, the data sources (CALVIN, VLABench, RoboCOIN, AgiBotWorld-Beta) already provide ground-truth robot trajectories synchronized with video frames. The curation here is mechanical rather than semantic: "We obtain the requisite image pairs and action trajectories by sampling observations at a fixed temporal stride" (Appendix B.2, "Regression" paragraph). Specifically, the start frame and end frame (separated by stride $s$) are paired with the continuous sequence of absolute kinematic actions occurring strictly between them. No VLM is needed because the action labels come from the robot's own joint encoders and kinematics.

The paper does deliberately handle one nuance: for RoboCOIN, which records 10 diverse bimanual platforms with dexterous hand joints, the authors "deliberately mask the dexterous hand joint data... to focus the evaluation on macroscopic arm displacements, as fine-grained finger articulation remains an ill-posed inverse problem for current visual encoders" (Section 3.2). This is a pragmatic design choice: recovering 20+ finger joint angles from two frames is effectively impossible from visual data alone, and including finger joints would add noise to the regression target without testing anything the representations could plausibly capture. The masked action space for RoboCOIN is 12-DoF (position + Euler angles for two arms).

Why this curation approach over alternatives: The alternative—using existing dataset annotations directly—would introduce several systematic biases. Different datasets label actions at different granularities (e.g., SSv2 has fine-grained verbs like "pushing [something] from left to right" while EPIC-KITCHENS has coarser verbs like "cut"), use different temporal segmentation standards (some have frame-level boundaries, others clip-level), and include varying amounts of annotation noise. Using a single VLM to re-process all data ensures a consistent annotation policy applied uniformly across sources, which is essential for fair cross-domain evaluation. The specific choice of doubao-1.5-pro-vision is not extensively justified beyond being a VLM "with robust spatio-temporal video understanding capabilities" (Figure 3 caption), but the three-stage verification pipeline (segment → match → consistency-check) provides redundancy that should mitigate individual VLM errors. The manual sampling inspection in Stage 4 acts as a final safety net against systematic annotation biases.

Semantic Action Classification Track Protocol

The classification track (Section 3.1 and Section 4.2, "Probing Protocol for Semantic Action Classification") evaluates whether latent representations can distinguish between action categories. The protocol design addresses several challenges: how to sample frames fairly across datasets with different FPS, how to extract latent actions from video clips of varying lengths, and how to compare representations with different dimensionalities.

Frame Sampling with Motion-Guided Sampler. The source datasets in the composite classification tasks "differ in FPS and exhibit non-uniform motion speeds" (Section 4.2, "Sampling and Latent Action Extraction"). Simple uniform sampling "generally fails to capture the true temporal dynamics and often yields latent actions that are effectively insensitive to the underlying motion." For example, if the video has high FPS but slow motion, uniformly sampled frames may show almost no visual change, making the latent action degenerate.

To address this, the authors employ the Motion-Guided Sampler (MGSampler) from Zhi et al. (2021). This sampler selects a sequence of frames that exhibits "sufficient dynamic variation"—it identifies frames where meaningful motion occurs, avoiding redundant near-duplicate frames. The paper specifies that 9 frames are uniformly sampled from each video clip (which typically lasts under 5 seconds), and every frame is resized to 224 × 224. Latent action features are "subsequently extracted across all adjacent sampled frames to ensure the representation effectively encapsulates the transition dynamics." This means that from 9 frames, the model processes 8 adjacent pairs, and the resulting latent actions are aggregated (the exact aggregation method—concatenation, averaging, or treating them as a sequence—is implicit in how the probe processes them, described next).

Probe Architecture: 4-Layer Attentive Probe. The classifier mapping $z$ to action category $\hat{c}$ follows the architecture used in V-JEPA (Assran et al., 2025). The paper specifies "a 4-layer attentive probe as the classifier" (Section 4.2). The term "attentive probe" means the probe includes a multi-head attention mechanism (specifically, a temporal pooler with cross-attention, as visualized in Figure 6) that learns to weight the importance of different latent action tokens (or different spatial positions within latent feature maps) when making the classification decision. This is important because it allows the probe to focus on the most action-relevant parts of the latent representation rather than being forced to use all latent dimensions equally—it can learn that certain spatial positions in the feature map reliably encode motion information while others encode background.

Dimensionality Alignment. Since different models produce latent actions with different dimensionalities—V-JEPA 2's features have one dimension, DINOv3's have another, LAPA's VQ embeddings have yet another—the authors "incorporate a projector to align the continuous latent actions into a uniform dimensional space prior to classification, thereby ensuring a fair comparison" (Section 4.2). This projector is a learned linear (or possibly MLP) layer that maps each model's native latent dimension to a common size. The projector is trained as part of the probe—its parameters are learned during the 20 epochs of probe training—so models are not disadvantaged by having smaller latent spaces (the probe can expand them) or larger ones (the projector compresses). However, the quality of information in the original latent space constrains what the projector can recover: if a model has discarded action-relevant information during its own training, no probe projector can bring it back.

Training and Hyperparameters. The probe is trained for 20 epochs using bfloat16 precision. The optimization uses "a multi-head optimization strategy with learning rates from $3 \times 10^{-4}$ to $5 \times 10^{-3}$ and weight decay values from $0.01$ to $0.8$" (Section 4.2). The phrase "multi-head optimization strategy" likely means a hyperparameter sweep over these ranges, with the best configuration selected per model (or the best result reported). The wide range of weight decay values (from 0.01 to 0.8) suggests significant regularization is sometimes needed, possibly to prevent the probe from overfitting to small latent spaces. The batch size, optimizer, and other standard hyperparameters are not specified in the main text (and the Appendix doesn't provide probe-specific training details beyond the ranges given here).

Data Split. For Atomic Robot, a 75:25 stratified split is used. For Composite Human and Composite Robot, a 70:30 stratified split is used. The stratification ensures that the class distribution is preserved across train and test splits, which matters because the action categories have a strong long-tail distribution (Figure 2 shows some actions having orders of magnitude more samples than others). Without stratification, random splitting could place all samples of a rare action in one split, making classification performance estimates unreliable for that action.

Metric: Top-1 Accuracy Averaged Across Categories. The paper reports "the Top-1 Accuracy averaged across all action categories" (Section 4.2). The phrase "averaged across all action categories" is important: this is a macro-average, not a micro-average. A macro-average gives equal weight to each action class regardless of its frequency, meaning that performance on rare (long-tail) actions matters just as much as performance on common actions. This is a deliberate choice that penalizes models that only capture frequent action categories while ignoring rare ones—consistent with the benchmark's goal of testing generalization and representational richness, not just overall accuracy on head classes.

Why this protocol over alternatives: Three design choices stand out. First, the use of MGSampler rather than uniform sampling: the benchmark is testing latent action quality, and to do so fairly, the input frames must actually contain motion. Feeding near-identical frames to a LAM that expects to see change would produce degenerate latent actions that don't reflect the model's true capability. MGSampler ensures the evaluation tests the representation under conditions where it should work.

Second, the attentive probe: a simpler linear probe (commonly used in representation evaluation, e.g., in the self-supervised learning literature) might not have enough capacity to extract action category information from complex latent spaces, especially for models like DINOv3 that produce high-dimensional spatial feature maps. The attentive probe's ability to learn spatial attention weights (Figure 6 demonstrates this visually) gives it sufficient capacity to handle diverse representation formats while remaining lightweight enough that its performance primarily reflects the representation quality rather than probe architecture. A ResNet-50 or ViT probe head would be powerful enough to potentially compensate for poor representations.

Third, the macro-averaging: if the benchmark used overall accuracy (micro-average), a model could achieve high scores by only recognizing the top 10 most frequent actions and ignoring the remaining 141 categories entirely. Such a model would be useless for generalization, which is the entire point of latent action representations. Macro-averaging forces models to perform well across the full action taxonomy.

Low-Level Control Regression Track Protocol

The regression track (Section 3.2 and Section 4.2, "Action Experts for Low-level Control Regression") evaluates whether latent representations preserve sufficient physical detail to reconstruct end-effector trajectories. This track introduces additional complexity: the action space dimensionality varies across robot embodiments, and the representations must generalize not just across appearance variations but across fundamentally different kinematic structures.

Latent Action Extraction with Fixed Stride. For the regression tasks, latent actions are "derived from image pairs separated by a frame interval of $s = 5$" (Section 4.2). This means the input to the LAM is two frames: the observation at time $t$ and the observation at time $t + 5$. The LAM produces a single latent vector $z$ that should encode the motion that occurred during those 5 time steps. The default stride of 5 is chosen as a moderate value—too small a stride captures negligible motion, making the regression task trivial; too large a stride captures complex, multi-phase motion that may be difficult to encode in a single latent vector. The stride ablation in Section 5.3 tests robustness by varying this to 15 and 30.

Action Expert Architecture. The regression head (called the "Action Expert") is deliberately kept simple to ensure the evaluation depends on representation quality rather than decoder capacity. The paper specifies: "It adopts a standard MLP with residual connections architecture featuring 2 residual blocks and a hidden dimension of 4096" (Section 4.2). An MLP with residual connections means each block computes:

$h_{i+1} = h_i + \text{MLP}(h_i)$

where the MLP applies two (or more) fully-connected layers with non-linear activation between them, and the residual connection adds the block input back to block output (enabling gradient flow and preventing degradation as depth increases). The hidden dimension of 4096 provides substantial capacity per block (two blocks, each with an internal 4096-dimensional representation), but the overall architecture remains an MLP—no attention, no recurrence, no convolutional structure. This means any spatiotemporal reasoning must already be encoded in the latent $z$; the Action Expert cannot discover new structure that the LAM failed to capture.

Action Chunk Prediction. The Action Expert "directly maps latent actions to an action chunk of size $s$" (Section 4.2). The phrase "action chunk" means the model predicts all $s$ future actions simultaneously from a single latent vector $z$, rather than autoregressively predicting one step at a time. The dimensionality per step depends on the robot:

• For CALVIN and VLABench (single-arm Franka): 7-DoF per step (likely 3D position, 3D rotation as Euler angles or quaternion, 1D gripper state).
• For RoboCOIN (bimanual, 10 platforms): 12-DoF per step (3D position + 3D Euler angles for each of two arms, with dexterous hand joints masked out).
• For AgiBotWorld-Beta (AgiBot G1, bimanual semi-humanoid): 16-DoF per step. The paper specifies this "targets a 16-DoF space containing absolute position, quaternions, and gripper state" (Section 3.2). Since quaternions are 4D and position is 3D, a 16-DoF space for two arms plus grippers would be: 3 (position) + 4 (quaternion) + 1 (gripper) = 8 DoF per arm × 2 arms = 16 DoF.

With a stride of $s = 5$, the Action Expert predicts a vector of size $5 \times D$ where $D \in \{7, 12, 16\}$. This is a substantial output space—for AgiBotWorld-Beta, the model predicts $5 \times 16 = 80$ scalar values from a single latent vector, testing whether the compressed representation has retained enough fine-grained motion detail.

Metric: Mean Squared Error. The evaluation metric is "Mean Squared Error (MSE) to measure physical fidelity" (Section 4.2). All trajectory values are absolute (not relative to the start position), meaning the Action Expert must predict the exact end-effector coordinates, not just the displacement. This is a harder task than predicting relative motion because it requires the latent representation to encode not just the direction and magnitude of movement but also information about the robot's current configuration (which determines where the end-effector is in absolute space).

Cross-Environment Train/Validation Splits. The regression tasks use environment-level splits where possible, which tests out-of-distribution generalization:

• CALVIN: "environments A, B, and C are allocated for training, with D designated for validation" (Section 4.2). This is a standard cross-environment split that tests whether the representation generalizes to visual conditions (lighting, background, object placement) not seen during probe training.
• VLABench: "standard 75:25 train-validation split" (within the same environment, since VLABench has a single environment configuration).
• RoboCOIN and AgiBotWorld-Beta: 75:25 splits for training and validation (these are large-scale datasets where environment-level splits may not be feasible or meaningful).

Why this design over alternatives: Three choices are notable. First, the simple MLP Action Expert: the paper could have used a sequence model (LSTM, Transformer) that autoregressively predicts actions step-by-step, which might better handle the temporal structure of trajectories. However, a more powerful decoder would conflate representation quality with decoder capacity, violating the benchmark's core principle of decoupled evaluation. The MLP forces the latent $z$ to carry all temporal information internally.

Second, the choice to predict absolute rather than relative actions: relative action prediction (predict the delta from the current position) is simpler and more common in behavior cloning, but it allows the policy to accumulate errors. Absolute prediction, in contrast, tests whether the latent representation encodes enough information to recover the full robot state. This tests a stronger notion of physical grounding.

Third, the cross-environment split on CALVIN: this is the most rigorous generalization test in the benchmark. If a latent representation works well on environments A-C but fails on D (which has different visual appearance), it suggests the representation has captured superficial visual correlations rather than genuine action structure. The benchmark doesn't heavily emphasize this split's results (they're aggregated into the main MSE numbers), but it provides a valuable diagnostic.

General Latent Action Model Construction

The paper introduces a new model family—General LAMs—that bridges the gap between off-the-shelf vision encoders and purpose-built embodied LAMs (Section 4.1, "General Latent Action Models," and Appendix C). Understanding how these models are constructed is essential because they serve as an ablation that isolates the effect of the latent action training paradigm from the effect of the visual backbone.

Core Idea: Feature-Level Latent Action Modeling. The key insight is to "substitute the visual representation from pixels to features generated by vision encoders in the LAPA framework, while retaining its VQ-VAE structure" (Section 4.2, "Feature-Level Latent Action Model Training"). This means taking the standard LAPA architecture—which normally takes raw pixel frames as input, encodes them through a trainable vision encoder, applies a spatial-temporal VQ-VAE to compress frame-to-frame changes into discrete latent action tokens, and reconstructs the target frame—and replacing the pixel-level encoder-decoder with a feature-level one.

Concretely, the architecture works as follows:

1. Frozen Vision Backbone: A pre-trained vision encoder (DINOv2, DINOv3, SigLIP2, or MAGVIT2) processes each input frame and outputs a feature map. These backbone weights are kept frozen throughout training—the vision encoder provides a fixed feature space, and the LAM learns to represent changes within that feature space.

2. Feature-Level VQ-VAE: The original LAPA VQ-VAE, which reconstructs pixel values, is modified to reconstruct feature values instead. The paper states: "their training objective is formulated to reconstruct these high-level latent features rather than raw pixels" (Appendix C.1). This is a critical difference: the model learns to predict what the target frame's DINO (or SigLIP, etc.) features should look like, not what the target frame's pixels should look like. Feature reconstruction is inherently easier than pixel reconstruction (features are lower-dimensional and discard photometric details like exact texture, lighting, and background appearance), which means the model can focus its capacity on representing motion rather than appearance.

3. VQ Bottleneck: The spatial-temporal transformer architecture from LAPA is retained unchanged: "Both the spatial and temporal encoders are configured with a depth of 4 layers, utilizing 16 attention heads with a dimension of 64 per head" (Appendix C.2). The VQ codebook compresses the frame-to-frame feature changes into discrete tokens. The initial quantization configuration uses: quantization dimension = 32, codebook size = 8, sequence length = 16 (Appendix C.2). These are deliberately conservative (small codebook, short sequence) to establish a baseline; the ablations in Section 5.3 vary these to study their effect on representation quality.

4. Training Data and Procedure: The General LAMs are "trained on an expansive internal video dataset comprising over 2 million frames... curated to encompass a diverse mixture of human demonstrations, robotic manipulations, and general environmental dynamics" (Appendix C.2). This dataset is described as "internet data, containing both human motions and non-human motions (e.g., car driving), which is very different from the test dataset" (Section 4.3). The training uses a "randomized temporal stride within a predefined range" to sample start and end frames, preventing overfitting to specific frame rates. All models are trained for exactly 100,000 steps using 8 NVIDIA H800 (141GB) GPUs, with input images standardized to 224 × 224 resolution.

Backbone-Specific Configurations (Appendix C.2):

• LAPA-DINOv3: Input feature dimension 1024, patch size 16. (DINOv3 produces 1024-dimensional features; the spatial grid uses 14 × 14 patches for 224 × 224 images, yielding a 14 × 14 feature map.)
• LAPA-DINOv2: Input feature dimension 1024, patch size 14. (DINOv2 produces 16 × 16 patches, yielding a 16 × 16 feature map at 224 × 224.)
• LAPA-SigLIP2: Input feature dimension 768, patch size 16.
• LAPA-MAGVIT2: This variant "operates as a generative-based variant, where the reconstruction target remains in the visual pixel space" (Appendix C.1). Unlike the other three, MAGVIT2 is a video VAE designed for pixel reconstruction, so the LAPA-MAGVIT2 model retains the standard pixel-level objective. Its input feature dimension is 18 (this likely refers to the MAGVIT2 VAE's internal channel dimension) with patch size 16.

Why this construction over alternatives: The General LAM family serves a specific ablation purpose: it isolates whether the LAPA training paradigm (VQ-VAE over frame differences) adds value on top of pre-trained vision features. If LAPA-DINOv3 significantly outperforms DINOv3 alone, then the VQ-VAE latent action training is contributing something beyond the backbone's native features. If LAPA-DINOv3 underperforms DINOv3, then the VQ-VAE bottleneck is discarding useful information that the backbone already encodes. The results in Tables 1 and 2 provide evidence for the latter: DINOv3 achieves 68.68% average classification accuracy versus LAPA-DINOv3's 49.17%, and 0.19 regression MSE versus 0.60. This suggests that the VQ-VAE compression, at least with the default hyperparameters (small codebook size 8, short sequence length 16), substantially degrades the representation quality that the backbone alone provides.

The paper then uses the General LAM framework as a testbed for ablating the VQ-VAE hyperparameters (codebook size, sequence length, latent dimension) in Section 5.3 and Figure 4, systematically exploring how to close the gap between the compressed (General LAM) and uncompressed (backbone encoder) representations. The findings—that expanding the codebook, increasing sequence length to 49, and scaling latent dimension to 256 substantially narrow the gap—demonstrate that the VQ bottleneck, not the feature-level training paradigm itself, is the primary source of degradation.

Evaluation Settings for Different Model Paradigms

The paper evaluates models under three distinct paradigms (Section 4.1), and the evaluation protocol must handle each differently because the models produce latents in different ways.

Embodied LAMs (LAPA, UniVLA, villa-X): These are purpose-built to output latent actions from frame pairs. For the Atomic Robot and regression tasks (which use image pairs), the latents are extracted directly from the model's standard inference pipeline. For composite classification tasks (which use 9-frame video clips), latents are extracted across adjacent sampled frame pairs and aggregated (likely concatenated or averaged before being fed to the probe).

A critical detail mentioned in Section 4.2: "To retain as much information as possible, we work with the continuous latent embeddings instead of discretized codebook indices, thereby avoiding quantization-related information loss." This means that when a model like LAPA or UniVLA processes a frame pair, the evaluation uses the continuous vector before the nearest-codebook-index lookup—the pre-quantization embedding. This is a fair choice because quantization is a practical trick for autoregressive prediction, not a fundamental property of the action representation; evaluating the continuous embedding tests whether the model learned a good representation, not whether its discretization scheme is well-tuned.

General Vision Encoders (DINOv3, V-JEPA 2, Wan2.2, FLUX.2-dev): These were not designed to output action representations at all. The evaluation must extract features from them and treat those features as if they were latent actions. For semantic encoders (DINOv3, V-JEPA 2), the paper likely uses penultimate-layer features—the feature map before the final projection head. The phrase "evaluated here for their zero-shot temporal modeling capacity" (Section 4.1) means no fine-tuning is done; the encoder processes each frame pair (or each frame in the video clip) independently and outputs features that are then used as the latent representation. For pixel encoders (Wan2.2 VAE, FLUX.2-dev VAE), the latent is the VAE bottleneck representation—the compressed latent that the VAE's decoder would use to reconstruct the frame. The paper notes (Section 4.1) that Wan2.2 and FLUX.2-dev "leverage spatial-temporal compression and pixel-level reconstruction to implicitly capture motion priors," meaning their VAE training on video data may have incidentally learned to encode motion information even though that was not the explicit objective.

An important subtlety: V-JEPA 2 is described as learning "directly from visual latent features instead of pixel features" (Section 4.3), meaning its training objective operates in a latent space (predicting masked latent features from visible ones). This is contrasted with Wan2.2 and FLUX.2-dev, which reconstruct pixels. The paper hypothesizes that this latent-level training is why V-JEPA 2 outperforms pixel-level models.

General LAMs (LAPA-DINOv2, LAPA-DINOv3, LAPA-SigLIP2, LAPA-MAGVIT2): These are evaluated by running them through their standard inference pipeline: the frozen backbone extracts features from the input frames, the VQ-VAE encoder compresses the frame-to-frame feature change into a latent, and the continuous pre-quantization embedding is extracted. The probe is trained on these embeddings identically to how it is trained on Embodied LAM embeddings.

Probe Training Across Model Paradigms

A subtle but important aspect of the evaluation is that the same probe architecture is trained separately for each model's latent space. The probe does not receive the raw video frames—it receives only the frozen latent $z$. This means the probe cannot "see around" a bad representation by learning its own visual features; it is entirely dependent on what the LAM provides. The projector layer (Section 4.2) aligns different latent dimensionalities to a common probe input size, but it is trained per-model, meaning each model gets its own optimal projection. This is necessary because projecting a 1024-D DINOv3 feature to a common space is a fundamentally different mapping than projecting a 32-D LAPA latent token.

The 20-epoch training duration with bfloat16 precision is noteworthy: bfloat16 provides the same dynamic range as float32 but with reduced precision, offering a balance between training speed and numerical stability. The wide hyperparameter sweep (learning rates from 3e-4 to 5e-3, weight decay from 0.01 to 0.8) suggests the authors are searching for optimal probe configurations per model rather than using a fixed configuration, which again ensures that no model is disadvantaged by suboptimal probe hyperparameters.

Data Scale and Composition Summary

To ground the foregoing technical descriptions in concrete numbers, the final curated benchmark comprises (consolidated from Table 1 and Section 3):

• 1,216,760 total classification samples: 25,940 atomic image pairs (LIBERO), plus 1,190,820 composite video clips (692,297 human + 538,423 robot) spanning 151 unique action categories.
• 595,237 total regression trajectories: 209,921 from CALVIN, 112,030 from VLABench, 148,767 from RoboCOIN, 124,519 from AgiBotWorld-Beta.
• Coverage: 11 robotic embodiments (Franka emika, AgileX Cobot, Realman series, AgiBot G1, and others) plus human hands; egocentric and exocentric viewpoints; simulated, laboratory, residential kitchen, commercial, and industrial environments; human activities ranging from frequent (pick, place) to long-tail (shovel snow, float balloon).

The paper emphasizes (Section 3, final paragraph) that the automated curation engine is designed to be extensible: "Beyond this initial curation, our automated engine holds the potential to continuously process future data streams, ensuring the ever-growing nature of the dataset." This positions LARYBench as a living benchmark that can grow as new video datasets and robot platforms become available, rather than a static snapshot.

4. Key Insights and Innovations

Innovation 1: Decoupled Evaluation as a Conceptual Reframing — Latent Action Quality Is a First-Class Property, Not a Policy Component

The paper's most fundamental contribution is not any specific result but rather the conceptual reframing of what it means to evaluate a latent action representation. Before LARYBench, the field treated latent action quality as indistinguishable from downstream policy performance. The evaluation methodology across the literature—from LAPA to UniVLA to GR00T—was uniformly end-to-end: plug the latent action model into a behavior cloning or reinforcement learning pipeline, run the resulting policy on a manipulation benchmark, and report the task success rate. If the policy succeeded, the latent actions were presumed good; if it failed, the diagnosis was ambiguous.

This paper argues—and operationalizes the argument as a benchmark—that latent action quality should be treated as a first-class property of the representation itself, independently measurable without training a full policy. This is conceptually analogous to what ImageNet did for visual representations: before ImageNet, the quality of a visual feature extractor was evaluated by how well a downstream classifier (often a complex, multi-stage pipeline) performed on some application-specific task. ImageNet decoupled this by establishing a standardized, fixed probe head evaluated on a common classification task, so that improvements in feature quality could be directly attributed to the representation rather than to the classifier's architecture or training procedure.

The paper's key insight is that latent actions have two orthogonal quality dimensions that must be evaluated separately because they can be in tension. A representation that perfectly preserves semantic action identity might discard the fine-grained physical detail needed for control (e.g., a discrete VQ-VAE with a small codebook that collapses similar-looking but kinematically distinct motions into the same token). Conversely, a representation that faithfully encodes pixel-level motion might be terrible at distinguishing action categories because it captures all motion equally—the motion of a hand picking up a cup, the motion of a curtain blowing in the wind, and the motion of a camera panning might all produce similar latent vectors if the representation focuses exclusively on low-level optical flow rather than semantic intent. By separating the evaluation into two fixed, lightweight probes—a 4-layer attentive classifier and a residual MLP regressor—the benchmark makes it impossible for a representation to hide behind a powerful policy head or to succeed on one dimension while failing on the other without that failure being explicitly measured.

This is a methodological innovation rather than an algorithmic one, but it is fundamental, not incremental, because it changes what the field considers to be a valid evaluation. Prior evaluation approaches conflated three things: the quality of the latent action representation, the capacity of the policy architecture, and the difficulty of the downstream task. LARYBench disentangles the first from the other two. The implications extend beyond this paper's empirical results: any future work proposing a new latent action model can now be evaluated on LARYBench to determine whether the model actually produces better representations (higher classification accuracy, lower regression MSE) versus whether it simply works better in a specific policy pipeline for contingent reasons (better hyperparameter tuning, a more favorable task distribution, a more powerful decoder).

The benchmark's design also enables a form of diagnostic decomposition that was previously impossible. The paper's error analysis in Section 5—examining performance across the long-tail action distribution, visualizing cross-attention to diagnose spatial grounding failures, and ablating temporal stride to test dynamic trajectory encoding—all depend on having a clean, decoupled signal of representation quality. If the evaluation were still end-to-end policy success, none of these diagnostics would be interpretable because policy failures could be attributed to planner errors, exploration noise, or reward shaping rather than to representation deficiencies.

Innovation 2: The Provocative Empirical Finding That Specialized Embodied Training Degrades Representation Quality — General Vision Encoders as a Strong Upper Bound

The paper's most striking empirical result is not merely that general vision encoders can serve as latent action representations—it is that they systematically and substantially outperform purpose-built Embodied LAMs on both semantic classification and physical control regression. This result is provocative because it inverts the dominant assumption in the VLA field.

The field's operating assumption has been that extracting useful action representations from video requires specialized training: inverse dynamics models to isolate the motion signal from background appearance, vector quantization to produce discrete action tokens suitable for autoregressive prediction, and often auxiliary objectives like language grounding (UniVLA) or trajectory priors (LatBot) to improve physical fidelity. The logic is intuitive: general vision models are trained to recognize objects and scenes, not to represent actions, so they should require domain-specific adaptation to be useful for robotic control. The entire LAM research program—LAPA, Moto, LAPO, CoMo, UniVLA, villa-X—is built on this premise.

LARYBench's results challenge this premise at its foundation. V-JEPA 2—a model trained with self-supervised video prediction, with no inverse dynamics head, no action labels, and no robotics-specific data—achieves 76.62% average classification accuracy across all tasks (Table 1), compared to LAPA's 20.17%, UniVLA's 17.99%, and villa-X's 20.90%. On regression, DINOv3 achieves 0.19 average MSE compared to LAPA's 0.97, UniVLA's 0.87, and villa-X's 0.87 (Table 2). These are not marginal improvements—they are multiplicative gaps that suggest the specialized training is not merely failing to add value but is actively destroying information that the general vision backbone already encodes.

The mechanism of this degradation is hinted at by the paper's analysis. The General LAM experiments (LAPA-DINOv3, LAPA-SigLIP2) provide the key diagnostic: taking a strong frozen vision backbone and applying the LAPA VQ-VAE training paradigm reduces classification accuracy from DINOv3's 68.68% to LAPA-DINOv3's 49.17% (Table 1) and increases regression MSE from DINOv3's 0.19 to LAPA-DINOv3's 0.60 (Table 2). The VQ bottleneck—the quantization step that compresses continuous features into discrete tokens—is the primary source of this degradation, as the ablation studies in Figure 4 and Tables 3-5 demonstrate: expanding the codebook size, increasing the sequence length, and scaling the latent dimension can partially close the gap, but the uncompressed backbone remains superior.

This finding represents a fundamental empirical challenge to the LAM research program, not an incremental refinement. If specialized latent action training on scarce robotic data systematically produces worse representations than off-the-shelf vision encoders trained on internet-scale data with no action supervision, then the field's resource allocation—investing in better LAM architectures rather than better general vision models—may be misdirected. The paper is careful not to overstate this (it is framed as a hypothesis to be tested rather than a definitive conclusion), but the evidence is consistent and substantial across 11 models, multiple tasks, and multiple embodiments.

The finding also explains—retrospectively—the puzzlingly low absolute performance of Embodied LAMs. A classification accuracy of ~20% on a 151-way task is barely above random chance (which would be ~0.66%). This suggests that these models, despite being the state of the art in the VLA literature, are failing to capture even the most basic semantic structure of actions. The paper's cross-attention visualizations (Figure 6, extended in Appendix D.2) provide a mechanistic explanation: Embodied LAMs like LAPA produce "broad, uninformative attention blobs" that fail to localize the interacting objects, whereas V-JEPA 2 and DINOv3 produce "sharp, geometry-aware focus on the active end-effector." The representation collapse is not a subtle degradation—it is a catastrophic failure of spatial grounding.

Innovation 3: The Latent-vs-Pixel Dichotomy as an Organizing Principle — Why Feature-Level Training Is Fundamentally Better Aligned to Physical Action

The paper's third conceptual contribution is to surface and empirically validate a dichotomy that was implicit in prior work but never systematically tested: the distinction between representations learned at the pixel level (by reconstructing raw video frames) versus those learned at the feature level (by operating in a pre-trained visual encoder's latent space). This is not merely a taxonomy—it is a claim about which type of representation is fundamentally better aligned to physical action space, with evidence suggesting that feature-level training provides a more direct pathway from vision to action.

Prior work had explored both approaches without explicitly comparing them. Pixel-level models—exemplified by video VAEs like MAGVIT2, Wan2.2, and FLUX.2-dev, as well as world models that generate future frames—operate on the assumption that reconstructing pixels forces the model to capture all the information needed for action, because pixels contain everything. Feature-level models—exemplified by V-JEPA 2's latent-space prediction objective and DINOv3's contrastive learning in feature space—operate on the assumption that discarding photometric details (texture, lighting, background appearance) enables the model to focus on structural information that is more relevant to action.

LARYBench provides the first head-to-head comparison of these paradigms on the same evaluation tasks. The results are consistent across both classification and regression. In classification (Table 1): semantic feature-level encoders (V-JEPA 2 at 76.62%, DINOv3 at 68.68%) substantially outperform pixel-level encoders (Wan2.2 at 49.36%, FLUX.2-dev at 47.48%). In regression (Table 2): the same pattern holds, with DINOv3 at 0.19 MSE versus Wan2.2 at 0.30 and FLUX.2-dev at 0.35. The paper's interpretation (Section 4.3, in the "Do Latent Actions Encode Enough for Control?" subsection) is direct: "latent-based visual space is better aligned to robotic action space."

Why should this be the case? The paper does not fully explain the mechanism, but the results suggest an answer. Pixel-level reconstruction allocates model capacity to representing everything in the scene equally—the robot arm, the object, the table, the background, the lighting—because the pixel-space loss function penalizes errors on all pixels uniformly. But for action understanding, only a small fraction of the pixels matter: the regions corresponding to the agent's end-effector and the object being manipulated. Feature-level training, particularly contrastive learning (DINOv3) and latent-space prediction (V-JEPA 2), can learn to selectively prioritize action-relevant features because the training objective (discriminating between views, predicting masked features in context) naturally encourages the model to focus on what changes meaningfully between frames rather than what stays the same.

This insight has direct implications for the design of future VLA systems. The paper explicitly advocates (Section 6) for "a paradigm shift in Vision-Language-Action (VLA) design: instead of learning action spaces from limited robotic data, future research should focus on aligning control policies with the robust feature spaces of general vision models." This is a stronger claim than simply saying general vision models work better—it is a claim about which architectural pathway is more promising. The pixel-level pathway (encode → reconstruct pixels → decode to actions), exemplified by recent video world model approaches, may be fundamentally less efficient than the feature-level pathway (encode → operate in latent space → decode to actions), because the pixel reconstruction step forces the model to solve a harder problem (photorealistic generation) that is not necessary for action understanding.

The stride ablation in Section 5.3 (Table 6) provides additional evidence for this interpretation. Pixel-level FLUX.2-dev achieves excellent regression at stride=5 (0.04 MSE, the best of all models) but catastrophically degrades at stride=30 (0.62 MSE), while feature-level V-JEPA 2 and DINOv3 remain relatively stable (0.16 and 0.25 respectively at stride=30). This suggests that pixel-level models capture short-range motion well (because nearby frames look similar and pixel reconstruction is easy) but fail to encode the dynamic trajectory over longer horizons, whereas feature-level models capture something more abstract and temporally robust. The paper phrases this as latent action spaces encoding "robust dynamic trajectories" that are "uniquely robust for long-horizon intents."

Innovation 4: The Long-Tail and Mid-Frequency Semantic Aliasing as Diagnostic Concepts for Representation Failure

The paper's error analysis introduces a set of diagnostic concepts that help explain why representations fail in specific, interpretable ways. These are not new methods but new ways of thinking about and decomposing representation errors, and they are enabled by the benchmark's decoupled evaluation design.

The Long-Tail Dilemma (Section 5.1, Figure 5) refers to the observation that while stronger models (V-JEPA 2, DINOv3) outperform weaker ones (LAPA) across the full action frequency distribution, the performance gap widens as action frequency decreases. This means the advantage of general vision encoders over Embodied LAMs is not uniform—it is concentrated in the long tail of rare actions. For common actions like "pick" and "place," even Embodied LAMs achieve non-trivial accuracy because the training data (limited as it is) contains enough examples. For rare actions like "shovel" (snow) or "float" (balloon), Embodied LAMs fail entirely while general vision encoders maintain some level of performance.

The paper interprets this through the lens of data diversity: "stronger models exhibit better generalization capabilities in long-tail scenarios" (Section 5.1). But the underlying conceptual point is more interesting: data scarcity in Embodied LAM training does not merely reduce overall performance—it selectively destroys the representation's ability to handle rare events. This has practical implications for robotics, where the most safety-critical scenarios (unusual object interactions, rare failure modes) are precisely the ones that appear least frequently in training data. A latent action model that collapses on long-tail actions is not just quantitatively worse—it is qualitatively unsafe in a way that aggregate metrics might mask.

Mid-Frequency Semantic Aliasing (Section 5.1, title) is a subtler phenomenon. Even for actions of moderate frequency that appear in both human and robot training data, representations can confuse distinct action categories that share visual similarities but have different physical semantics. The cross-domain gap analysis (Figure 7, Appendix D.1) provides concrete examples: for the action "roll," LAPA-DINOv3 achieves 0.90 F1 on human videos but only 0.04 F1 on robot videos. This is not because "roll" is rare (it appears in both domains) but because the visual appearance of rolling—a human hand rotating an object versus a robot gripper rotating an object—is sufficiently different that the representation fails to recognize them as the same action. Meanwhile, DINOv3 achieves 0.80 F1 on human and 0.63 on robot for the same action, demonstrating that robust visual priors can bridge the morphological gap.

This diagnostic concept—semantic aliasing across embodiments—is important because it directly challenges the cross-embodiment transfer hypothesis that motivates learning from human videos. If a latent action representation cannot recognize that a human "roll" and a robot "roll" are the same action, then learning from human videos provides no transfer benefit for that action. The paper's General LAMs (LAPA-DINOv3, etc.) show a partial improvement—they achieve higher overall F1 than Embodied LAMs but still exhibit a "distinct preference for human embodiments" (Appendix D.1), suggesting that frozen visual backbones carry human-centric biases from their internet pretraining data.

Spatiotemporal Grounding Failure (Section 5.2, Figure 6) introduces a third diagnostic concept: the idea that representation quality can be decomposed into whether the model looks at the right thing (spatial grounding) and whether it tracks the right thing over time (temporal grounding). The cross-attention visualizations provide direct evidence that predictive failure is correlated with attention dispersion. Embodied LAMs produce "broad, uninformative attention blobs" while V-JEPA 2 "exhibits the most accurate attention distribution, sharply localizing on the exact interaction points." This suggests that the representation degradation from specialized LAM training is not just a matter of lost information—it is a failure of attention to focus on the dynamically interacting objects, which is a specific, interpretable mechanism that future work could target for improvement.

These diagnostic concepts are significant beyond this paper because they provide a vocabulary and a methodology for analyzing representation failures that the field previously lacked. Prior work could say "the policy fails on this task" but not "the representation fails because its attention is diffuse on mid-frequency actions across embodiments." The benchmark's decoupled design makes such decomposition possible, and the paper's error analysis demonstrates how to use it.

5. Experimental Analysis

Evaluation Methodology

• Dataset. LARYBench comprises over 1.2M video clips (totaling >1,000 hours), 620K image pairs, and 595K motion trajectories spanning 151 unique action categories, curated from 11 source datasets including LIBERO, Ego4D, SSv2, HoloAssist, EPIC-KITCHENS, TACO, EgoDex, AgiBotWorld-Beta, CALVIN, VLABench, and RoboCOIN (Section 3, Table 1). The data spans human and robotic agents from egocentric and exocentric perspectives across simulated and real-world environments. For classification tasks, atomic actions use a 75:25 stratified split and composite actions use a 70:30 stratified split; for regression, CALVIN uses an environment-level split (A/B/C for training, D for validation) while VLABench, RoboCOIN, and AgiBotWorld-Beta use standard 75:25 splits (Section 4.2).

• Base models. The paper evaluates 11 models across three paradigms (Section 4.1): Embodied LAMs (LAPA with 343.80M parameters, UniVLA with 287.75M, villa-X with 238.71M), purpose-built for robot manipulation via VQ-VAE plus inverse/forward dynamics modeling; General Vision Encoders (V-JEPA 2 with 303.89M parameters, DINOv3 with 303.13M, Wan2.2 VAE with 704.69M, FLUX.2-dev VAE with 84.05M), off-the-shelf models trained without action supervision; and General LAMs (LAPA-MAGVIT2 with 116.40M, LAPA-SigLIP2 with 200.83M, LAPA-DINOv2 with 473.69M, LAPA-DINOv3 with 472.45M), which graft the LAPA VQ-VAE training paradigm onto frozen pre-trained vision backbones. These models were chosen to span the space from domain-specific embodied training to domain-agnostic visual pretraining, enabling direct comparison of whether specialized action training adds value beyond general visual features (Section 4.1).

• Metrics. For semantic action classification, the paper reports Top-1 Accuracy averaged across all action categories (macro-average, giving equal weight to rare and frequent actions) (Section 4.2). This is computed by training a 4-layer attentive probe on frozen latent representations and measuring the fraction of correctly predicted action labels on the held-out test split. For low-level control regression, the paper reports Mean Squared Error (MSE) between predicted and ground-truth end-effector trajectories, computed per trajectory dimension and averaged across all samples and all action degrees of freedom (7-DoF, 12-DoF, or 16-DoF depending on the robot embodiment) (Section 4.2). Lower MSE indicates better physical grounding of the latent representation.

• Baselines. The primary baselines are the three Embodied LAMs: LAPA (Ye et al., 2024), UniVLA (Bu et al., 2025), and villa-X (Chen et al., 2025), representing the state-of-the-art in purpose-built latent action extraction from videos (Section 4.1). These are compared against General Vision Encoders (DINOv3, V-JEPA 2, Wan2.2, FLUX.2-dev) as zero-shot baselines that have never been trained on action-specific objectives, and against General LAMs (LAPA-DINOv2, LAPA-DINOv3, LAPA-SigLIP2, LAPA-MAGVIT2) as hybrid baselines that combine general visual backbones with the LAM training paradigm (Section 4.1). There is no "majority voting" or random baseline explicitly reported for classification, though with 151 classes, random chance would be approximately 0.66% accuracy.

• Generation budget / compute accounting. The evaluation does not measure compute in FLOPs or generations—instead, it measures representation quality directly through probe accuracy and regression MSE on frozen latent representations (Section 4.2). All models are evaluated under identical protocols: the same input frames, the same probe architectures, and the same training budgets for the probes (20 epochs, bfloat16 precision, hyperparameter sweep over learning rates and weight decay). The "cost" of different representations is implicitly captured by their parameter counts (reported in Tables 1 and 2), but no explicit compute-matched comparison is performed. For the General LAM training (Appendix C.2), all variants use exactly 100,000 training steps on 8 NVIDIA H800 GPUs, ensuring a fair comparison within that family.

• Cross-validation / statistical protocol. The paper does not report traditional k-fold cross-validation or confidence intervals on benchmark results. Instead, it employs several forms of robustness checking: (i) stratified sampling for classification splits to preserve class distributions across train and test sets (Section 4.2); (ii) environment-level splits for CALVIN regression to test out-of-distribution generalization (environments A/B/C for training, D held out); (iii) a hyperparameter sweep for probe training ("multi-head optimization strategy with learning rates from 3e-4 to 5e-3 and weight decay values from 0.01 to 0.8") to ensure no model is disadvantaged by suboptimal probe configuration (Section 4.2); and (iv) ablation studies on General LAM hyperparameters (codebook size, sequence length, latent dimension) to verify that findings are not artifacts of specific quantization settings (Section 4.3, Tables 3-5). The absence of confidence intervals or standard deviations on the reported accuracy and MSE numbers is a limitation—the reader cannot assess whether the performance differences between models (e.g., V-JEPA 2 at 76.62% vs. DINOv3 at 68.68%) are statistically significant or within sampling noise, particularly given that some test splits (e.g., atomic actions with 25,940 samples split 75:25) may produce relatively small validation sets for certain rare action categories.

Main Quantitative Results

Do Latent Actions Capture Diverse Actions?

The classification results in Table 1 establish a clear and consistent hierarchy across all evaluated models. The headline finding is that general vision foundation models, trained without any action supervision, dramatically outperform specialized Embodied LAMs on action classification. V-JEPA 2 achieves 76.62% average accuracy across the three classification tasks (Atomic Robot, Composite Human, Composite Robot), compared to LAPA's 20.17%, UniVLA's 17.99%, and villa-X's 20.90%—a gap of approximately 56 percentage points, or nearly 4× higher accuracy (Table 1). This is not a marginal improvement; it is a qualitative difference between representations that capture meaningful action structure and those that are barely above random chance.

Breaking down by task reveals important patterns. On Atomic Robot classification (28 fine-grained kinematic primitives like directional translations and gripper states), V-JEPA 2 achieves 79.09% accuracy versus LAPA's 22.27% and villa-X's 15.00% (Table 1). This task tests the finest spatial resolution—distinguishing "move forward left up" from "move forward right up"—and the fact that general vision encoders can make these distinctions without any action-specific training suggests that self-supervised visual pretraining inherently learns to encode fine-grained spatial displacements.

On Composite Human classification (123 action categories from egocentric human videos), V-JEPA 2 achieves 80.35% versus LAPA's 14.61% and UniVLA's 19.08% (Table 1). The 80.35% accuracy is remarkably high for a 123-way classification task, indicating that V-JEPA 2's video prediction pretraining has produced representations that are semantically well-aligned with human action categories, even though the model was never trained to predict action labels.

On Composite Robot classification (54 action categories from AgiBotWorld-Beta bimanual robot demonstrations), the gap narrows somewhat: V-JEPA 2 achieves 70.43% versus LAPA's 23.64%, but villa-X achieves 29.90%—its best result and the strongest performance among Embodied LAMs (Table 1). This relative improvement for villa-X on robot-specific actions is consistent with its design as a VLA model trained on robotic data, but it still underperforms general vision encoders by 40+ percentage points.

A critical secondary finding concerns General LAMs—the hybrid models that apply the LAPA VQ-VAE training paradigm to frozen general vision backbones. LAPA-DINOv2 achieves 49.36% average accuracy, which is substantially better than Embodied LAMs (20.90% for villa-X) but substantially worse than its backbone, DINOv3, which achieves 68.68% (Table 1). The gap between DINOv3 (68.68%) and LAPA-DINOv3 (49.17%)—approximately 19.5 percentage points—represents the degradation caused by the VQ-VAE bottleneck, which compresses continuous, high-dimensional features into discrete action tokens. Similarly, LAPA-SigLIP2 achieves 43.67% and LAPA-MAGVIT2 achieves 40.78%, both significantly below their backbone encoders' potential (Table 1).

The pixel-level generative encoders present an intermediate case: Wan2.2 achieves 49.36% and FLUX.2-dev achieves 47.48% (Table 1). These are substantially better than Embodied LAMs but worse than semantic feature-level encoders (V-JEPA 2, DINOv3), supporting the paper's claim that "latent-based visual space is better aligned to robotic action space" (Section 4.3). The key distinction is not "trained on more data" (FLUX.2-dev has 84M parameters, far fewer than DINOv3's 303M) but rather what the model is trained to do: reconstruct pixels versus operate in a learned feature space.

The paper explicitly interprets these results in Section 4.3: "This phenomenon demonstrates that the visual self-supervised training with large-scale data inherently yields general semantic action representations covering both robot and human data." The phrase "inherently yields" is a strong claim—it suggests that action understanding is not something that needs to be explicitly trained; it emerges naturally from large-scale visual pretraining objectives like masked feature prediction (V-JEPA 2) or contrastive learning (DINOv3).

Do Latent Actions Encode Enough for Control?

The regression results in Table 2 mirror the classification hierarchy: general vision encoders consistently achieve lower MSE (better physical fidelity) than Embodied LAMs or General LAMs, confirming that physical control information, like semantic information, is better preserved by general visual pretraining than by specialized action extraction training.

The headline numbers: DINOv3 achieves 0.19 average MSE across all four regression datasets (CALVIN, VLABench, RoboCOIN, AgiBotWorld-Beta), while V-JEPA 2 achieves 0.25, compared to LAPA's 0.97, UniVLA's 0.87, and villa-X's 0.87 (Table 2). The gap is approximately 5×—DINOv3's representations preserve roughly five times more trajectory-predictive information than the best Embodied LAM.

Breaking down by dataset reveals embodiment-specific patterns. On CALVIN (single-arm Franka, exocentric, 7-DoF), DINOv3 achieves 0.22 MSE versus LAPA's 0.96 and UniVLA's 0.82 (Table 2). On VLABench (single-arm, exocentric, 7-DoF), DINOv3 achieves 0.06 MSE—an exceptionally low error—versus LAPA's 0.95 and villa-X's 0.72. The 0.06 MSE on VLABench is notable because it represents near-perfect trajectory prediction from a single latent vector, suggesting that DINOv3's feature space encodes precise spatial displacements with minimal noise.

On RoboCOIN (bimanual, egocentric, 12-DoF, 10 diverse platforms), FLUX.2-dev achieves 0.04 MSE—the single best result on any regression dataset (Table 2). This is a striking exception to the general pattern of semantic encoders outperforming pixel encoders, and it warrants careful interpretation. FLUX.2-dev is a generative VAE trained for image synthesis; its strong performance on RoboCOIN may reflect the dataset's particular characteristics (egocentric bimanual manipulation with relatively constrained motion patterns) rather than general superiority. The paper does not analyze this anomaly in detail, but the full stride ablation in Table 6 provides context: FLUX.2-dev's performance degrades catastrophically with longer strides (0.62 MSE at stride=30), suggesting its strength is in short-range pixel correspondence rather than robust dynamic trajectory encoding.

On AgiBotWorld-Beta (semi-humanoid bimanual AgiBot G1, egocentric, 16-DoF), DINOv3 achieves 0.22 MSE versus LAPA's 0.96 and villa-X's 0.97 (Table 2). This is the highest-dimensional action space (16 DoF × 5 stride = 80 continuous values predicted from a single latent vector), making the 0.22 MSE particularly impressive—it indicates that the latent representation encodes sufficient geometric information to reconstruct complex bimanual trajectories including absolute position, quaternion orientation, and gripper states.

The General LAMs again underperform their backbone encoders. LAPA-DINOv3 achieves 0.60 average MSE versus DINOv3's 0.19—approximately 3× worse (Table 2). LAPA-DINOv2 (0.63), LAPA-SigLIP2 (0.65), and LAPA-MAGVIT2 (0.65) show similar degradation. This consistent pattern across all four General LAM variants and all four regression datasets strongly suggests that the VQ-VAE compression is the primary source of information loss, not any specific backbone choice.

The paper's interpretation of these results (Section 4.3) is direct: "latent-based visual space is better aligned to robotic action space." The comparison between DINOv3 (0.19 MSE) and Wan2.2 (0.30 MSE) supports this: both are general vision encoders with similar parameter counts (~303M vs ~704M), but DINOv3's feature-level contrastive training produces features that are fundamentally better suited to predicting physical trajectories than Wan2.2's pixel-level reconstruction training.

A more nuanced finding emerges from comparing DINOv3 (0.19 MSE) with V-JEPA 2 (0.25 MSE). Although V-JEPA 2 dominates classification (76.62% vs. DINOv3's 68.68%), DINOv3 dominates regression. The paper attributes this to DINOv3's roots in visual contrastive learning, which "retains fine-grained recognition capabilities, which in turn improves the precision of fine-grained regression in robotic control tasks" (Section 4.3). This suggests that the two evaluation axes—semantic classification and physical regression—are genuinely orthogonal; a representation that excels at one may not excel at the other, and the optimal representation for a VLA system may depend on whether semantic accuracy or physical precision is more important for the target application.

What Constitutes an Effective Latent Action Model?

The ablation studies on the General LAM framework (Section 4.3, Figure 4, Tables 3-5) systematically explore how VQ-VAE hyperparameters affect representation quality, using LAPA-DINOv3 as the base architecture. The starting configuration (codebook size=8, sequence length=16, latent dimension=32, denoted with * in Figure 4) achieves moderate performance: approximately 55% Composite Human classification accuracy and approximately 0.88 MSE on AgiBotWorld-Beta regression (read from Figure 4).

Codebook size ablation (Table 3): Increasing the codebook from 8 to 64 improves regression performance (MSE drops from 0.88 to 0.83 on AgiBotWorld-Beta) without harming classification (Composite Human remains stable at 70.15–71.31%, Composite Robot at 63.82–64.89%). However, further increasing to 256 causes codebook utilization to drop from 100% to 89.5%, and regression MSE slightly increases to 0.85. The paper interprets this as evidence that "overly large sizes... cause codebook utilization drops without further gains, making a moderate size (cs=64) optimal for dense representations" (Section 4.3). This is a standard finding in VQ-VAE literature: beyond a certain codebook size, many codes go unused (the "codebook collapse" problem), and the effective codebook size is smaller than the nominal size.

Sequence length ablation (Table 4): This is the most dramatic ablation. At sequence length=16 (the default), codebook utilization collapses to 1.6%—a catastrophic failure where the VQ-VAE maps almost all inputs to a single code, producing a degenerate latent space. Increasing to sequence length=49 restores 100% utilization and substantially improves classification (Composite Human from 69.23% to 70.15%, Composite Robot from 63.33% to 64.04%). Further increasing to 64 yields mixed results: codebook utilization drops to 79.7% but classification accuracy improves (71.37% Composite Human, 64.70% Composite Robot) and regression MSE decreases to 0.72. The paper's interpretation: "short sequences trigger catastrophic codebook collapse, whereas a moderate length ensures 100% utilization and robust generalization" (Section 4.3). The mechanism is likely that short sequences provide insufficient temporal diversity for the VQ-VAE to learn meaningful discrete codes—with only 16 temporal positions, the model cannot distinguish between different types of motion and collapses to a trivial solution.

Latent dimension ablation (Table 5): Scaling latent dimension from 32 to 256 improves performance (Composite Human from 60.94% to 70.15%, Composite Robot from 57.69% to 64.04%, MSE from 0.87 to 0.83), but further increases to 512 cause codebook utilization to collapse to 1.6% while classification continues improving (Composite Human: 71.25%, Composite Robot: 64.97%). At dimension 1024, utilization partially recovers to 84.4% and performance peaks (72.55% Composite Human, 65.78% Composite Robot, 0.81 MSE). The paper notes that "scaling the latent dimension improves theoretical capacity but introduces quantization instability, evidenced by sudden utilization collapses at intermediate sizes" and concludes that "dim=256 strikes the best capacity-stability balance" (Section 4.3). The non-monotonic relationship between dimension and utilization is a known phenomenon in VQ-VAE training, related to the initialization and optimization dynamics of the codebook vectors in high-dimensional spaces.

The paper synthesizes these ablations into a two-component recipe: "an effective latent action space requires two key components: robust self-supervised visual priors to capture precise spatial-temporal dynamics, and a strictly regularized quantization bottleneck to maintain stability and dense utilization" (Section 4.3). Figure 4 visually maps this as a "performance evolution path bridging the gap between the worst baseline (LAPA) and the continuous upper bound (V-JEPA2)," showing how each hyperparameter improvement incrementally closes the gap but never fully reaches the uncompressed encoder's performance.

Ablation Studies and Robustness Checks

• Codebook size (Table 3): Expanding codebook size from 8 to 64 improves regression MSE from 0.88 to 0.83 on AgiBotWorld-Beta without degrading classification (Composite Human: 71.31% vs. 70.15%); however, codebook size 256 reduces utilization to 89.5% with no additional gains, establishing a moderate codebook as optimal. This demonstrates that the VQ bottleneck's capacity is saturable—beyond a point, adding more codes provides no benefit because the training data does not contain enough distinct motion patterns to populate them.

• Sequence length (Table 4): Sequence length 16 causes catastrophic codebook collapse (1.6% utilization), while length 49 restores 100% utilization and length 64 yields the best regression MSE (0.72 on AgiBotWorld-Beta) but with reduced utilization (79.7%), indicating that temporal diversity is critical for preventing VQ-VAE degeneracy. This is a non-obvious finding: the sequence length does not just affect temporal modeling capacity—it determines whether the VQ-VAE can learn a non-trivial codebook at all.

• Latent dimension (Table 5): Scaling from 32 to 256 improves classification (Composite Human: 60.94% → 70.15%) without instability, but dimension 512 collapses utilization to 1.6% while dimension 1024 partially recovers (84.4% utilization) and achieves peak classification (72.55%), revealing non-monotonic stability behavior in VQ training. The recovery at very high dimensions is intriguing—it suggests that the collapse at 512 is a training dynamics phenomenon rather than a fundamental capacity limitation, possibly related to how codebook vectors are initialized and optimized in different-dimensional spaces.

• Sampling stride for regression robustness (Table 6): Pixel-level FLUX.2-dev excels at stride=5 (0.04 MSE, best overall) but catastrophically degrades to 0.62 MSE at stride=30, while latent action models (Embodied LAMs and General LAMs) maintain stability (LAPA-DINOv3: 0.25 → 0.26 MSE, LAPA: 0.95 → 0.77 MSE), proving that the latent action paradigm encodes temporally robust dynamic trajectories rather than static correspondences. The LAPA model's MSE actually improves with longer strides (0.95 → 0.77), which may indicate that its representations are so noisy that the longer trajectory provides more signal for the regressor to latch onto, or that short-range LAPA latents are dominated by noise while longer-range latents capture coarser but more reliable motion signals.

• Cross-embodiment generalization (Figure 7, Appendix D.1): General LAMs (LAPA-DINOv3, LAPA-SigLIP2) exhibit a human-centric preference (e.g., for "roll": 0.90 F1 on human vs. 0.04 on robot for LAPA-DINOv3), while DINOv3 maintains balanced cross-domain performance, and Embodied LAMs (LAPA, UniVLA) show a slight robotic preference due to their robot-dominated training data, demonstrating that training data distribution fundamentally shapes cross-embodiment transfer capabilities. This is a negative result for the General LAM approach: despite training on internet data containing both human and non-human motions, the frozen visual backbones carry human-centric biases that the VQ-VAE training does not overcome.

• Spatiotemporal grounding visualization (Figure 6, Appendix D.2): V-JEPA 2 produces the most accurate cross-attention maps, sharply localizing on interaction points; DINOv3 maintains geometry-aware focus on the active end-effector; pixel encoders (FLUX.2-dev, Wan2.2) exhibit dispersed, unfocused attention; Embodied LAMs (LAPA, UniVLA, villa-X) produce broad, uninformative attention blobs; and General LAMs (LAPA-DINOv2) partially inherit backbone localization capabilities, establishing that predictive failure correlates with attention dispersion. The paper's claim that "predictive failure fundamentally stems from an inability to concentrate attention on the corresponding dynamically interacting objects" (Section 5.2) is directly supported by the qualitative visualizations in Figures 6, 8, 9, 10, and 11, which show consistent patterns across multiple action categories.

• Action Expert architecture simplicity: The regression head is deliberately a "standard MLP with residual connections" (2 blocks, hidden dimension 4096) with no attention, recurrence, or convolutional structure (Section 4.2). This is a robustness check by design: any spatiotemporal reasoning must be encoded in the latent z itself, confirming that the regression MSE differences reflect representation quality rather than decoder capacity.

Critical Assessment

Do the experiments support the central claim that general vision foundation models consistently outperform specialized Embodied LAMs?

This is the paper's strongest and most directly supported claim. The classification results (Table 1) and regression results (Table 2) provide consistent, cross-task, cross-embodiment evidence that V-JEPA 2 and DINOv3 substantially outperform LAPA, UniVLA, and villa-X on every single metric reported. The gaps are large enough (56 percentage points in classification, ~5× in regression MSE) that even without reported confidence intervals, the direction and magnitude of the effect are unambiguous. The General LAM results further strengthen the claim by showing that the VQ-VAE training paradigm—the defining feature of Embodied LAMs—actually degrades the quality of representations extracted from strong visual backbones.

However, the scope of this claim is narrower than it might appear. The evaluation covers exactly three Embodied LAMs (LAPA, UniVLA, villa-X) and tests them only on the specific latent representations they produce out-of-the-box, with no fine-tuning or adaptation for the benchmark tasks. A defender of the LAM paradigm could argue that these models are designed for end-to-end policy learning, and that their latent representations are optimized for autoregressive behavior cloning rather than for the probe-based classification and regression tasks the benchmark uses. The continuous pre-quantization embeddings used in the evaluation (Section 4.2) may not be the intended representation for downstream use; the discrete codebook indices might capture different information that is more useful for policy learning even if less useful for probe-based decoding. The benchmark would be strengthened by also evaluating the discrete codebook indices (with appropriate modifications to the probe architecture) to test whether the quantization that makes autoregressive prediction possible also preserves semantically meaningful structure.

Additionally, the Embodied LAMs are trained on much smaller datasets than the general vision encoders. The paper acknowledges this in Section 4.3 ("due to the limited amount of training data or the early constraints to low-level actions") but does not perform a controlled experiment that isolates data scale from training paradigm. A comparison between LAPA-DINOv3 (trained on 2M+ internet frames) and DINOv3 itself conflates two variables: the VQ-VAE bottleneck and the training data distribution. To isolate the bottleneck effect, one would need to train LAPA from scratch on the same internet-scale data used for the General LAMs, but this experiment was not performed. Without it, we cannot be certain whether Embodied LAMs fail because of their VQ-VAE architecture or because of their training data scarcity.

Do the experiments support the claim that latent-based visual spaces are fundamentally better aligned to physical action than pixel-based spaces?

This claim is supported but with important caveats. The comparison between DINOv3 (latent feature space, 0.19 MSE) and Wan2.2/FLUX.2-dev (pixel space, 0.30/0.35 MSE) in Table 2 shows a consistent advantage for feature-level representations on regression. In classification (Table 1), V-JEPA 2 and DINOv3 (76.62%, 68.68%) similarly outperform Wan2.2 and FLUX.2-dev (49.36%, 47.48%).

However, the pixel-level encoders evaluated are not the strongest possible representatives of their paradigm. Wan2.2 and FLUX.2-dev are primarily image/video generation models, not models designed for representation learning. A more competitive pixel-level baseline would be a model like VideoMAE or a video-pretrained ViT that uses pixel reconstruction as a pretraining objective but is evaluated on its representation quality, similar to how masked autoencoders (He et al., 2022) have shown strong representation learning through pixel reconstruction. The paper's pixel-level baselines conflate "trained to reconstruct pixels" with "using a VAE bottleneck designed for generation," and the relatively poor performance may reflect the VAE bottleneck (similar to how the VQ bottleneck degrades General LAM performance) rather than the pixel-level training objective itself.

Furthermore, FLUX.2-dev's anomalously strong performance on RoboCOIN (0.04 MSE at stride=5, Table 6) and Wan2.2's strong Composite Robot classification (65.39%, competitive with DINOv3's 69.06%, Table 1) suggest that the latent-vs-pixel distinction is not absolute—pixel-level models can excel in specific regimes. The stride ablation in Table 6 provides the most nuanced picture: pixel-level models are strong at short temporal horizons but degrade rapidly with longer horizons, while feature-level models are more robust. The paper's claim of "fundamentally better alignment" would be more precisely stated as "better alignment for temporally extended, dynamic trajectories," which the evidence supports, rather than "better alignment in all cases," which FLUX.2-dev's stride=5 performance contradicts.

Do the ablation studies support the claimed recipe for effective latent action models ("robust self-supervised visual priors + strictly regularized quantization bottleneck")?

The ablations in Tables 3-5 and Figure 4 provide solid support for the importance of the quantization bottleneck. The catastrophic codebook collapse at sequence length=16 (Table 4) and the non-monotonic utilization behavior with latent dimension (Table 5) demonstrate that VQ-VAE training is highly sensitive to hyperparameter choices and that poor choices can destroy representation quality entirely. The moderate codebook size (64), sufficient sequence length (49+), and balanced latent dimension (256) that the paper recommends are empirically justified.

However, the "robust self-supervised visual priors" half of the recipe is less thoroughly ablated. The General LAM experiments compare DINOv3, DINOv2, SigLIP2, and MAGVIT2 backbones, but these differ along multiple axes simultaneously: training objective (contrastive vs. vision-language alignment vs. pixel reconstruction), training data, model architecture, and feature dimensionality. The fact that DINOv3-based models outperform MAGVIT2-based models (Table 1: 49.17% vs. 40.78% for General LAMs) could reflect any of these differences. A controlled ablation that varies only the pretraining objective while holding data, architecture, and dimensionality constant would be needed to establish that "self-supervised" (as opposed to "vision-language aligned" or "reconstructive") is the key property.

Additionally, the General LAM models are trained for exactly 100,000 steps (Appendix C.2) with a fixed initial quantization configuration, and then ablated by varying individual hyperparameters. This does not test whether the optimal hyperparameters interact—for example, whether the ideal codebook size depends on the sequence length or latent dimension. A factorial experiment varying multiple hyperparameters simultaneously, or a compute-matched comparison where models with different hyperparameters are trained to convergence, would provide stronger evidence for the recommended configuration.

What experiments are missing that would strengthen the paper?

The most significant omission is the lack of confidence intervals or statistical significance tests on any reported results. With classification test sets of varying sizes (25,940 atomic samples split 75:25 gives ~6,500 test samples, but composite tasks with 70:30 splits give much smaller per-class test sets for the long tail of rare actions), the variance in accuracy estimates could be substantial, particularly for macro-averaged metrics where rare classes with few test samples contribute equally to the average. A 2-3 percentage point difference between models (e.g., LAPA-DINOv2's 49.36% vs. LAPA-DINOv3's 49.17%) may not be statistically significant, but the paper treats all differences as meaningful without qualification.

Second, the paper does not evaluate what happens when the probe architecture is varied. The 4-layer attentive probe is a specific architectural choice; would a linear probe produce the same ranking of models? Would a deeper probe allow Embodied LAMs to close the gap, suggesting that their representations are informative but require more decoding capacity? The benchmark's claim to decouple representation quality from downstream architecture would be stronger if it demonstrated that the ranking is robust to probe architecture choices.

Third, there is no compute-matched comparison between Embodied LAMs and General LAMs. The General LAMs are trained on 2M+ internet frames for 100,000 steps, while the Embodied LAMs (LAPA, UniVLA, villa-X) are evaluated as-is from their original papers, with whatever training data and compute budget their authors used. A defender of Embodied LAMs could argue that given the same large-scale, diverse training data, LAPA would perform much better than it does on LARYBench. The paper does not test this because training LAPA from scratch on internet-scale data was presumably infeasible, but the absence of this experiment means we cannot distinguish "Embodied LAM architectures are fundamentally limited" from "Embodied LAMs are trained on insufficient data."

Fourth, the paper does not evaluate whether the benchmark's difficulty estimation—the process of re-annotating videos using doubao-1.5-pro-vision—introduces systematic biases. The VLM-based annotation pipeline is treated as ground truth, but if the VLM has particular failure modes (e.g., systematically confusing certain action categories, or being biased toward certain embodiments), these would propagate into the benchmark labels and affect all model rankings. An inter-annotator agreement study between the VLM and human annotators, or an analysis of VLM confidence on difficult action categories, would help assess this risk.

Fifth, the regression evaluation uses a single temporal stride for the default comparison (stride=5) but only ablates stride for VLABench (Table 6). Extending the stride ablation to all four regression datasets would test whether the finding that latent action models encode "robust dynamic trajectories" generalizes across embodiments, or whether it is specific to the VLABench task configuration.

When do the claims hold, and when do they break?

The claim that general vision encoders outperform Embodied LAMs holds strongly on the LARYBench tasks as configured, but these tasks have specific properties that may favor general vision models. The classification task requires distinguishing 151 action categories that are visually diverse and semantically well-defined; a model trained on internet-scale data to recognize visual patterns may have an inherent advantage over a model trained primarily on robot manipulation videos. On a hypothetical benchmark where the action categories are more fine-grained and robot-specific (e.g., distinguishing between different grasp types or different contact force profiles), Embodied LAMs might perform better because their training data is more closely aligned to the evaluation distribution.

The claim that latent-based spaces are better aligned to physical action holds for temporally extended motion (stride=15, 30) but not necessarily for short-range motion (stride=5), where FLUX.2-dev achieves the best MSE (0.04, Table 6). This suggests that the optimal representation choice depends on the control frequency: for high-frequency control (short horizons, small displacements), pixel-level features may be superior; for low-frequency planning (long horizons, large displacements), feature-level representations dominate. The paper's advocacy for feature-level representations (Section 6) would benefit from acknowledging this regime-dependence.

The General LAM framework, while showing that VQ-VAE compression degrades backbone features, does not prove that all forms of specialized training are harmful—only that this specific quantization-based approach is. Alternative LAM paradigms not evaluated in this paper (e.g., continuous latent action models like CoMo, or non-VQ approaches like world model latents) might avoid the bottleneck degradation while still benefiting from domain-specific training objectives. The paper's conclusion that "future research should focus on aligning control policies with the robust feature spaces of general vision models" (Section 6) is a plausible direction suggested by the evidence, but the evidence does not rule out the possibility that novel LAM architectures—with different bottlenecks or different training objectives—could outperform general vision encoders by combining the benefits of both approaches.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted For in the Evaluation Protocol

The assumption or constraint. The entire LARYBench evaluation protocol—from data curation to probe training—assumes that clean, temporally segmented, semantically verified video clips with consistent action labels are available as ground truth. The paper's automated data engine achieves this by deploying the doubao-1.5-pro-vision API across 1.2M+ video clips in a four-stage pipeline (action segmentation, video-description matching, video-verb consistency check, and manual sampling inspection). Section 3.1 and Appendix B.2 describe this pipeline in detail, and the paper explicitly acknowledges the engine's role: "To handle diverse data quality across existing datasets, we develop an automated and scalable data engine to perform precise temporal segmentation and semantic alignment."

The consequence. The benchmark's quality—and therefore all reported model rankings—depends on the reliability of the VLM annotations, but the paper provides no measurement of that reliability. If the doubao-1.5-pro-vision API has systematic failure modes (e.g., confusing visually similar action categories like "place" and "put," or being biased toward egocentric human actions over exocentric robot actions), these errors become absorbed into the "ground truth" labels against which all models are evaluated. A model that achieves high classification accuracy might simply be agreeing with the VLM's systematic errors rather than capturing genuine action structure. The paper does mention that the action categories include semantically overlapping terms (Section 3: "pick" and "place" appear in both human and robotic domains), but does not report inter-annotator agreement between the VLM and human annotators on the final curated taxonomy.

What evidence exists in the paper. The data curation pipeline is described in detail (Appendix B.2, including the full prompt text), confirming reliance on a single external API as the annotation oracle. Stage 4 ("Manual Sampling Inspection") provides a human-filtered quality check, but the paper states only that it is used to "exclude verbs that lack explicit or well-defined kinematic meanings" such as "apply, arrange, clean"—it is not a systematic validation of annotation correctness. The paper provides no confusion matrix for the VLM's predictions, no inter-annotator agreement scores, and no analysis of how annotation errors might differentially affect model rankings. Figure 2 shows the action category distribution, which reveals strong long-tail effects (some actions have orders of magnitude more samples than others), but does not break down annotation confidence by category frequency.

Mitigation status. Not addressed. The paper treats the VLM annotations as ground truth without quantifying their noise level. The four-stage pipeline (segmentation → matching → consistency check → manual inspection) provides redundancy that likely reduces single-point VLM failures, but does not eliminate the fundamental dependency on a single annotator model. Future work could strengthen the benchmark by conducting a systematic human-VLM inter-annotator agreement study, particularly on the long-tail action categories where VLM performance may degrade.

---

Limitation 2: The Benchmark's Difficulty Estimation Produces No Uncertainty Quantification, Making Statistical Comparisons Impossible

The assumption or constraint. The paper reports all results as single-point estimates—top-1 accuracy percentages and MSE values—without confidence intervals, standard deviations, or statistical significance tests. The test sets for different tasks vary substantially in size: atomic actions have 25,940 samples split 75:25 (~6,500 test samples across 28 classes), while composite tasks use 70:30 splits on 1.2M samples (Section 4.2). Some rare action categories may have very few test samples—the long-tail distribution shown in Figure 2 spans six orders of magnitude in sample count—meaning per-class accuracy estimates for tail categories could be extremely noisy.

The consequence. Without uncertainty estimates, the reader cannot determine whether the reported performance differences between models reflect genuine representation quality differences or sampling noise. For example, the gap between LAPA-DINOv2 (49.36%) and LAPA-DINOv3 (49.17%) in Table 1 is 0.19 percentage points—well within likely sampling error for macro-averaged accuracy over a test set of ~360K composite samples with 123 human action classes, many of which are long-tail. Similarly, the regression MSE differences in Table 2 between General LAMs (LAPA-MAGVIT2 at 0.65, LAPA-SigLIP2 at 0.65, LAPA-DINOv2 at 0.63, LAPA-DINOv3 at 0.60) are small relative to the unknown variance. The paper's ranking of models within paradigms and its ablation recommendations (e.g., "cs=64 optimal," "dim=256 strikes the best capacity-stability balance" from Section 4.3) rely on these point estimates being meaningfully different.

This is particularly consequential for the paper's strongest normative claims. When the paper states that "general vision foundation models consistently outperform specialized Embodied LAMs" (Section 4.3), the gap between V-JEPA 2 (76.62%) and LAPA (20.17%) is large enough that statistical significance is not in doubt. But for finer-grained claims—such as whether V-JEPA 2 (76.62%) significantly outperforms DINOv3 (68.68%) in classification, or whether Wan2.2 (0.30) significantly differs from FLUX.2-dev (0.35) in regression—uncertainty quantification is essential to avoid over-interpreting rankings.

What evidence exists in the paper. The paper reports no variance estimates anywhere in the main text or appendix. The closest approximation to a robustness check is the hyperparameter sweep over probe learning rates and weight decay values (Section 4.2: "learning rates from 3e-4 to 5e-3 and weight decay values from 0.01 to 0.8"), which presumably selects the best-performing configuration per model, and the use of stratified sampling to preserve class distributions (Section 4.2). Neither addresses the fundamental question of how reliable the reported numbers are given finite test sets.

Mitigation status. Not addressed. The paper does not acknowledge the absence of uncertainty quantification as a limitation. A straightforward remedy—bootstrapping confidence intervals over the test set, or reporting per-class standard deviations—would substantially strengthen the reliability of the reported rankings without requiring additional experiments.

---

Limitation 3: The Embodied LAM Baseline Is Not Matched for Training Data Scale, Confounding Architecture with Data

The assumption or constraint. The paper's central empirical claim—that general vision foundation models outperform specialized Embodied LAMs—relies on a comparison where the two model classes are trained on fundamentally different data scales and distributions. The General Vision Encoders (V-JEPA 2, DINOv3) are trained on massive internet-scale datasets (WebVid, ImageNet, and other large corpora, though exact sizes are model-specific and not reported in the paper). The General LAMs are trained on "an expansive internal video dataset comprising over 2 million frames... curated to encompass a diverse mixture of human demonstrations, robotic manipulations, and general environmental dynamics" (Appendix C.2). In contrast, the Embodied LAMs (LAPA, UniVLA, villa-X) are evaluated using their publicly released checkpoints, which were trained on substantially smaller, domain-specific robotic manipulation datasets.

The paper acknowledges this asymmetry in passing (Section 4.3): Embodied LAMs "exhibit restricted generalization on diverse actions (averaging 17.99%–20.90%), due to the limited amount of training data or the early constraints to low-level actions." But it does not isolate data scale from architectural design in the evaluation.

The consequence. The comparison conflates two hypotheses: (1) the LAPA VQ-VAE architecture is fundamentally limited in its ability to produce quality latent actions, and (2) the LAPA checkpoints evaluated were trained on insufficient data to learn generalizable action representations. Hypothesis (2) could be true even if hypothesis (1) is false—a LAPA model trained on the same 2M+ frame internet dataset used for the General LAMs might close a substantial portion of the gap. The paper cannot distinguish these hypotheses with its current experimental design.

The General LAM results provide partial evidence favoring hypothesis (1): LAPA-DINOv3, which uses the LAPA VQ-VAE architecture on top of a frozen DINOv3 backbone and is trained on the 2M+ frame dataset, achieves 49.17% classification accuracy versus DINOv3's 68.68% (Table 1). This shows that even with large-scale training data, the VQ-VAE bottleneck degrades representation quality. However, this experiment does not test whether the end-to-end trained LAPA architecture (with a trainable pixel encoder, not a frozen backbone) would benefit similarly from internet-scale data. The pixel encoder in the original LAPA might learn different features than a frozen DINOv3 when trained on diverse internet data, potentially avoiding some of the degradation seen in the General LAM experiments.

What evidence exists in the paper. Table 1 and Table 2 report model performance but do not report training data scale for each model. Appendix C.2 reports the General LAM training data (2M+ frames, 100K steps) but provides no comparable information for the Embodied LAM baselines (their training data comes from the original papers, which used different datasets at different scales). The paper's statement that Embodied LAMs suffer from "limited amount of training data" (Section 4.3) implicitly acknowledges the confound but does not address it experimentally.

Mitigation status. Partially acknowledged but not addressed experimentally. The paper could strengthen this claim by training LAPA from scratch on the same 2M+ frame dataset used for the General LAMs, or by evaluating General LAMs trained on the same small-scale robotic datasets as the Embodied LAM baselines. Neither experiment is performed. The authors are transparent about the asymmetry in their qualitative analysis but do not control for it in their experimental design.

---

Limitation 4: The Benchmark Scope Is Limited to Closed-Form Action Categories and Trajectories, Excluding Open-Ended or Compositional Action Understanding

The assumption or constraint. LARYBench evaluates latent action representations exclusively through two tasks—classification into 151 pre-defined action categories and regression of end-effector trajectories—that assume all actions can be cleanly categorized or parameterized as continuous motion curves. The classification taxonomy was manually curated to exclude "overly abstract or kinematically ambiguous operations (e.g., apply, arrange, clean)" (Appendix B.2), and the regression tasks predict only "macroscopic arm displacements, as fine-grained finger articulation remains an ill-posed inverse problem for current visual encoders" (Section 3.2). This means the benchmark deliberately excludes the types of actions that are most common in real-world manipulation: compositional actions (multiple sub-actions performed in sequence or in parallel), contact-rich actions (where the physics of touch matters as much as the kinematics of motion), dexterous manipulation (finger articulation), and actions whose correct execution depends on object state rather than just trajectory shape.

The consequence. The benchmark's narrow scope means that strong performance on LARYBench may not translate to strong performance in real VLA systems. A representation that achieves 76.62% classification accuracy on isolated, well-segmented action clips might fail entirely when deployed in a continuous video stream where actions are not pre-segmented and action boundaries are ambiguous. A representation that achieves 0.19 regression MSE on predicting absolute end-effector trajectories for a single-arm Franka might fail to capture the contact dynamics, force modulation, or reactive behaviors needed for tasks like peg insertion, deformable object manipulation, or bimanual coordination. The paper's advocacy for using general vision encoders instead of specialized LAMs in VLA systems (Section 6) is based on benchmark results that test only the representation's internal structure, not its utility for downstream control in dynamic, open-ended environments—which is precisely the setting where VLAs must operate.

Furthermore, the exclusion of dexterous manipulation from RoboCOIN (the paper "deliberately mask[s] the dexterous hand joint data... to focus the evaluation on macroscopic arm displacements," Section 3.2) means the benchmark provides no signal about whether any representation—general vision or embodied—can capture the fine motor control that distinguishes skilled manipulation from coarse reaching. This is significant because dexterous manipulation is arguably the hardest and most data-scarce robotics problem, and the one where latent actions from human video would be most valuable if they worked.

What evidence exists in the paper. The action taxonomy is explicitly described as curated to exclude kinematically ambiguous actions (Appendix B.2). The RoboCOIN masking is described in Section 3.2. The paper's composite actions are temporally segmented into isolated clips by the VLM pipeline (Appendix B.2, Stage 1), meaning the benchmark never tests whether a representation can identify action boundaries in continuous video. The classification probe receives pre-segmented clips of 9 frames; the regression probe receives pre-paired start and end frames at a fixed stride.

Mitigation status. Not addressed. The paper does not discuss the gap between benchmark performance and real-world VLA utility. This is a structural limitation of any benchmark—it must simplify reality to be tractable—but the paper's advocacy for a "paradigm shift in VLA design" (Section 6) based on benchmark results would benefit from an explicit discussion of what LARYBench does and does not test about real-world action understanding.

---

Limitation 5: The General LAM Framework Cannot Recover Backbone Performance Even with Optimized Hyperparameters, Suggesting a Fundamental Degradation from the VQ Bottleneck That the Paper Does Not Resolve

The assumption or constraint. The General LAM experiments show that applying the LAPA VQ-VAE training paradigm to frozen vision backbones systematically degrades both classification accuracy and regression precision relative to the backbone alone. Even after optimizing the quantization hyperparameters—codebook size (Table 3), sequence length (Table 4), and latent dimension (Table 5)—the best General LAM configurations still substantially underperform their uncompressed backbone encoders. The paper's own "performance evolution path" (Figure 4) visualizes this gap and shows it narrowing but not closing.

The paper presents this as evidence that "an effective latent action space requires two key components: robust self-supervised visual priors to capture precise spatial-temporal dynamics, and a strictly regularized quantization bottleneck to maintain stability and dense utilization" (Section 4.3). But the bottleneck is not just a regularization mechanism—it is a hard information ceiling. The best LAPA-DINOv3 configuration after hyperparameter optimization (cs=64, sl=49, dim=256; Figure 4) achieves roughly 70–71% Composite Human accuracy and ~0.82 regression MSE, compared to V-JEPA 2's 80.35% and DINOv3's 0.22 (Tables 1–2). This ~10 percentage point classification gap and ~4× regression MSE degradation are not marginal—they represent substantial information loss that the hyperparameter ablations cannot eliminate.

The consequence. The paper's conclusion—that "future research should focus on aligning control policies with the robust feature spaces of general vision models" rather than learning action spaces from limited robotic data (Section 6)—is partially supported but underspecified. The evidence shows that frozen general vision features are superior to VQ-VAE latent actions, but it does not show how to use those features for downstream control. The VQ-VAE bottleneck exists for a reason: discrete action tokens enable autoregressive behavior cloning, which is the dominant paradigm for training VLA policies (as noted in Section 2: discrete methods "employ Vector Quantization to facilitate autoregressive behavior cloning"). If the recommendation is to abandon quantization and use continuous features instead, the paper does not address how these continuous features integrate into autoregressive policy architectures, how they handle the exploration-exploitation tradeoff in continuous action spaces, or how they scale with sequence length (continuous features of dimension 1024 produce substantially larger action spaces than discrete tokens from a codebook of size 64–1024).

The paper is essentially advocating for a different technical pathway without demonstrating that the pathway is viable end-to-end. The benchmark shows that continuous features contain more information—but more information does not automatically translate to better policies, because policy learning in high-dimensional continuous spaces is harder than in low-dimensional discrete spaces. The tradeoff between representation fidelity and policy learnability is not explored.

What evidence exists in the paper. Figure 4 explicitly shows the gap between optimized General LAMs and the V-JEPA 2 upper bound. Tables 3-5 show that hyperparameter tuning improves performance but does not close the gap. The paper acknowledges this implicitly by positioning V-JEPA 2 as the "continuous upper bound" (Figure 4 caption: "bridging the gap between the worst baseline (LAPA) and the continuous upper bound (V-JEPA2)"), but does not frame the persistent gap as a fundamental limitation of the VQ-VAE approach.

Mitigation status. Not addressed as a limitation. The paper presents the hyperparameter ablations as a recipe for building better latent action models (Section 4.3) but does not acknowledge the ceiling effect—that even optimally configured VQ-VAE latent actions systematically underperform continuous features on the benchmark's own metrics. The transition from "continuous features are better" (benchmark finding) to "use continuous features for control" (policy recommendation) assumes a step—making continuous features work in autoregressive policy architectures—that the paper does not validate.

---

Limitation 6: Single VLM Pipeline and Single Vision Model Family Create an Unquantified Annotation Bias Risk

The assumption or constraint. Every annotation decision in the LARYBench data curation pipeline—temporal segmentation boundaries, action verb labels, perspective classification, and semantic consistency verification—is made by the doubao-1.5-pro-vision API (Appendix B.2). The paper states that this is a deliberate design choice: "To ensure absolute annotation consistency across the entire dataset, we uniformly employ the doubao-1.5-pro-vision API for all processing steps" (Appendix B.2). The four-stage pipeline (segmentation → matching → consistency check → manual inspection) uses the same VLM at every stage, with the same prompts, applied uniformly across all source datasets.

The consequence. This creates a single-point-of-failure risk for annotation bias. If the doubao-1.5-pro-vision API has systematic biases—for example, being better at recognizing human actions than robot actions (due to its training data distribution), or being more reliable on common actions than rare ones, or having particular confusion patterns between certain action pairs—then these biases are embedded in the benchmark labels. Because all models are evaluated against the same labels, a model that shares the VLM's biases will appear to perform better than one that does not, even if the latter better captures the true underlying action structure.

This risk is particularly acute for the Composite Human vs. Composite Robot comparison. The VLM was presumably trained predominantly on human-centric video data (since that dominates internet video), which could make its annotations more reliable for human actions than robot actions. If so, the benchmark would systematically underestimate the quality of robot-centric representations (like Embodied LAMs trained on robotic data) relative to human-centric representations (like general vision encoders trained on internet data). The paper's cross-domain analysis (Figure 7, Appendix D.1) shows that DINOv3 exhibits "balanced cross-domain robustness" while General LAMs exhibit a "human-centric preference," but does not control for the possibility that the VLM annotations themselves have a human-centric bias.

What evidence exists in the paper. The paper reports no quantitative validation of the VLM's annotation quality: no inter-annotator agreement with human labelers, no analysis of per-category annotation confidence, no comparison between VLM-generated labels and the original dataset labels for overlapping action categories. The manual inspection in Stage 4 (Appendix B.2) filters out abstract actions but is not described as a systematic accuracy check. The paper does not discuss the risk of VLM annotation bias.

Mitigation status. Not addressed. The uniform use of a single VLM is presented as a strength (ensuring consistency) without acknowledging the corresponding risk (baking in systematic biases). A multi-annotator approach—using multiple VLMs, or combining VLM annotations with human annotations on a representative subset—would provide a partial safeguard against single-model bias. The paper's design choice to prioritize consistency over robustness to annotator error is reasonable but should be explicitly discussed as a limitation, particularly given the cross-domain claims that depend on annotation reliability across human and robot embodiments.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the evaluation of latent action representations from a policy-conflated afterthought to a decoupled, first-class scientific question. Before LARYBench, the field had no way to ask "is this latent action representation good?" without also asking "does the downstream policy succeed?"—two questions that were hopelessly entangled. A policy could compensate for a poor representation through memorization or powerful decoder capacity; conversely, a good representation could be dismissed because an undertuned policy failed to exploit it. LARYBench breaks this entanglement by establishing two fixed, lightweight probes—a 4-layer attentive classifier and a residual MLP regressor—that measure representation quality directly, independently of any downstream task architecture. This is a methodological reframing of the same type that ImageNet brought to visual representation learning: rather than evaluating features through the lens of a specific application pipeline, evaluate them through a standardized, minimal decoding head that makes the representation's internal structure legible. The conceptual move is to treat latent action spaces not as intermediate variables in a policy-learning recipe, but as objects of scientific study in their own right, with measurable properties (semantic discriminability, physical fidelity) that can be compared across models, architectures, and training paradigms.

The empirical findings reconfigure the dominant assumptions about where action-relevant knowledge comes from. The paper's most disruptive result—that V-JEPA 2, a model trained purely for self-supervised video prediction with no action labels and no robotics data, achieves 76.62% average classification accuracy versus Embodied LAMs' ~20% (Table 1), and that DINOv3 achieves 0.19 regression MSE versus Embodied LAMs' ~0.90 (Table 2)—is not merely a quantitative improvement. It is a qualitative challenge to the premise that specialized action extraction training is necessary or beneficial. The field's operating assumption has been that converting visual observations into useful action representations requires domain-specific architectures (inverse dynamics models, forward dynamics models, vector quantization) trained on domain-specific data (robot manipulation videos). LARYBench demonstrates that this specialized training does not add value on top of general visual pretraining—it destroys value. The General LAM experiments isolate the mechanism: taking a frozen DINOv3 backbone and applying the LAPA VQ-VAE training paradigm reduces classification accuracy from 68.68% to 49.17% (Table 1) and increases regression MSE from 0.19 to 0.60 (Table 2). The quantization bottleneck—the defining architectural feature of discrete latent action models—is not a neutral compression step; it is an active degrader of representation quality.

This finding reconciles a latent tension in the literature that the paper's introduction surfaces. Prior work offered contradictory signals: LAPA, Moto, and UniVLA demonstrated that latent actions could support policy learning on benchmark tasks, but diagnostic studies (Zhang et al., 2025a) showed that many LAMs "struggle with distractor robustness," and no one had explained why models that succeed on LIBERO fail to generalize. LARYBench provides the explanation: the representations themselves are impoverished. The policy succeeds on LIBERO not because the latent actions capture rich action structure, but because the policy head learns to compensate for representation deficiencies on a narrow task distribution. When probed directly—stripped of the policy's compensatory capacity—the representations are revealed to be barely above random chance for 151-way classification. The paper's cross-attention visualizations (Figure 6) make this degradation mechanistically interpretable: Embodied LAMs produce "broad, uninformative attention blobs" that fail to localize the interacting objects, while general vision encoders produce "sharp, geometry-aware focus on the active end-effector." The representation failure is not subtle—it is a catastrophic collapse of spatial grounding.

The paper redirects research investment from LAM architecture design to visual backbone selection. If the primary determinant of latent action quality is the visual backbone's pretraining—not the action extraction architecture built on top of it—then the field's resource allocation needs to shift. Developing a new VQ-VAE variant with a clever auxiliary loss (language grounding, trajectory priors, saliency constraints) is unlikely to help if the underlying visual features are the bottleneck. The paper's results suggest that the marginal return on improving the visual backbone (e.g., scaling up DINO-style contrastive pretraining, improving V-JEPA-style latent-space prediction) is far higher than the marginal return on improving the action tokenizer. The ablation studies (Tables 3-5) show that hyperparameter tuning of the VQ bottleneck can recover some of the gap—optimal codebook size, sequence length, and latent dimension bring LAPA-DINOv3 from ~55% to ~71% classification accuracy on Composite Human (Figure 4)—but the recovered performance never reaches the uncompressed backbone's level. This ceiling effect implies that the VQ bottleneck imposes a fundamental information limit, not a correctable implementation flaw.

The paper establishes the latent-vs-pixel distinction as a design axis with measurable consequences. By benchmarking semantic feature-level encoders (V-JEPA 2, DINOv3) against pixel-level generative encoders (Wan2.2, FLUX.2-dev) on identical tasks, the paper demonstrates that the choice of which space to operate in—latent features or raw pixels—has systematic, quantifiable effects. The stride ablation (Table 6) provides the most nuanced picture: pixel-level FLUX.2-dev achieves the best performance at stride=5 (0.04 MSE) but degrades catastrophically at stride=30 (0.62 MSE), while latent action models (both Embodied and General LAMs) maintain stability, and continuous feature encoders (DINOv3, V-JEPA 2) show moderate degradation (0.25 and 0.16 respectively at stride=30). This reframes the pixel-vs-latent debate: pixel-level models excel at short-range correspondence (where nearby frames share most pixels) but fail to encode temporally extended dynamic trajectories; latent-level models sacrifice short-range precision for long-range robustness. The paper's advocacy for latent-level representations (Section 6: "latent-based visual space is better aligned to robotic action space") is most strongly supported for the long-horizon regime that matters for planning and control, not for the high-frequency regime that matters for reactive servoing. This nuance was invisible in prior work because no benchmark tested representation quality across temporal horizons.

The diagnostic concepts introduced—long-tail dilemma, mid-frequency semantic aliasing, spatiotemporal grounding failure—provide a vocabulary for future analysis. Before LARYBench, the field had no language for decomposing why a latent action representation fails. The paper's error analysis (Section 5) introduces specific, interpretable failure modes: the widening performance gap between strong and weak models as action frequency decreases (Section 5.1, Figure 5), the confusion of semantically distinct actions that share visual appearance across different embodiments (Section 5.1, Figure 7), and the correlation between predictive failure and diffuse attention (Section 5.2, Figure 6). These are not merely descriptive—they are actionable. A researcher developing a new latent action model can now ask: does my model's cross-attention focus on the interacting objects? Does it maintain performance on the long-tail of rare actions? Does it recognize the same action across human and robot embodiments? These questions were unaskable in the prior evaluation paradigm because downstream task success provided no diagnostic signal—a policy failure could mean anything. LARYBench makes representation failure legible, which makes representation improvement systematic.

Follow-Up Research This Work Enables

Training LAPA from scratch on internet-scale data to isolate architecture from data scale. The paper's comparison between Embodied LAMs and general vision encoders conflates two variables: the VQ-VAE architecture and the training data scale. Embodied LAMs were trained on relatively small, domain-specific robotic datasets; general vision encoders were trained on massive internet corpora. A clean ablation would train the original LAPA architecture (with a trainable pixel encoder, not a frozen backbone) on the same 2M+ frame internet dataset used for the General LAMs, using the same 100,000-step budget on 8 NVIDIA H800 GPUs (Appendix C.2). If LAPA trained on internet data approaches General LAM performance (~49% classification accuracy), then the Embodied LAM deficit is primarily a data scale problem, not an architectural one—and the pathway to better LAMs is scaling training data, not abandoning the VQ-VAE paradigm. If LAPA trained on internet data remains at ~20% accuracy, the architectural critique stands. This experiment is directly enabled by LARYBench (the evaluation protocol exists and is standardized) and would resolve the paper's central confound.

Evaluating continuous latent action models (CoMo, IGOR) on LARYBench to test whether the VQ bottleneck, specifically, causes the degradation. The paper's General LAM experiments implicate the VQ-VAE compression as the primary source of representation degradation, but all Embodied LAMs evaluated use discrete codebooks. Continuous latent action models like CoMo (Yang et al., 2025), which avoid quantization entirely, or IGOR (Chen et al., 2024a), which operates on goal-image representations rather than frame-to-frame changes, may avoid the bottleneck while still benefiting from domain-specific training objectives. Evaluating CoMo and IGOR on LARYBench's classification and regression tracks—using their continuous latent representations directly, with the same probe architectures and training protocol—would test whether the degradation is specific to discrete VQ-based LAMs or inherent to any form of specialized action extraction. If CoMo significantly outperforms LAPA and approaches general vision encoder performance, the field's path forward is continuous latent actions with better visual backbones, not abandoning LAMs entirely. If CoMo performs similarly to LAPA, the degradation is more fundamental—perhaps any training that optimizes for reconstructing frame-to-frame changes, regardless of quantization, discards action-relevant semantic information that general pretraining preserves.

Multi-annotator validation of the VLM curation pipeline to quantify annotation bias risk. The entire LARYBench label set depends on a single VLM (doubao-1.5-pro-vision) applying a single annotation policy across all source datasets. A systematic validation study would sample 500-1000 video clips stratified by action category frequency and embodiment (human vs. robot), have 3-5 human annotators independently label each clip using the same 151-category taxonomy, and compute inter-annotator agreement (Cohen's kappa) between the VLM and human consensus. The analysis should specifically test whether VLM-human agreement differs between human-action clips and robot-action clips—if the VLM is more accurate on human actions (as its training data distribution would suggest), then LARYBench's cross-embodiment comparisons are systematically biased toward human-centric representations. This study requires no new models or training, only annotation effort, and would substantially strengthen (or appropriately qualify) all of the paper's cross-domain claims.

Extending the stride ablation to all regression datasets to test the generality of the temporal robustness claim. The paper's finding that latent action models encode "robust dynamic trajectories" while pixel-level models degrade with longer strides is based on a single dataset (VLABench, Table 6). Running the identical stride ablation (stride=5, 15, 30) on CALVIN, RoboCOIN, and AgiBotWorld-Beta—using the same Action Expert architecture and training protocol—would test whether this finding generalizes across robot morphologies and action dimensionalities. The key comparison is whether FLUX.2-dev's catastrophic degradation at stride=30 (0.04 → 0.62 MSE on VLABench) replicates on bimanual platforms (RoboCOIN's 12-DoF, AgiBotWorld's 16-DoF) and exocentric settings (CALVIN). If the pattern holds, the latent-vs-pixel distinction is a robust principle for temporal action encoding. If FLUX.2-dev maintains competitive performance at stride=30 on some datasets, the principle is domain-dependent, and the paper's advocacy for latent-level representations needs to be qualified by embodiment and viewpoint.

Probe architecture robustness study: linear probe vs. attentive probe vs. MLP probe. The paper uses a 4-layer attentive probe for classification and a 2-block residual MLP for regression as fixed evaluation heads. A systematic robustness check would vary the probe architecture: for classification, compare the attentive probe against a simple linear classifier (one fully-connected layer), a 2-layer MLP, and a deeper attentive probe (6-8 layers); for regression, compare the residual MLP against a linear regressor and a small Transformer decoder. If the model ranking (V-JEPA 2 > DINOv3 > LAPA-DINOv3 > LAPA) is preserved across all probe architectures, the benchmark's claim to decouple representation quality from probe capacity is strongly supported. If Embodied LAMs substantially close the gap with more powerful probes, then their representations contain latent structure that simple probes cannot access—which would qualify the paper's claim that the representations themselves are impoverished and suggest that Embodied LAMs may still be useful with appropriate decoder architectures. This experiment is straightforward—vary the probe, retrain on frozen latents, report the ranking—and directly tests LARYBench's core methodological premise.

Training a policy on general vision encoder features versus LAM latents to close the loop between representation quality and downstream task performance. LARYBench demonstrates that general vision encoders produce better representations by the benchmark's internal metrics, but does not demonstrate that this translates to better policy performance—the original evaluation paradigm the paper critiques. A closing-the-loop experiment would take the best-performing representation from each paradigm (e.g., V-JEPA 2 features vs. LAPA latents vs. LAPA-DINOv3 latents), train an identical behavior cloning policy on top of each for a standard manipulation benchmark (LIBERO, CALVIN, or a subset of tasks from each), and measure task success rate. If V-JEPA 2 features lead to higher success rates than LAPA latents, the decoupled evaluation is validated as predictive of downstream utility. If LAPA latents achieve similar or better success rates despite worse probe performance, then the probes are measuring something different from what policies need, and the benchmark's claim to evaluate "latent action quality" needs to be scoped more carefully. This experiment is the natural next step for validating LARYBench's practical relevance.

Practical Applications and Downstream Use Cases

Model selection for VLA system builders without running full policy training. A robotics team developing a new VLA system typically must choose a latent action representation (or visual encoder) before investing weeks of GPU time in policy training. Before LARYBench, the only way to make this choice was to train a complete policy with each candidate representation and compare success rates—a prohibitively expensive ablation. With LARYBench, the team can evaluate candidate representations on the benchmark's classification and regression tracks in a matter of hours (the probes train for 20 epochs on frozen latents), and use the resulting ranking to select the most promising representation for full policy training. The 56-percentage-point classification gap between V-JEPA 2 and LAPA (Table 1) and the ~5× regression MSE gap between DINOv3 and LAPA (Table 2) provide strong prior evidence that general vision encoders will outperform Embodied LAMs for most manipulation tasks, enabling teams to skip the LAM training step entirely and invest directly in policy learning on top of frozen vision backbones.

Difficulty-aware budget allocation for data curation pipelines. The paper's automated data curation engine (Section 3.1, Appendix B.2) uses a three-stage VLM verification pipeline (segmentation → matching → consistency check) to produce clean action annotations. The benchmark results reveal which action categories are hardest for current representations—specifically, the long-tail actions where even the best models show degraded performance (Figure 5). This information can be used to allocate curation resources efficiently: invest more human verification effort (beyond the Stage 4 manual inspection) on the action categories where model performance drops, since those are likely the categories where VLM annotation errors are most frequent or most consequential. For categories where all models achieve high accuracy (the head of the distribution in Figure 5), the VLM annotations are likely reliable and require minimal human oversight.

Pretraining data mixture optimization for embodied foundation models. The General LAM experiments (Tables 1-2) demonstrate that frozen vision backbones carry biases from their pretraining data—DINOv3 shows balanced human/robot performance while General LAMs show a human-centric preference (Figure 7, Appendix D.1). This suggests that the data mixture used to train visual backbones for robotics applications matters: a backbone trained predominantly on human-centric internet video will produce features that work well for human action understanding but poorly for robot control. LARYBench's cross-embodiment evaluation (Composite Human vs. Composite Robot classification, plus the four regression datasets spanning different robot morphologies) provides a quantitative signal for data mixture optimization: train several visual backbone variants with different ratios of human to robot video data, evaluate all on LARYBench, and select the mixture that minimizes the cross-embodiment performance gap. This is a concrete, benchmark-driven approach to what is currently an ad-hoc design choice in VLA development.

Diagnostic tool for identifying when a trained policy will fail to generalize. After training a VLA policy on a specific robot and task distribution, a practitioner can extract the policy's latent action representations (from the policy's visual encoder) and evaluate them on LARYBench. If the representations achieve high classification accuracy on Composite Robot but low accuracy on Composite Human (or vice versa), the policy is likely to fail when deployed on the other embodiment—even if the task semantics are identical. If the representations show poor stride robustness (large MSE increase from stride=5 to stride=30, similar to FLUX.2-dev in Table 6), the policy will likely fail on tasks requiring extended temporal reasoning. This diagnostic use of LARYBench does not require retraining the policy; it only requires running the frozen representations through the benchmark's probe training pipeline, which is computationally cheap. The output is a structured report on the representation's likely failure modes, enabling targeted data collection or fine-tuning before deployment rather than after failure.