RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

🎯 Pitch

State-of-the-art real-world robot policies fail 72% of the time on a new simulation benchmark that prevents training–evaluation overfitting, despite showing partial goal understanding in nearly half of tasks. The worst performance collapse comes not from visual confusion but from cluttered scenes and vague instructions, while even slight wrist-camera shifts zero out success rates—revealing brittleness that pure real-world testing misses.

1. Executive Summary

This paper introduces RoboLab, a simulation benchmarking framework designed to rigorously evaluate real-world-trained robot task-generalist policies in high-fidelity simulation, decoupling training and evaluation domains to prevent overfitting and benchmark saturation. The authors present RoboLab-120, a hand-curated benchmark of 120 pick-and-place tasks spanning three competency axes—visual (linking perceptual attributes to reasoning, e.g., recognizing color or size), procedural (action-oriented reasoning, e.g., stacking or reorientation), and relational (interpreting linguistic conjunctions and spatial relationships, e.g., "put the apple and orange on the plate")—across simple, moderate, and complex difficulty levels. The framework introduces a scalable AI-enabled scene and task generation pipeline, granular evaluation metrics including normalized subtask scores and trajectory quality measures (spectral arc-length for motion smoothness), and a Bayesian sensitivity analysis using Mixed Neural Posterior Estimation to identify which environmental parameters (e.g., wrist-camera displacement, object pose distance) most strongly associate with policy success. Evaluating state-of-the-art generalist policies trained exclusively on the real-world DROID dataset reveals that the best-performing model, π0.5, achieves only 28.0% success rate on RoboLab-120 while attaining a normalized score of 0.43—indicating meaningful partial progress even in failed episodes—and that success degrades sharply with increasing instruction vagueness, scene clutter, and task horizon, establishing that current policies exhibit brittle task-level reasoning and limited multi-step compositional planning even when they partially understand the goal.

2. Context and Motivation

The Core Problem: Evaluating Task-Generalist Policies Without Overfitting

The fundamental challenge this paper tackles stems from a tension at the heart of modern robot learning: we are building increasingly capable foundation models for robot control, but our evaluation frameworks have not kept pace. The last few years have seen an explosion of vision-language-action (VLA) models—large neural networks trained on diverse robot demonstration data that can follow natural language instructions to perform manipulation tasks. Models like π0, π0.5, GR00T N1, and others represent a shift toward task-generalist policies: policies trained on broad, heterogeneous datasets with the explicit goal of performing well on novel tasks without task-specific fine-tuning.

Yet the paper argues that the benchmarks used to evaluate these policies are fundamentally misaligned with the evaluation goals. To understand why, we need to examine what "evaluation" actually means for generalist policies and how existing benchmarks fail to deliver meaningful signals.

Three Critical Failures of Existing Benchmarks

The paper identifies three distinct, interrelated gaps in current simulation-based robotics evaluation (Section II, opening paragraphs):

Failure 1: Training-Evaluation Domain Overlap Trivializes Success

Popular benchmarks like LIBERO, RLBench, MetaWorld, and robosuite operate under a specific paradigm: policies are fine-tuned on simulation-specific demonstration data, then evaluated on those same simulated environments—often with the exact same visual textures, object models, and physics parameters. When a policy trains on LIBERO's specific table textures, object appearances, and scene layouts, and then gets tested on those identical environments, high success rates can reflect memorization of simulator-specific quirks rather than genuine task understanding.

This is not a hypothetical concern. The paper cites Zhou et al. (2025), who demonstrated that VLA policies evaluated on LIBERO achieve inflated success rates precisely because of this training-evaluation domain overlap. The problem is analogous to what happened in early ImageNet work: when the test set is too similar to the training set, metrics saturate rapidly and stop providing discrimination between genuinely better and marginally better models. In robotics, this saturation is particularly dangerous because it creates a false sense of progress—a policy that achieves 95% success on a benchmark may completely fail when faced with a visually novel but conceptually identical task, because what it learned was not the task logic but the surface-level appearance statistics of the training environment.

The paper's response is to decouple training and evaluation domains entirely. In RoboLab, all evaluated policies are trained exclusively on real-world data (the DROID dataset), and evaluated in a high-fidelity simulation that the policy has never seen. This forces the evaluation to measure genuine visual and procedural generalization rather than test-time familiarity with the simulator. Section IV states this principle explicitly:

"To prevent evaluation of policies overfitted to a particular simulation domain, we evaluate policies trained exclusively on the real-world DROID dataset."

Failure 2: Rapid Performance Saturation on Static Task Sets

When a benchmark ships with a fixed set of scenes and tasks, the robotics community can—and does—optimize against them. This is the familiar "test-set overfitting" problem from machine learning, but it is exacerbated in robotics because the cost of running physical experiments is so high that researchers often do not bother with held-out real-world tasks at all. Instead, they tune models on simulation benchmarks and treat high scores as evidence of generalization, even when the benchmark tasks are indistinguishable from the training tasks.

The paper argues that truly useful benchmarks must be extensible: they must make it easy to create genuinely new tasks that have not been seen during development. Without this extensibility, benchmarks have a limited useful lifespan—once the community saturates the existing task set, the benchmark ceases to provide discriminatory power between models. RoboLab addresses this through its AI-enabled scene and task generation pipeline (Section III-A), which can produce hundreds of novel configurations without requiring manual scene authoring for each one.

Failure 3: Lack of Granular Failure-Mode Analysis

The paper identifies a subtle but important shortcoming: most benchmarks report only binary success rates, sometimes accompanied by coarse difficulty breakdowns. A binary success rate tells you whether a policy succeeded, but it tells you almost nothing about why it failed when it did. Did the policy fail because it could not visually identify the correct object? Because it correctly identified the object but could not plan the manipulation sequence? Because it grasped the object but dropped it en route to the goal? Because it completed the task but took an inefficient path?

Without answering these questions, benchmarking results provide weak signals for model improvement. A policy developer who sees "40% success" on a relational reasoning task does not know whether to improve the vision encoder, the language grounding module, the action prediction head, or the overall planning architecture. The paper responds by introducing a suite of granular metrics designed to decompose performance along multiple axes simultaneously: normalized subtask scores that capture partial progress, trajectory-quality measures that quantify motion efficiency, event tracking that logs specific failure behaviors (grasping the wrong object, dropping objects, colliding with fixtures), and Bayesian sensitivity analysis that identifies which scene parameters most strongly correlate with success or failure.

The Real-World Evaluation Bottleneck

Why do these benchmark failures matter so much? The paper's answer is pragmatic: real-world evaluation is prohibitively expensive. Evaluating a generalist policy on 120 diverse manipulation tasks in the physical world would require setting up 120 distinct tabletop scenes with specific objects, positioning a robot precisely, running multiple trials per scene to account for stochasticity, resetting everything between trials, and dealing with inevitable hardware failures, calibration drift, and variability in environmental conditions (lighting, background clutter, etc.). This is simply not feasible at the scale needed for rigorous evaluation—especially when the most interesting scientific questions involve comparisons across multiple policies, multiple difficulty levels, and multiple environmental perturbation conditions.

Simulation-based benchmarking is the obvious alternative, but—as the paper documents—existing simulators create a dilemma. Low-fidelity simulators with simple graphics (figure 2, left panel) are fast and scalable but create a large sim2real transfer gap: policies that work in simple simulation may fail in the real world because the visual domain shift is too large. High-fidelity simulators produced via real-to-sim reconstruction techniques (figure 2, middle panel)—methods like Polaris that use 3D Gaussian Splatting to reconstruct photorealistic environments from real-world video—close the visual gap but introduce a new bottleneck: they require approximately one hour of computation per scene due to the need for camera calibration, mesh extraction, semantic segmentation, and physics property assignment. At this speed, creating a 120-task benchmark would take weeks of continuous computation, making iterative development and extension impractical.

RoboLab positions itself in figure 2 (right panel) as achieving high visual fidelity with low generation overhead. The paper's approach is to use a carefully constructed catalog of ~300 high-quality 3D object assets (drawn from established pose estimation benchmarks including YCB, HOT3D, HOPE, HANDAL, and VoMP), placed in a physically realistic simulation environment (Isaac Sim) with high-quality rendering, but without requiring per-scene real-world reconstruction. This means scenes can be authored in minutes—by arranging objects on a virtual tabletop and attaching language instructions—rather than hours. The tradeoff is that the simulation is representative of real-world visual complexity without being a pixel-perfect replica of any specific physical scene.

Contradictory Signals in the Literature

The paper is implicitly motivated by a tension that pervades recent robotics research. On one hand, models like π0.5 are reported to exhibit "open-world generalization" and impressive real-world capabilities in demo videos. On the other hand, systematic evaluations—including the very RoboLab results in this paper—reveal that the best model achieves only 28% success on a benchmark of what are ultimately simple tabletop pick-and-place tasks.

Where does this disconnect come from? The paper suggests it arises partly because most reported results are on tasks that closely resemble the training distribution. When a model is evaluated on tasks that look different (novel objects, novel visual backgrounds, novel scene compositions) even if the underlying manipulation skills are conceptually identical, performance drops sharply. This is the classic overfitting story, but with an important robotics-specific twist: in robot learning, "looking different" can mean something as simple as a differently colored bowl or a table with a wood texture instead of white laminate, because the visual encoder has never seen that texture during training and may fail to parse the scene geometry correctly.

The paper's framework is designed to surface these generalization failures systematically rather than allowing them to remain hidden behind cherry-picked demos or narrowly scoped evaluations.

How This Paper Positions Itself

The paper frames RoboLab not as a replacement for existing benchmarks but as a complementary tool that fills a specific gap: evaluating real-world-trained policies under controlled, reproducible, and systematically varied conditions. The abstract frames this as answering two questions:

"(1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior."

The first question is about benchmark validity: does RoboLab-120 performance correlate with real-world performance? The paper provides preliminary evidence for this in Section IV-D (figure 10), showing that the Spearman rank correlation between RoboLab-120 success rates and real-world RoboArena Elo scores is 1.00 across the four policies where both measurements exist. This is a small sample (four policies) on a specific benchmark, and the paper is appropriately cautious in presenting this as "benchmark-level correlation" rather than claiming perfect real-world prediction. The Spearman correlation captures whether the two benchmarks rank policies in the same order—a less demanding but more practically useful property than per-task performance agreement.

The second question is about diagnostic capability: can the benchmark identify what makes policies fail? This is where the competency axes, difficulty decomposition, sensitivity analysis, and sub-task scoring provide value that simple success rates cannot.

The paper explicitly contrasts its design philosophy with prior work in the "Real-to-sim Evaluation" paragraph of Section II:

"In contrast, our framework produces large-scale, photorealistic scenes and tasks within minutes rather than hours, while preserving sufficient geometric and visual fidelity for policy evaluation, thereby making real-to-sim benchmarking practical at the scale needed for modern generalist robot policies."

The key word here is sufficient. The paper does not claim that RoboLab's visual fidelity is perfect—Section V explicitly acknowledges a "residual visual distribution shift" as a limitation—but it argues that the fidelity is sufficient for distinguishing between policies and diagnosing failure modes, which is the purpose of a benchmark. If the visual gap were so large that all policies failed uniformly, the benchmark would not be useful. The results in Table I show strong differentiation between policies (π0.5 at 28.0% vs. PaliGemma at 3.4%), suggesting that the simulation is sufficiently realistic to preserve the performance ordering seen in the real world while being sufficiently synthetic to enable rapid authoring and controlled perturbation.

The Broader Significance

Beyond the specific benchmark, the paper is pushing on a broader question that is increasingly urgent for the robot learning community: what does it mean for a generalist policy to "generalize"? The current dominant narrative—that scaling up model size and training data diversity will naturally produce robust generalization—is challenged by results showing that a state-of-the-art 28%-success policy fails when instructions become vaguer or scenes become more cluttered. These are not failures on exotic adversarial examples; they are failures on tasks that a human would consider trivially similar to tasks the policy has presumably encountered during training on DROID's diverse dataset.

RoboLab provides a language for talking about these failures more precisely. Instead of saying "the policy doesn't generalize well," the framework enables statements like "the policy retains procedural understanding (the score remains at 0.44 on complex tasks even when success drops to 13.5%) but fails in execution, particularly on tasks requiring reorientation (10.0% success on affordance tasks) and is highly sensitive to wrist-camera displacement (the posterior concentrates sharply near zero in figure 9)." This granularity is what turns a benchmark from a scoreboard into a diagnostic tool, and it is the paper's most significant conceptual contribution beyond the specific task set.

3. Technical Approach

3.1 Reader Orientation

The RoboLab system is a simulation-based evaluation platform—think of it as a highly configurable virtual testing facility for robot policies, complete with the ability to automatically generate diverse test scenes and tasks. The core problem it solves is that evaluating whether a generalist robot policy truly "generalizes" requires systematic testing across many visually and procedurally diverse scenarios, which is prohibitive to do in the real world but trivial in simulation—provided the simulation is realistic enough to preserve the policy's failure modes and scalable enough to generate novel configurations without exhausting manual effort.

The "shape" of the solution is a pipeline that separates concerns cleanly: (1) a flexible scene and task authoring system that lets humans or LLMs specify what objects go where and what the robot should do, (2) a high-fidelity simulation environment built on Isaac Sim that renders these scenes with physically realistic lighting, collisions, and dynamics, (3) a set of evaluators—language-conditioned robot policies trained entirely on real-world data—that run in simulation and produce action commands from camera observations, and (4) a multi-faceted analysis toolkit that scores not just whether the task succeeded but how gracefully it failed, how efficiently the robot moved, and which environmental factors most strongly predicted success or failure.

3.2 Big-Picture Architecture (Diagram in Words)

The RoboLab framework has six major components, connected in a sequential pipeline from scene specification to diagnostic reporting:

Asset Catalog (B): A library of approximately 300 high-quality 3D objects—bowls, fruits, tools, containers—drawn from established pose estimation benchmarks. Each object has a visual mesh, collision mesh, mass and friction properties, a language description, and a label. This catalog is the vocabulary from which all scenes are built.
Scene Generator: Takes a theme (e.g., "messy kitchen counter") and produces a structured scene plan specifying which objects to place and what spatial relationships they should satisfy (e.g., "apple inside bowl," "mug on coaster"). This can be authored manually by drag-and-drop or automatically via an LLM prompted with the asset catalog and a formal predicate language for placement constraints.
Spatial + Physical Solver: Converts the symbolic placement predicates into concrete 3D positions and orientations. Runs forward physics simulation (300 timesteps under gravity) to verify stability—if any object falls or drifts more than 0.02m, a text error is fed back to the LLM for refinement.
Task Generator: Given a scene, produces language instructions and associated success criteria expressed as predicate functions. Each task decomposes into subtasks, each subtask into parallel events (grasp, hover, drop, done). Tasks can be manually specified or auto-generated by an LLM given competency-axis templates.
Simulation Environment (E): Instantiates the scene in Isaac Sim [22] with a specific robot embodiment, policy configuration, and environmental variations (camera pose, lighting, background, object pose). The policy receives camera images and proprioceptive state from the simulator and produces action commands that control the robot. The environment records the full trajectory of object poses, gripper states, and success/failure events.
Analysis Toolkit: Post-processes the recorded trajectories to compute normalized subtask scores, trajectory quality metrics (SPARC smoothness, path length, end-effector speed), event statistics (wrong object grasped, object dropped, gripper collisions), and Bayesian sensitivity posteriors via MNPE that identify which environment parameters most strongly correlate with successful outcomes.

The flow is: scene specification → physical instantiation → task definition → simulation with policy evaluation → multi-metric analysis. Crucially, the scene and task generators can be invoked iteratively: if the solver detects instability, feedback is sent to the LLM for refinement; if task code fails validation, a fix prompt is constructed and the LLM regenerates.

3.3 Roadmap for the Deep Dive

First, the formal problem definition: defining what a scene, task, policy, and environment are mathematically, establishing the vocabulary that governs everything downstream. This is essential because RoboLab's agnosticism—scenes don't care about robots, tasks don't care about policies—depends on clean modular interfaces between these abstractions.
Second, the asset catalog and simulation fidelity: understanding what objects are available, where they come from, and how physical properties are assigned. This matters because the benchmark's visual diversity and realism depend entirely on the quality and range of the asset library.
Third, the scene generation pipeline: the LLM-to-solver loop that scales scene authoring from manual to automated. This is the mechanism that enables RoboLab to avoid benchmark saturation—understanding the feedback loop explains why the system can produce hundreds of valid, diverse scenes without human intervention per scene.
Fourth, the task generation pipeline: how tasks are specified as language instructions plus predicate-based success conditions, how subtask decomposition works, and how difficulty scores are computed. This is central to the competency-axis evaluation structure.
Fifth, the evaluation environment and metrics: how policies are connected to the simulator, what constitutes a trial, and how the normalized score, trajectory quality, and event tracking metrics are computed. This is where the benchmark produces its diagnostic signals.
Sixth, the Bayesian sensitivity analysis (MNPE): the statistical machinery that identifies which environmental parameters most strongly predict success. This is the most technically involved analysis component and deserves careful mathematical exposition.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and benchmarking paper whose core idea is that rigorous evaluation of generalist robot policies requires systematic variation across task complexity axes in a decoupled simulation environment, and that the analysis must go beyond binary success rates to include partial credit, trajectory quality, and sensitivity to environmental parameters.

Formal Problem Definition: Scenes, Tasks, Policies, and Environments

The paper begins Section III-A by establishing precise mathematical definitions for its four core abstractions. These definitions are not merely formal—they encode the design principle of deferred binding, meaning that scenes are defined independently of robots and policies, allowing the same scene-task pair to be evaluated across multiple robot embodiments and policy variants without re-authoring.

Scene (S):

$S = \{(b_i, p_i, q_i)\}_{i=1}^{N}$

where $b_i$ is an object instance drawn from the asset catalog $\mathcal{B}$ , $p_i \in \mathbb{R}^3$ is the object's 3D position in the world frame, and $q_i \in \text{SO}(3)$ is the object's orientation represented as a unit quaternion in the rotation group. The number of objects $N$ varies per scene.

What it computes: A scene is simply a collection of object identities with their spatial configurations. It encodes "what is where" without any reference to what the robot should do or which robot is present. This separation means the same scene can be reused across different tasks—a kitchen counter scene with a bowl, an apple, and a banana could support tasks like "put the apple in the bowl" or "put the banana on the plate" or "move the bowl to the bin."

Why this form: The tuple representation $(b_i, p_i, q_i)$ captures the minimal information needed to instantiate an object in a physics simulator: identity (which determines geometry, mass, and visual appearance) and 6-DoF pose. No additional scene-level structure (e.g., scene graphs, PDDL definitions) is required at this level because the task definition handles success criteria, and the spatial solver handles physical constraints. This contrasts with benchmarks like LIBERO that use BDDL (Behavior Domain Definition Language) files encoding complex scene graphs—RoboLab's simpler representation makes authoring faster because users only need to specify object placement, not relational structure.

Task (T):

$T = \{S, l\}$

where $S$ is a scene as defined above, and $l$ is a natural language instruction specifying the goal to be achieved in that scene (e.g., "Put the apple and orange on the plate, then put the banana in the bowl").

What it computes: A task binds a specific scene configuration to a specific language goal. The instruction $l$ is a string—not a formal specification—because the evaluated policies consume natural language, and the benchmark's role is to test how well they interpret it.

Why this form: The task is deliberately minimal: scene + instruction. This contrasts with PDDL-based approaches that require goals to be expressed as formal logical formulas. The paper's design acknowledges that generalist policies take natural language as input, so the benchmark's task definition should match that interface. The success criteria for evaluation are separate from the task definition—they are encoded as predicate functions (discussed under task generation) that are invisible to the policy, used only for automatic scoring.

Policy (π):

$\pi: \mathcal{O} \rightarrow \mathcal{A}$

where the action space $\mathcal{A} \in \{\mathcal{A}_{\text{joint}}, \mathcal{A}_{\text{EE}}, \dots\}$ specifies the control modality (joint positions for the evaluated policies, but the framework supports end-effector commands and other action spaces), and the observation space $\mathcal{O} = (\mathcal{O}_{\text{proprio}}, \mathcal{O}_{\text{rgb}}, \mathcal{O}_{\text{depth}}, \dots)$ is policy-dependent, typically including proprioceptive state (joint angles, gripper position), wrist-mounted RGB images, and external camera RGB images.

What it computes: A policy is any function mapping observations to actions—the framework is policy-agnostic. It accepts any PyTorch policy with a standard act(obs) interface.

Why this form: The policy abstraction is a black box. RoboLab does not need to know the policy's architecture, training data, or internal representations. It only needs to provide observations in the expected format and accept actions in the expected format. This agnosticism is what enables evaluating models as diverse as π0.5, π0-FAST, GR00T N1, and PaliGemma within the same framework.

Environment (E):

$E = (T, \mathcal{R}, \mathcal{O}, \mathcal{A}, \xi)$

where $T$ is a task, $\mathcal{R}$ is the robot embodiment (e.g., DROID Franka Panda with Robotiq-2F-85 gripper), $\mathcal{O}$ and $\mathcal{A}$ define the policy-specific observation and action spaces, and $\xi = (\xi_{\text{camera}}, \xi_{\text{light}}, \xi_{\text{background}}, \xi_{\text{pose}})$ captures environmental variations—camera pose perturbations, lighting conditions (color temperature, exposure, directional shadows), background textures, and initial object pose randomization.

What it computes: An environment is the fully specified experimental configuration: which task, which robot, which policy, and what variations to apply. This is what gets instantiated in Isaac Sim for a single trial.

Why this form: The variation parameters $\xi$ are explicitly included in the environment definition because controlled perturbation is a central feature of the benchmark. By randomizing $\xi$ within specified ranges, the benchmark can measure sensitivity—does the policy still succeed when the wrist camera is displaced by 5cm? By 10cm? When the lighting changes from warm to cool? This decomposition of variation into named parameters enables the sensitivity analysis in Section IV-C.

Asset Catalog and Simulation Fidelity

The asset catalog $\mathcal{B}$ (Appendix A) contains approximately 300 object models sourced from established 3D pose estimation benchmarks: YCB [32] (the Yale-CMU-Berkeley object set containing common household items), HOT3D [3] (hand-object tracking objects), HOPE [17] (household objects for pose estimation), HANDAL [8] (manipulable objects with affordance annotations), and VoMP [6] (objects with volumetric mechanical property annotations). Each object asset includes four components:

Visual mesh: A triangulated surface model used for rendering RGB images from camera viewpoints. The visual quality is "high-fidelity" in Isaac Sim's path-traced renderer, meaning lighting, shadows, and material properties (specular highlights, diffuse reflectance) are physically simulated.
Collision mesh: A simplified or identical triangulated model used for physics collision detection and response. Separating visual and collision meshes (or using identical meshes) is standard practice in simulation—it allows the renderer to use high-poly models while the physics engine uses computationally cheaper proxies.
Mass and friction properties: Each object is assigned a mass (typically derived from volume × density) and friction coefficients (static and dynamic) that govern how it slides, stacks, and interacts with the gripper. The paper notes that for objects from VoMP, spatially varying density is available, enabling realistic mass distribution (e.g., a hammer has more mass in the head than the handle).
Language description and label: A text string identifying the object (e.g., "apple," "red bowl," "banana") used both for generating language instructions that reference specific objects and for the scoring predicates to check whether the correct object was manipulated.

The paper also describes an extension to Gaussian Splat + Mesh scenes (Appendix C, Figure 13). In this variant, the scene background is a 3D Gaussian Splat reconstruction (capturing photorealistic appearance from real-world data) with a collision mesh estimated via 3DGRUT [21, 31], while foreground objects remain standard mesh assets. This hybrid approach further closes the sim2real visual gap for the static environment while maintaining fast collision queries for the manipulated objects. However, the main RoboLab-120 benchmark uses the mesh-only pipeline for speed and simplicity.

Scaling Scene Generation: The LLM-to-Solver Feedback Loop

The manual scene authoring process—drag objects onto a virtual table, set their positions, orient them—is fast for individual scenes but does not scale to hundreds of diverse configurations. The paper's automated scene generation pipeline (Section III-A1, detailed in Appendix C) is a closed-loop system with three stages:

Stage I: Semantic Planning via LLM. Given a theme string (e.g., "messy counter," "tea time," "garage workstation") and the full asset catalog $\mathcal{B}$ with object names and bounding box dimensions $d_i \in \mathbb{R}^3$ , the LLM is prompted to produce a structured scene plan. The prompt (reproduced in Appendix C, Figures 14–16) instructs the LLM to use four placement predicate types:

place-on-base(object, x, y, yaw): Place the object directly on the table surface at the specified 2D position and orientation. This is for "anchor" objects—containers, supports, and loose items that sit directly on the table.
place-in(objects, container): Place one or more objects inside a container (bowl, bin). The container must already have a place-on-base assignment.
place-on(object, support, position): Stack an object on top of a support surface (plate, tray). The support must already be placed. The position parameter is one of "center," "edge," or "random."
cluster-around(objects, anchor, radius): Distribute objects in a loose cluster near an anchor object within the specified radius (typically 0.10–0.20m).

The LLM's output is JSON containing a list of selected objects and a list of placement predicates. The prompt includes explicit strategies for different object counts (sparse: <10 objects, medium: 10–14 objects, dense: 15+ objects), with heuristics like "Use MORE containment (place-in) to reduce table crowding" and "Use MORE stacking (place-on) to utilize vertical space" for dense scenes.

Stage II: Geometric Constraint Solving. The symbolic predicates from Stage I must be converted into concrete 3D poses $(p, q)$ —positions and orientations—for every object. This is non-trivial because the predicates specify relational constraints (e.g., "object A is inside container B," "objects C and D cluster around anchor E") but not exact coordinates. The paper's spatial solver (Algorithm 1 in Appendix C) processes predicates in dependency order:

Anchor placement first: All place-on-base objects (containers, supports) get positions sampled first, because other objects depend on them. For anchors, the solver uses rejection sampling on the table surface bounds $X \in [0.25, 0.85]$ m, $Y \in [-0.40, 0.40]$ m, checking collisions against already-placed objects using Separating Axis Theorem (SAT) on Oriented Bounding Boxes (OBBs).
Containment and stacking: For place-in(b_i, b_{\text{container}})$, the solver computes the container's interior volume, discretizes its floor into a grid with cell size $s = \max(d_i^x, d_i^y) + \epsilon_{\text{margin}}$ , and assigns objects to unoccupied cells within the container's bounds scaled by $\gamma = 0.7$ (to avoid edge collisions). The height is set to $z_i = z_{\text{container}} + h_{\text{container}}/2$ (center of the container's interior volume). For place-on(b_i, b_{\text{support}})$, the solver samples positions on the support's top surface via rejection sampling, checking for collisions with peers already on that support.
Clustering: For cluster-around({b_i}, b_{\text{anchor}}, r)$, objects are placed at random polar coordinates (angle uniformly sampled, distance uniformly sampled up to $r$ ) relative to the anchor's position.
Adaptive relaxation: If the solver cannot find collision-free placements for all objects (common in dense scenes), it progressively increases collision margins in four steps: $[\mu, 1.25\mu, 1.5\mu, 2.0\mu]$ , where $\mu$ is the nominal margin. If positions remain in local minima with unresolved collisions for 10 consecutive iterations, a stochastic perturbation step randomly jitters all object positions to escape the local minimum.

Stage III: Physical Validation and Feedback. After geometric placement, the scene is forward-simulated in Isaac Sim for 300 timesteps under gravity. An object $b_i$ is flagged as unstable if its maximum Euclidean displacement during simulation exceeds a threshold (typically 0.02m). If any object is unstable, a natural language error message is generated:

"Object 'apple' fell off 'plate' with displacement 0.15m."

This error message is concatenated to the original LLM prompt (Figure 17 in Appendix C) as a fix prompt $Q_{\text{fix}}$ , and the LLM generates a revised scene plan. The loop continues until physical validity is achieved or a maximum iteration count is reached.

Baseline comparison (Appendix C, Section C-D). The paper implements a single-pass baseline inspired by domain randomization: the LLM selects objects and suggests a grid layout (rows × columns), objects are assigned to cells with uniform jitter, and a physics simulation pass resolves interpenetration. This baseline lacks the iterative feedback loop and cannot handle stacking or containment—objects are placed at safe heights above the table. Evaluated across 100 scenes and 10 scene themes, the paper's method substantially outperforms the baseline on all metrics reported in Tables VI–VIII (e.g., visual realism 8.755 vs. 6.889, GPT preference 82% vs. 18%).

Why this design: The iterative loop with physical validation addresses a fundamental challenge in LLM-based scene generation: LLMs can suggest plausible spatial configurations in text, but they cannot simulate physics. An LLM might suggest placing a heavy object on the edge of a small plate, which is geometrically possible but physically unstable. The feedback loop converts physics simulation results into natural language that the LLM can understand and use to revise its plan. This is conceptually similar to the "self-reflective agent" paradigm in LLM-based code generation, where compiler errors are fed back to the LLM for revision, but applied to spatial-physical reasoning.

Scaling Task Generation: From Scenes to Evaluable Instructions

Given a scene $S$ , the task generator must produce tasks $T = \{S, l\}$ where $l$ is a natural language instruction and the task is automatically evaluable via success predicates. The paper's automated task generation pipeline (Section III-A2, evaluated in Appendix D) mirrors the scene generation loop but operates at the level of task code rather than spatial predicates.

Input to the LLM (Appendix D):

Scene object catalog $B_S$ with metadata and dimensions—this tells the LLM which objects are available to reference in instructions.
Task examples demonstrating the expected structure—few-shot prompts showing how tasks map to code.
Complete predicate library defining sub-task success and termination conditions. These are function templates like Grasp(obj), Release(obj), ObjectInContainer(obj, container), ObjectOnSupport(obj, support), etc.
Competency-axis language templates with placeholders for objects, spatial verbs, and attributes. For example, a "relational" template might be: "Put the [object A] and [object B] on the [container], then put the [object C] in the [bin]."
Constraints including difficulty levels and physical feasibility requirements (e.g., containment size constraints—a banana cannot be placed inside a shot glass; stacking stability—an object placed on a support must fit within its surface area).

Validation and feedback loop:

Syntax check: The generated task code is parsed and checked for Python syntax validity.
Asset validation: All referenced objects are checked against the forbidden set (objects present in the scene but unsuitable for manipulation, or objects already used in other tasks). For containment tasks, inner object dimensions are checked against container dimensions with a clearance margin.
Feedback construction: If validation fails, feedback is gathered into a fix prompt $Q_{\text{fix}}$ that includes the original prompt $Q$ , the invalid output, and an error message $E$ describing the specific failure (e.g., "Object 'banana' does not fit inside container 'shot glass'—clearance 0.01m is below minimum 0.03m"). The LLM revises and the loop repeats.

Task generation evaluation (Appendix D, Table IX). The paper evaluates 812 automatically generated tasks across 59 scenes and 7 competency categories using an LLM-as-judge framework. A separate LLM (OpenAI o1) scores each task on six dimensions:

Relation match: Does the spatial/logical relationship in the termination conditions match the instruction?
Target match: Are the goal states correctly specified?
Object match: Are the referenced objects correct?
Quantifier match: Are quantifiers ("all," "any," specific counts) handled correctly?
Instruction clarity: Is the language unambiguous and well-formed?
Physical feasibility: Is the task achievable given typical robot dexterity?

Each dimension is scored 0–1, and an aggregate alignment score is computed as the weighted mean. The overall alignment is 0.91, with 76% of tasks receiving a "fully aligned" verdict. Recognition and conjunction tasks achieve the highest alignment (0.96 and 0.97), while color-based tasks score lower (0.81), reflecting the challenge of grounding color references to specific object instances in generated code.

Difficulty scoring. Each task receives a difficulty score computed as:

$\text{DifficultyScore} = N_{\text{subtasks}} + \max(w_{\text{skill}})$

where $N_{\text{subtasks}}$ is the number of sequential subtasks in the task (e.g., "put A and B on the plate" → 2 subtasks, so $N_{\text{subtasks}} = 2$ ), and $w_{\text{skill}}$ is a competency weighting: $0$ for pure visual identification, $1$ for spatial reasoning, $2$ for procedural reasoning, and $3$ for reorientation and dynamic tasks. Based on this score, tasks are classified as:

Simple: DifficultyScore $\leq 2$
Moderate: DifficultyScore $= 3$ or $4$
Complex: DifficultyScore $\geq 5$

The RoboLab-120 benchmark contains 65 simple, 38 moderate, and 18 complex tasks, with overlapping competency axis labels (44 relational, 91 visual, 36 procedural) since tasks can exercise multiple competencies simultaneously (e.g., "Put the red apple in the blue bowl" tests both visual color recognition and relational containment).

Why this design: The difficulty score is deliberately simple—it combines task horizon (number of sequential steps) with a coarse competency multiplier. This differs from more elaborate difficulty metrics that might weight specific perceptual challenges, but the paper argues that the score is a starting point for stratification, with more granular analysis available through the scene composition and task horizon breakdowns shown in Figure 8c. The explicit design goal is interpretability: a score of 2 means "either a 2-step visual task or a 1-step spatial task," which is easy to reason about when analyzing results.

Task Decomposition into Subtasks and Events

Each task in RoboLab is decomposed into a sequential list of subtasks, where each subtask can contain parallel events (events that must all be satisfied before the subtask is considered complete). This decomposition serves two purposes: it enables the normalized score computation (partial credit), and it structures the event tracking for failure mode analysis.

The paper gives an example of the decomposition for the instruction:

"Put the apple and orange on the plate, then put the banana in the bowl."

This decomposes into:

Subtask 1: PickPlace(orange) & PickPlace(apple) (parallel—both must succeed, but order doesn't matter)
Subtask 2: PickPlace(banana) (sequential—must happen after Subtask 1)

Each PickPlace(obj) operation is further decomposed into a sequence of events:

$\text{PickPlace}(obj) = (\text{Grasp}(obj) \rightarrow \text{Hover}(\text{target}) \rightarrow \text{Drop} \rightarrow \text{Done})$

These events are automatically detected during simulation by predicate functions that check object-gripper contact, object-container overlap, and object-support contact. The events form a state machine: the Grasp event is triggered when the gripper closes around the specified object, the Hover event when the grasped object is positioned over the target location, the Drop event when the gripper opens and the object is released, and the Done event when object-in-target conditions are satisfied.

Why this decomposition matters: It enables the normalized score metric (described next) to award credit for partial progress—a policy that grasps the correct object but drops it before reaching the container still gets credit for the Grasp event even though the task is not completed. It also enables failure mode tracking: if the policy consistently fails after Hover (i.e., it reaches the target location but drops the object prematurely), that suggests a gripper control issue rather than a perception or planning failure.

Evaluation Metrics: Beyond Binary Success

The paper introduces a multi-layered evaluation system (Section III-C) designed to produce complementary signals about policy quality:

Normalized Score (Sc). For each task $T$ , the score is computed hierarchically:

$Sc(T) = \frac{1}{|T|} \sum_{\tau \in T} w_\tau \cdot Sc(\tau)$

$Sc(\tau) = \left\|\sum_{e} w_e\right\| \quad \text{(clipped to [0,1])}$

where $T$ is the set of subtasks, $\tau$ is an individual subtask, $w_\tau$ and $w_e$ are configurable weights for subtasks and events (defaulting to 1.0 so all milestones are equally weighted), $e$ denotes an event within a subtask (e.g., Grasp, Hover), and the norm $\|\cdot\|$ clips the sum to the $[0, 1]$ range so that fully completing a subtask yields a score of 1.

What it computes: The normalized score measures partial task completion. A fully successful episode earns a score of 1.0. An episode that grasps the correct object but never deposits it earns a fractional score (e.g., 0.25 if the subtask has four events of equal weight). Importantly, the paper reports both the overall score and the score on failed episodes only (Table IV, "Score (fail)" column)—this isolates how much progress the policy makes even when it ultimately falls short. For π0.5 on complex tasks, the overall score is 0.44 while the fail-only score is 0.35, meaning that even within episodes that do not complete, the policy reaches more than one-third of the milestones on average.

Why this form: This is the robotics analog of partial-credit scoring in educational testing: it rewards getting partway through a problem. Without partial credit, a policy that consistently grasps the correct object but drops it would receive the same score (0) as a policy that flails randomly. The normalized score discriminates between these cases. The hierarchical structure (tasks → subtasks → events) reflects the natural compositional structure of manipulation tasks—you cannot put an apple in a bowl until you have picked up the apple, so events within a subtask are ordered but subtasks themselves may be parallel.

Trajectory Metrics. The paper computes three trajectory-quality measures, all derived from the end-effector position trajectory $\{p_k\}_{k=0}^{N-1}$ recorded at discrete timesteps. These metrics are reported in Table V for all policy-category pairs.

Path length ( $l$ ):

$l = \sum_{k=0}^{N-1} \|p_{k+1} - p_k\|$

where $p_k$ is the end-effector position at timestep $k$ .

What it computes: The total Euclidean distance traveled by the end-effector during the episode. Shorter paths indicate more direct trajectories and generally reflect better motion planning—the robot takes efficient routes rather than wandering.

Why this form: Path length is a standard measure of trajectory efficiency. However, it does not account for time or velocity, so a slow but direct trajectory and a fast but circuitous trajectory could have similar path lengths. This motivates the complementary speed metric.

Mean speed ( $\bar{v}$ ):

$\bar{v} = \frac{1}{N} \sum_{k=0}^{N-1} \|v_k\|$

where $v_k$ is the end-effector velocity at timestep $k$ .

What it computes: Average end-effector speed in cm/s across the entire episode. Higher speeds indicate more confident, less hesitant motion—the robot commits to actions rather than oscillating.

Spectral arc-length (SPARC):

$\text{SPARC} = -\int_{0}^{\omega_c} \sqrt{\left(\frac{1}{\omega_c}\right)^2 + \left(\frac{d\hat{V}(\omega)}{d\omega}\right)^2} \; d\omega$

where $\hat{V}(\omega) = V(\omega) / V(0)$ is the normalized Fourier magnitude spectrum of the end-effector velocity profile $v(t)$ , and $\omega_c$ is an adaptive cutoff frequency.

What it computes: SPARC evaluates motion smoothness by measuring the arc length of the normalized Fourier magnitude spectrum [2]. If a velocity profile is smooth and contains mostly low-frequency components, its Fourier spectrum decays rapidly, producing a short arc length and a SPARC value close to zero (e.g., -2 or -3). If the velocity profile is jerky and contains significant high-frequency energy, the spectrum decays slowly, producing a longer arc length and a more negative SPARC value (e.g., -15 or -20). The negative sign makes SPARC < 0, with less negative = smoother.

Cutoff frequency ( $\omega_c$ ): To avoid including irrelevant high-frequency noise, the integration is truncated at an adaptive cutoff: $\omega_c = \min(10 \text{ Hz}, \omega_\alpha)$ where $\omega_\alpha = \max_{k \in K} \omega_k$ and $K = \{k \mid \hat{V}(\omega_k) \geq \alpha\}$ is the set of frequency bins where the normalized magnitude exceeds threshold $\alpha = 0.05$ .

Why this form: The SPARC metric is designed to be dimensionless (independent of movement amplitude and duration) and robust to measurement noise [2]. Unlike simpler smoothness measures like jerk integration (which scale with movement duration), SPARC captures the shape of the frequency spectrum—whether motion energy is concentrated at low frequencies (smooth) or spread across frequencies (jerky). A jittery movement has high-frequency components from rapid acceleration/deceleration spikes; SPARC penalizes this by integrating over the spectral derivative, which is large when the spectrum changes rapidly (indicating multiple frequency components). The adaptive cutoff ensures that the metric focuses on frequencies that actually contain signal rather than measurement noise.

Language Variations. Each task in RoboLab is paired with a set of language instructions spanning three specificity levels: vague ("tidy up"), default (the standard instruction), and specific ("place the red apple in the ceramic bowl on the left"). The policy should behave similarly across these variants if it has genuine task understanding rather than brittle keyword matching. Sensitivity to instruction specificity is one of the axes analyzed in Figure 8a.

Event Tracking (Task Adherence). The benchmark automatically records discrete behavioral events during policy rollout: grasping the wrong object, dropping an object, executing redundant actions, and causing gripper collisions. These events are detected via predicate functions that check object-gripper contact against the task's specified target objects. Even if the task is eventually completed successfully, these events signal potential biases or robustness issues—for example, the policy in Figure 3 (top row) successfully deposits an object in a bin but first grasps an extraneous orange, indicating imperfect visual grounding.

Bayesian Sensitivity Analysis via Mixed Neural Posterior Estimation (MNPE)

The sensitivity analysis framework (Section III-D, detailed in Appendix B) addresses the question: which environmental parameters most strongly influence whether a policy succeeds or fails? This is a causal attribution problem—we observe that the policy sometimes succeeds and sometimes fails across randomized trials, and we want to infer which variation parameters explain the success/failure pattern.

Parameter definition. Let $\theta = (\theta_{\text{cont}}, \theta_{\text{disc}})$ denote the vector of environment variation parameters. In the camera pose experiments, $\theta = (d_{\text{ext}}, d_{\text{wrist}}) \in \mathbb{R}^2$ , where each $d$ is a weighted distance metric combining position and orientation displacement from a reference camera configuration:

$d(T, T_{\text{ref}}) = \|p - p_{\text{ref}}\|_2 + \beta \cdot d_{\text{SO}(3)}(q, q_{\text{ref}})$

$d_{\text{SO}(3)}(q_1, q_2) = 2 \arccos\left(\min(1, |q_1 \cdot q_2|)\right)$

where $p \in \mathbb{R}^3$ is the camera position, $q \in \mathbb{H}$ is the unit quaternion orientation, $\|\cdot\|_2$ is Euclidean distance in meters, and $d_{\text{SO}(3)}$ is the geodesic (shortest-path) distance on the rotation group $\text{SO}(3)$ measured in radians. The weighting factor $\beta = 1.0$ balances translational (meters) and rotational (radians) contributions on approximately equal footing (1 radian of rotation ≈ 1 meter of arc-length displacement at a 1-meter distance from the camera).

Why this distance metric: Simply treating position and orientation as independent coordinates would miss their geometric coupling—rotating a camera by 30° while keeping its position fixed changes what it sees, and the magnitude of that change depends on the scene distance. The weighted distance collapses the 7-DoF pose (3 position + 4 quaternion, but quaternions have 1 DoF redundancy) into a single scalar representing "how far" the camera is from its nominal configuration. This scalar is what the MNPE posterior conditions on.

Outcome variable. $x \in \{0, 1\}$ is the binary task success indicator (1 if task completed, 0 otherwise). After running $N$ randomized episodes, we obtain a dataset $\mathcal{D} = \{(\theta_i, x_i)\}_{i=1}^N$ of parameter-outcome pairs.

The Bayesian inference target. We want the posterior distribution $p(\theta \mid x_o)$ , which answers: "given that a policy achieved outcome $x_o$ (e.g., $x_o = 1$ for success), what values of $\theta$ are most probable?" If the posterior for wrist camera displacement is sharply concentrated near zero, that tells us successful episodes required the wrist camera to stay close to its nominal pose—the policy is sensitive to wrist camera calibration. If the posterior is broad and flat, the policy is robust to that parameter.

Why a Bayesian approach: The frequentist alternative would be a correlation analysis (e.g., logistic regression of success on parameters). That would tell us whether a parameter is associated with success, but it would not give us the full conditional distribution, which is more informative for sensitivity analysis. The posterior tells us not just that wrist camera displacement matters, but how much displacement can be tolerated before success probability drops significantly. Furthermore, a Bayesian framework naturally handles the mixed continuous-discrete parameter spaces that arise in robotics evaluation (camera displacement is continuous, lighting type is discrete, table material is categorical), through the factorization described below.

Mixed Neural Posterior Estimation (MNPE). MNPE is a likelihood-free inference method: it does not require specifying a parametric likelihood $p(x \mid \theta)$ (which would be extremely difficult for a neural network policy), but instead trains a neural density estimator to directly approximate the posterior. The neural network $q_\phi(\theta \mid x)$ with parameters $\phi$ learns to map observations $x$ to distributions over parameters $\theta$ .

For mixed continuous-discrete parameter spaces, MNPE factorizes the posterior:

$q_\phi(\theta \mid x) = q_\phi(\theta_{\text{cont}} \mid \theta_{\text{disc}}, x) \cdot q_\phi(\theta_{\text{disc}} \mid x)$

where the discrete component $q_\phi(\theta_{\text{disc}} \mid x)$ uses a categorical distribution (softmax over possible discrete values), and the continuous component $q_\phi(\theta_{\text{cont}} \mid \theta_{\text{disc}}, x)$ uses a normalizing flow—a flexible density model that transforms a simple base distribution (e.g., a standard Gaussian) through a series of invertible transformations into a complex target distribution. Normalizing flows are chosen because they can represent multi-modal, skewed, and heavy-tailed distributions that might arise from policy behavior (e.g., success might be possible at both "very close" and "moderately far" camera distances if the policy has learned to adapt, producing a bimodal posterior).

Training objective. The network is trained by minimizing the negative log-likelihood of the observed parameters under the estimated posterior:

$\mathcal{L}(\phi) = -\frac{1}{N} \sum_{i=1}^N \log q_\phi(\theta_i \mid x_i)$

What it computes: For each training example $(\theta_i, x_i)$ , the network predicts a distribution over $\theta$ given outcome $x_i$ . The loss penalizes the network for assigning low probability to the actual $\theta_i$ that produced that outcome. Minimizing this loss over the dataset learns the mapping from outcomes to parameter distributions—when $x_i = 1$ , the network learns to assign high probability to parameter values that occurred in successful episodes; when $x_i = 0$ , it learns the distribution for failed episodes (though the paper primarily analyzes the success-conditional posterior).

Why this form: Maximum likelihood training of a conditional density estimator is standard for neural posterior estimation. The alternative—training a classifier to distinguish success from failure and then applying Bayes' rule—would require the prior $p(\theta)$ , which is hard to specify for high-dimensional or mixed parameter spaces. Direct density estimation avoids this by learning the posterior mapping end-to-end.

Importance sampling correction for non-uniform priors. The experimental data may not sample parameters uniformly (e.g., if some camera poses are harder to randomize or if the sampling procedure has biases). To recover the posterior under a uniform prior $p(\theta) \propto 1$ (which represents "no prior knowledge about which parameters should be associated with success"), importance sampling weights are applied:

$w_i = \frac{p(\theta_i)}{\tilde{p}(\theta_i)}$

where $\tilde{p}(\theta)$ is the empirical proposal distribution estimated via Gaussian kernel density estimation on the training data. The effective sample size:

$\text{ESS} = \frac{1}{\sum_i \bar{w}_i^2}$

quantifies how much information is retained after correction—an ESS close to $N$ means the proposal was nearly uniform; a low ESS means the correction is unreliable.

Posterior inference and visualization. Given a query outcome $x_o$ (typically $x_o = 1$ for success), the method draws $N_s = 5000$ samples from the learned posterior:

$\{\theta^{(i)}\}_{i=1}^{N_s} \sim q_\phi(\theta \mid x_o)$

For each continuous parameter, the posterior mean and 95% credible interval are computed:

$\hat{\mu}_j = \frac{1}{N_s} \sum_{i=1}^{N_s} \theta_j^{(i)}$

$\text{CI}_{95\%}^{(j)} = \left[Q_{0.025}\left(\{\theta_j^{(i)}\}\right), Q_{0.975}\left(\{\theta_j^{(i)}\}\right)\right]$

where $Q_\alpha$ is the $\alpha$ -quantile.

Interpreting Figure 9. The paper's Figure 9 shows the posterior distributions for camera displacement (wrist and external) and object pose distance, conditioned on task success. The wrist-camera posterior is sharply concentrated near zero displacement (the peak is tight and close to the origin), indicating that successful episodes almost exclusively occurred when the wrist camera was near its nominal pose—the policy is highly sensitive to wrist camera calibration. The external-camera posterior is broader, suggesting more tolerance. The object pose posterior peaks at approximately 0.5m from the robot's origin, indicating that objects placed at this distance have the highest probability of success—likely because this falls within a comfortable reachability zone where the robot's inverse kinematics have good solutions without singularities or extreme joint angles.

Experimental Configuration and Hardware

The simulation environment is built on Isaac Lab [29], a GPU-accelerated simulation framework, using Isaac Sim [22] as the underlying physics and rendering engine. The evaluated policies all use the DROID robot configuration [13]:

Robot: Franka Panda 7-DoF arm with Robotiq-2F-85 parallel-jaw gripper (1-DoF binary open/close command).
External camera: ZED 2i with focal length $f = 2.1$ mm, mounted externally (typically at a fixed world-frame position looking at the table).
Wrist camera: ZED mini, mounted on the robot's wrist, moving with the arm.

The action space is 7-DoF Franka joint positions (continuous) plus a 1-DoF binary gripper command (open or close). The observation space includes wrist-camera RGB images, external-camera RGB images, and proprioceptive state (joint angles, gripper position). These are designed to match the real-world DROID data distribution, so policies trained on DROID can operate in simulation without domain adaptation.

Default environment parameters: A default office-like background and natural lighting mimic typical DROID dataset setups. Camera poses are set to strongly match the real-world DROID camera configuration. For variation experiments, these parameters are perturbed:

Lighting variations: Color temperature shifts (warm/cool white balance), lighting exposure changes (up to 500× intensity variation), and directional shadows cast by a strong light source.
Visual variations: 10 different background textures and 4 table textures.
Camera pose variations: Both wrist and external cameras randomized in orientation and position (via the displacement metric).
Object pose variations: Initial object positions randomized via uniform distribution within 10cm, 20cm, or 30cm of their nominal placement.

Each task is evaluated over $N = 10$ episodes per policy to account for stochasticity in both the simulator (physics randomness) and the policy (sampling-based action generation). The paper notes (Appendix A-A) that with $N = 10$ , the 95% confidence interval on a per-task success rate is approximately $\pm 30\%$ at $p = 0.5$ , meaning per-task numbers are coarse indicators. Aggregate results across all 120 tasks benefit from a much larger effective sample size. The recommendation for per-task conclusions is $N \geq 100$ episodes per task.

Why N = 10: The paper acknowledges this is a limitation—10 trials per task is chosen as a practical compromise between statistical power and computational cost. Running 120 tasks × 10 episodes × 5 policies = 6,000 simulated episodes, each potentially lasting 30–60 seconds of simulation time with physics and rendering, represents a substantial compute investment. The aggregate statistics are sufficiently powered for ranking policies and analyzing broad trends, but the paper appropriately cautions against over-interpreting per-task differences.

Summary of Design Choices and Their Justifications

Policy- and robot-agnostic scene/task definitions: Separating scenes and tasks from robot embodiments enables systematic comparison across policies without re-authoring. The deferred binding is achieved through the formal abstraction $E = (T, \mathcal{R}, \mathcal{O}, \mathcal{A}, \xi)$ , where the robot and policy are specified only at environment instantiation time.
LLM-to-solver feedback loop for scene generation: Rather than requiring LLMs to produce physically valid configurations in one shot (which they reliably fail at), the iterative loop converts physics simulation results into natural language feedback. This mirrors successful paradigms in code generation and self-refinement.
Iterative task validation with code checks: Syntax and asset validation ensure generated tasks are executable and physically possible. The LLM-as-judge evaluation (Table IX) confirms that this produces high-quality task specifications with 0.91 mean alignment.
Normalized score as a complement to success rate: This is motivated by the observation (Table IV) that policies often make substantial progress on tasks they do not complete—π0.5 achieves 0.44 normalized score on complex tasks despite 13.5% success. Without partial credit, this important signal about near-success behavior is lost.
SPARC for motion smoothness: Unlike jerk integration, SPARC is dimensionless, amplitude-independent, and robust to measurement noise [2]. The adaptive cutoff frequency ensures the metric is not dominated by sensor noise.
MNPE for sensitivity analysis: Chosen over simpler correlation analyses because it provides the full conditional distribution $p(\theta \mid x)$ rather than point estimates. The mixed continuous-discrete factorization handles the heterogeneous parameter spaces that arise in robotics evaluation. Normalizing flows provide the flexibility to capture non-Gaussian, potentially multi-modal posteriors that might arise from non-monotonic policy behavior.
Decoupled training and evaluation domains: All evaluated policies are trained exclusively on real-world DROID data and evaluated in Isaac Sim with high-fidelity rendering. This forces the benchmark to measure genuine generalization rather than memorization of simulator-specific appearance or dynamics. The paper demonstrates (Figure 10) that the resulting rankings correlate with real-world performance (Spearman $\rho = 1.00$ across 4 policies), suggesting the sim2real gap is small enough to preserve policy ordering even if absolute performance levels differ.

4. Key Insights and Innovations

Innovation 1: Training-Evaluation Domain Decoupling as a Principle for Generalization Benchmarks

The most conceptually significant move in this paper is not any specific metric or method—it is the architectural decision to completely decouple the training domain from the evaluation domain as a foundational design principle for generalist policy benchmarking. This is a deceptively simple idea with profound implications for what a benchmark actually measures.

What the field did before. Nearly all prior simulation benchmarks for robot manipulation—LIBERO [19], RLBench [11], MetaWorld [34], robosuite [37], CALVIN [20], BEHAVIOR-1K [15]—operated under a shared assumption: the policy would be fine-tuned on data collected in the same simulator used for evaluation. Training and evaluation scenes shared the same visual textures, the same object models, the same physics parameters, and often the exact same scene layouts. Under this paradigm, a policy achieving 90% success could genuinely have learned the task—but it could also have memorized surface-level simulator statistics: the specific reflectivity of a particular table material, the way shadows fall from a specific directional light, or the exact collision geometry of a particular bowl mesh. The benchmark cannot distinguish between these.

The consequence, documented by Zhou et al. [36], is rapid performance saturation: success rates on LIBERO climb quickly as models overfit to simulator-specific quirks, and the benchmark loses discriminatory power. This is not merely a methodological nuisance—it is a fundamental threat to scientific progress in robot learning, because it creates a false signal that generalization has been achieved when in fact only simulator-specific memorization has occurred.

What RoboLab does differently. By evaluating policies trained exclusively on real-world DROID data [13] in a simulation environment the policy has never encountered, RoboLab forces the benchmark to measure genuine visual and procedural generalization. The policy cannot rely on having seen these specific objects, textures, lighting conditions, or background environments during training. If it succeeds, it must genuinely understand the task structure—the affordances of containers, the semantics of spatial prepositions, the action sequence for pick-and-place—rather than matching surface-level patterns from training.

This decoupling transforms the benchmark from a test of "how well does this policy fit the training distribution" into a test of "how well does this policy transfer to novel visual and procedural conditions." The paper's results validate that this distinction matters enormously: π0.5, a model described as exhibiting "open-world generalization" [9], achieves only 28.0% success on RoboLab-120 (Table I). This is not a failure of the policy in absolute terms—it is a signal that the policy's training-distribution performance substantially overstates its generalization capability, and that RoboLab-120 is measuring something more stringent than prior benchmarks.

Why this is fundamental, not incremental. This design principle addresses a structural problem in how the field evaluates generalist policies. It is not a refinement of an existing evaluation methodology—it is a shift in the evaluation paradigm from "test on held-out tasks from the training distribution" to "test on tasks from a different domain entirely." The paper explicitly argues that this shift is necessary because the former paradigm systematically overestimates generalization by conflating task understanding with domain familiarity. The evidence in Figure 10—showing that RoboLab-120 rankings correlate with real-world RoboArena rankings (Spearman ρ = 1.00 across 4 policies)—provides initial validation that decoupled simulation evaluation preserves real-world policy ordering even when absolute performance differs, making the approach both more rigorous and practically informative.

A subtler innovation within this decoupling principle is the sufficiency argument: the paper does not claim that RoboLab's simulation is a perfect replica of reality. Section V explicitly acknowledges a "residual visual distribution shift." But it argues—and demonstrates—that the fidelity is sufficient for distinguishing between policies and diagnosing failure modes. This reframes the sim2real gap from a binary "gap exists / gap doesn't exist" problem into a more nuanced question of how much fidelity is needed for a specific evaluation purpose. This is a pragmatic framing that enables scaling evaluation in ways that photorealistic but slow real-to-sim reconstruction methods (e.g., Polaris, ~1 hour per scene) cannot match.

Innovation 2: Partial-Credit Scoring and Multi-Faceted Metrics as a Diagnostic Lens

Existing robot manipulation benchmarks almost universally report binary success rates: either the task was completed (1) or it was not (0). This paper introduces a multi-layered evaluation toolkit that decomposes "success" into several independent, complementary signals—and in doing so, surfaces patterns about policy behavior that binary metrics systematically obscure.

What the field did before. Standard evaluation in robot learning reports success rate, sometimes broken down by task category or difficulty level. This is a scalar summary that collapses all variation in how a policy fails into a single dimension. A policy that never attempts the task (0% success, 0% partial progress) is indistinguishable from a policy that grasps the correct object, moves it to the target, and drops it at the last moment (0% success, near-complete partial progress). Both receive a score of zero.

Some recent work has recognized this limitation. Kress-Gazit et al. [14] argued for richer evaluation practices, and benchmarks like RobotArena∞ [12] incorporate human judgment for nuanced assessment. But human judgment is expensive, slow, and subjective—it cannot scale to systematic evaluation across hundreds of task variations, multiple policies, and controlled environmental perturbations.

What RoboLab does differently. The paper introduces three complementary metric families that operate independently of each other:

Normalized score (Table IV): Measures partial task completion via the hierarchical subtask/event decomposition. A policy earns credit for each milestone reached—grasping the correct object, moving it to the target location, depositing it—even if the full task is not completed. The paper reports this score both overall and for failed episodes only ("Score (fail)"), isolating how much progress the policy makes even when it ultimately falls short.
Trajectory-quality metrics (Table V): Measures how the policy moves, independent of task success. SPARC quantifies motion smoothness via spectral analysis of the velocity profile; path length measures trajectory directness; mean speed captures confidence and hesitation. These metrics can reveal systematic differences between policies—e.g., π0.5 achieves SPARC of −8.34 while PaliGemma achieves −16.52 (Table V), indicating substantially jerkier motion for PaliGemma even though both policies fail most tasks. This is a signal about control quality that success rates alone cannot provide.
Event tracking (Section III-C, Figure 3): Records discrete behavioral events—grasping the wrong object, dropping objects, executing redundant actions, causing gripper collisions—that may not affect final task success but reveal biases and brittleness. Figure 3 shows a policy that ultimately succeeds at depositing an object in a bin but first grasps an extraneous orange, indicating flawed visual grounding that binary success would never reveal.

The insight that emerges. The gap between success rate and normalized score reveals a phenomenon the paper terms near-success behavior: policies often partially understand the task but fail in the final stages of execution. On complex tasks, π0.5 achieves only 13.5% success but a normalized score of 0.44 (Table IV), meaning it reaches nearly half the milestones even within episodes that do not complete. On procedural affordance tasks, success is 10.0% but the score on failed episodes is 0.19—the policy consistently makes initial progress (grasps the object) but cannot complete the manipulation sequence.

This is a diagnostic signal, not just an evaluation signal. It tells a policy developer where to focus improvement efforts. If the score is high but success is low, the failure mode is late-stage execution, suggesting improvements to fine manipulation control or gripper coordination. If both score and success are low, the failure mode is early-stage perception or planning, suggesting improvements to visual grounding or task parsing. Without partial-credit scoring, both failure patterns look identical (0% success), and the developer learns nothing about what to fix.

Why this is fundamental, not incremental. Partial-credit scoring is well-established in educational testing and natural language processing (e.g., BLEU scores for translation), but it has been almost entirely absent from robot manipulation benchmarking, where the dominant assumption has been that tasks either succeed or fail. The paper's contribution is not inventing partial credit—it is recognizing that robot manipulation tasks have natural compositional structure (subtasks, events) that can be automatically scored via predicate functions, and that exploiting this structure yields diagnostic signals that are qualitatively different from—and complementary to—binary success. The fact that the scoring infrastructure (predicate library, event detection) is built into the benchmark framework rather than requiring per-task human annotation makes this approach scalable, which is what distinguishes it from prior calls for richer evaluation [14] that relied on human judgment.

Innovation 3: Bayesian Sensitivity Analysis as a Policy Understanding Tool

The paper introduces Mixed Neural Posterior Estimation (MNPE) for sensitivity analysis, but the innovation is not the statistical method itself—MNPE is a standard technique in simulation-based inference [Section III-D, Appendix B]. The innovation is what the method is used to answer: not "how often does the policy fail under perturbation X" (a robustness test), but "given that the policy succeeded, what environmental conditions must have been true?" This flips the analytical framing from forward (perturb → observe) to inverse (observe outcome → infer conditions), and in doing so reveals why policies are brittle in ways that forward analysis cannot.

What the field did before. Standard robustness evaluation in robot learning—and in machine learning more broadly—is forward: apply a perturbation (e.g., change lighting, add noise, randomize object poses), measure the drop in success rate, and report how "robust" the policy is to each perturbation. LIBERO [19] and REALM [28] use this approach for a limited set of task variations. The output is a number: "success drops by X% when lighting changes." This tells you that the policy is sensitive to lighting, but not how much lighting change causes failure, which specific aspects of lighting matter, or—crucially—whether different policies fail for the same or different reasons.

What the MNPE analysis does differently. By learning the posterior distribution $p(\theta \mid x = 1)$ —the distribution of environmental parameters conditioned on successful task completion—MNPE answers a fundamentally different question: "If this policy succeeded, what parameter values are consistent with that outcome?"

Consider the wrist-camera posterior in Figure 9. It is sharply concentrated near zero displacement for all policies. This does not merely tell us "policies are sensitive to wrist camera pose"—a forward robustness test would reveal that. It tells us something more specific: successful episodes almost exclusively occurred when the wrist camera was very close to its nominal calibration. If the policy could sometimes succeed with a displaced wrist camera by adapting its visual processing, the posterior would be broader. The sharp concentration indicates that the policy cannot compensate for wrist-camera displacement—it requires the camera to be in a narrow calibration range. This is a property of the policy's visual representation, not just its overall robustness, and it points to a specific improvement target: training with greater wrist-camera pose variation.

The object-pose posterior (peaking at ~0.5m from the robot base) provides a different kind of insight: it reveals the policy's effective workspace, not through direct measurement but through the statistical pattern of where successful grasps occur. This is information about the policy's reachability preferences, conditioned on the robot's kinematics, that emerges from the data rather than from an analytical model.

Why this is a conceptual advance. The forward-vs-inverse distinction is well-known in statistics but has been underutilized in robot policy evaluation, where the default mode is "measure performance degradation under perturbation." The paper demonstrates that the same experimental data—randomized parameter values and binary success indicators—can yield substantially richer insights when analyzed through an inverse lens. This is not an incremental improvement to robustness testing; it is a complementary analytical framework that answers questions forward testing cannot.

The practical value is concrete: if a policy developer knows that wrist-camera displacement is a hard constraint (narrow posterior), they can prioritize improving wrist-camera robustness in the next training iteration. If they only knew that success drops under camera perturbation (forward test), they would not know whether to fix wrist-camera processing, external-camera processing, or both. The MNPE posterior disaggregates these factors.

Limitations acknowledged. The method requires sufficient randomized data to learn a reliable posterior, and the paper's experiments are on a small set of basic tasks (BananaInBowl, BananaAndCubeInBowl) rather than the full RoboLab-120. Scaling this analysis to all 120 tasks, across all five policies, would require substantially more data collection. The paper treats MNPE as a proof-of-concept for the analytical approach rather than a fully deployed evaluation layer.

Innovation 4: Competency Axes as a Framework for Structured Generalization Diagnosis

The paper's decomposition of tasks into visual, procedural, and relational competency axes (Section III-B) might appear to be a simple organizational choice—labeling tasks by what skill they test. But it functions as something more ambitious: a structured diagnostic framework for understanding where generalization breaks down, not just whether it breaks down.

What the field did before. Most robot manipulation benchmarks categorize tasks by domain (kitchen, living room, warehouse) or by primitive skill (picking, placing, pushing, stacking). LIBERO [19] uses task families based on BDDL scene specifications. These categorizations are useful for organizing results but do not systematically isolate specific generalization dimensions—a "kitchen" task might require visual recognition (identifying the right utensil), procedural reasoning (opening a drawer before reaching inside), and relational understanding (putting the spoon in the drawer, not on the counter), but the category label does not tell you which of these caused a failure.

What the competency axes framework does differently. Each task is labeled with one or more competency attributes (color recognition, semantic identification, size discrimination for visual; affordance reasoning, reorientation, stacking for procedural; conjunction interpretation, counting, spatial reasoning for relational). These labels are non-exclusive: a task like "put the red apple and orange on the plate, then put the banana in the bowl" simultaneously tests visual color recognition ("red apple"), relational conjunction ("and"), and procedural affordance (containment). By aggregating results across tasks that share a competency label—even though those tasks may have different objects, scenes, and surface forms—the benchmark produces a competency profile for each policy.

Table IV demonstrates the diagnostic value of this profiling. Across all policies, counting tasks (relational competency) achieve the highest success rates (65.7% for π0.5), while affordance tasks (procedural competency) achieve the lowest (10.0% for π0.5). This is not a trivial finding—it would be masked in aggregate success rates, and it tells a clear story: current generalist policies understand object counts and conjunctions better than they understand how to physically manipulate objects (reorientation, stacking). Moreover, the gap between conjunction (43.8% success) and spatial reasoning (21.0% success) within the relational category reveals that "relational reasoning" is not a monolithic capability—policies are much better at interpreting "put A and B on C" than "put A to the left of B," even though both require relational language understanding.

Why this is a conceptual advance, not just a labeling scheme. The insight is that generalization failures are structured along competency dimensions rather than being uniform across tasks. A policy might have strong visual grounding but weak procedural planning, or strong relational language parsing but weak spatial reasoning. A single success rate collapses all these dimensions into one number. The competency-axis decomposition makes these differential strengths and weaknesses legible.

This is akin to the role that error taxonomy played in early computer vision: before the field could systematically improve object recognition, it needed to know whether failures were due to pose variation, illumination changes, occlusion, or category confusion. The competency axes serve the same diagnostic function for robot manipulation—they decompose "generalization failure" into specific, actionable failure categories.

Breadth of coverage. The labeling system is intentionally coarse (three axes, a few sub-attributes each) rather than exhaustively fine-grained. A task labeled "visual: color" may require other capabilities (e.g., basic grasping), but the dominant challenge is color-based object identification. This coarseness is a feature, not a bug: it makes labeling feasible (tasks are assigned attributes during authoring, not through post-hoc analysis) and produces competency profiles with enough tasks per category to be statistically meaningful (Table IV shows counts ranging from 6 for size/stacking/reorientation to 60 for semantics).

Evidence of diagnostic utility. The language variation experiments (Figure 8a, Figure 12a) use the competency framework implicitly: by showing that success degrades when instructions become vaguer ("tidy up" vs. "place the red apple in the ceramic bowl"), the benchmark identifies that task-level reasoning—not just visual recognition—is fragile. The scene complexity experiments (Figure 8b) show that visual grounding degrades with clutter. The task horizon experiments (Figure 8c) show that compositional planning degrades with length. Each of these maps cleanly onto the competency framework (relational reasoning for language, visual competency for clutter, procedural competency for horizon), transforming what could be a set of disconnected perturbation experiments into a coherent diagnosis of which competencies are most brittle under which conditions.

Innovation 5: Demonstrating That Test-Time Strategy Generalization Is Distinct from Training-Distribution Performance—and That the Gap Is Large

The paper's experimental results collectively surface an insight that is empirical rather than methodological: state-of-the-art generalist robot policies exhibit a large and structured gap between their training-distribution performance and their generalization performance on a decoupled benchmark, and this gap varies dramatically across competency dimensions.

This is not merely "policies don't generalize well"—a statement that has been made before. It is a precise characterization of how the generalization gap manifests: where it is largest, what forms it takes, and what it implies for the field's understanding of what "generalist" means.

What prior work suggested. Recent generalist policies—π0 [5], π0.5 [9], GR00T N1 [23]—are reported with impressive capabilities: multi-task performance across diverse manipulation skills, open-world object generalization, zero-shot task execution. The π0.5 paper [9] emphasizes "open-world generalization" as a key capability. The dominant narrative has been that scaling up model size and training data diversity leads to increasingly robust generalization.

What RoboLab-120 reveals. The highest-performing policy, π0.5, achieves 28.0% success on RoboLab-120 (Table I). This is not a failure of the benchmark—it is a revelation about the policy. The tasks in RoboLab-120 are conceptually simple pick-and-place operations. They involve common household objects (bowls, fruits, plates) and straightforward language instructions. A human given these instructions would achieve near-perfect performance. The 28.0% figure, viewed alongside the π0.5 paper's emphasis on generalization, suggests that "generalization" in prior work has been measured relative to a training distribution that substantially overlaps with the evaluation distribution—and that moving to a genuinely decoupled evaluation domain surfaces how much of that generalization was actually domain-specific adaptation.

The structure of the gap is equally informative. On conjunction tasks, π0.5 achieves 43.8% success; on affordance tasks, it achieves 10.0% (Table IV). These tasks are all pick-and-place—the same low-level skills are required—but the high-level reasoning demands differ. This means the generalization gap is not primarily about motor control (if it were, all categories would be similarly low). It is about the translation of language understanding and task reasoning into appropriate manipulation sequences. The policy can physically execute pick-and-place when it correctly identifies what to do; the bottleneck is the identification itself.

Score-vs-success discrepancy as a capability signal. The gap between normalized score and success rate (e.g., 0.44 score vs. 13.5% success on complex tasks for π0.5) reveals that policies often understand what to do but cannot execute it reliably. This is a fundamentally different failure mode than "does not understand the task at all," and it has different implications for model improvement. If the failure were in task understanding (score ≈ success rate ≈ 0), the fix would be better language grounding and planning. But score >> success suggests that planning and grounding are partially functional, and the failure is in precise execution—gripper coordination, approach angles, release timing. This nuanced diagnosis would be invisible without the partial-credit framework.

Difficulty-dependent trajectory metrics. Table V shows that trajectory smoothness (SPARC) degrades as task complexity increases—π0.5 goes from SPARC −7.38 on simple tasks to −10.83 on complex tasks, indicating jerkier motion. This is not inevitable: a policy could be equally smooth on all tasks if it executed them with consistent confidence, failing cleanly rather than hesitating. The degradation suggests that policies become more tentative and oscillatory when task demands exceed their comfortable operating regime—a behavioral signature of uncertainty that provides an independent signal of task difficulty beyond the success/success metric.

Negative result with implications: The annotation in Appendix A-D about the anomalous "recovery" at subtask=7 in Figure 8c is itself an important finding. π0.5 appears to recover to ~20% success at this horizon, but this is driven by a single task (CubesAndBlocksInBinTask) with repetitive, non-causal pick-and-place sequences of geometrically simple cubes—not a genuine long-horizon reasoning task. This illustrates a pitfall in interpreting horizon-vs-success curves: if the longest-horizon tasks are compositionally simpler than medium-horizon tasks, the curve can be misleading. The paper's transparency about this confound is a methodological contribution in itself.

Why this is significant beyond the numbers. The paper is making a credibility argument about the state of the field. The implication of these results—consistent across SOTA policies from different institutions—is that "task-generalist" capabilities, as currently measured by training-distribution benchmarks, substantially overstate true generalization. This is not a criticism of any specific policy; it is a call for the field to adopt evaluation practices that meaningfully test generalization rather than rewarding domain-specific adaptation. The competency-axis breakdown provides the diagnostic language for making this call precise rather than merely polemical: it's not that "policies are bad," it's that "policies have strong relational conjunction reasoning but weak procedural affordance understanding, and knowing which is which tells us where to invest improvement effort."

5. Experimental Analysis

Evaluation Methodology

Dataset. RoboLab-120, a hand-curated benchmark of 120 pick-and-place manipulation tasks, evaluated in the Isaac Sim-based RoboLab simulation environment. The benchmark comprises 65 simple, 38 moderate, and 18 complex tasks, with overlapping competency-axis labels: 44 relational, 91 visual, and 36 procedural tasks. All tasks involve tabletop object manipulation with natural language instructions (Section IV-A).
Base models. Five state-of-the-art generalist robot policies, all fine-tuned on the real-world DROID dataset [13] exclusively—no simulation data is used during training. The evaluated models are: π0.5 [9] (the most capable), π0-FAST [26] (a tokenization-efficient variant), GR00T N1.6 [23], π0 [5], and PaliGemma [4] (a 3B vision-language model adapted for robot control). The paper states these models represent the current frontier of task-generalist policies and explicitly chooses them to span a range of architectures and training recipes.
Metrics. The primary metric is task success rate (binary: completed or not). Complementary metrics include: (1) Normalized score (Section III-C), a partial-credit measure computed as the weighted average of completed subtask events, where 1.0 = full success and intermediate values capture partial progress—the paper reports this both overall and for failed episodes only; (2) Trajectory-quality metrics including SPARC (spectral arc-length, a dimensionless smoothness measure derived from the Fourier magnitude spectrum of the end-effector velocity profile, with values closer to zero indicating smoother motion), mean end-effector speed (cm/s), and path length (total Euclidean distance traveled); (3) Event tracking that logs discrete behavioral events including wrong object grasped, object dropped, and gripper collisions, even when the overall task succeeds; (4) Language variation sensitivity, measured as success rate under vague, default, and specific instruction phrasings of the same underlying task; (5) Scene complexity sensitivity, measured as success rate as a function of the number of visual distractor objects on the table; and (6) Task horizon sensitivity, measured as success rate as a function of the number of sequential subtasks (Section III-C, Figure 8).
Baselines. The paper does not use conventional baselines in the training-comparison sense because all policies are evaluated zero-shot—they are off-the-shelf models trained on DROID without any RoboLab-specific fine-tuning. The primary comparative framework is cross-policy ranking: all five policies are evaluated on the same 120 tasks under the same conditions, with π0.5 serving as the de facto upper bound. For the real-world correlation analysis (Section IV-D), the external baseline is RoboArena[1], an open-source real-world benchmarking system that produces Elo scores for the same policies. For the scene generation evaluation (Appendix C), the baseline is a single-pass LLM-based method that assigns objects to a grid with uniform jitter and runs a physics pass, without the iterative feedback loop.
Generation budget / compute accounting. The unit of evaluation is episodes per task per policy. Each task is evaluated over N = 10 episodes to account for stochasticity in both the simulator (physics randomness) and the policy (sampling-based action generation). Each episode consists of a complete rollout from initial state to termination (success, failure, or timeout), with the policy receiving camera images and proprioceptive state from the simulator and producing action commands at each timestep. The simulation runs on Isaac Sim with GPU-accelerated physics and rendering. The paper notes (Appendix A-A) that with N = 10, the 95% confidence interval on a per-task success rate is approximately ±30% at p = 0.5 and ±19% at p = 0.9, meaning per-task numbers are coarse indicators—aggregate results across all 120 tasks benefit from substantially larger effective sample sizes. The recommendation for per-task conclusions is N ≥ 100 episodes.
Cross-validation / statistical protocol. No formal cross-validation is used for the benchmark results because no model selection or hyperparameter tuning is performed on RoboLab data—the policies are evaluated zero-shot. Statistical significance is addressed through aggregate reporting: results are presented as averages across all 120 tasks (Table I), across difficulty levels, and across competency axes (Table IV, Table V), with standard deviations reported for trajectory metrics. The real-world correlation uses Spearman rank correlation (ρ) and Pearson correlation (r) between RoboLab-120 success rates and RoboArena Elo scores across the four policies where both measurements exist (Figure 10). For the sensitivity analysis (Section IV-C), the MNPE training uses the collected randomized episodes directly, with importance sampling correction to recover the posterior under a uniform prior. The 95% credible intervals are computed from 5000 posterior samples.

Main Quantitative Results

Overall Benchmark Performance (Table I, Table IV, Table V)

The headline result from Table I is that the best-performing model, π0.5, achieves only 28.0% success rate on RoboLab-120, with a normalized score of 0.43. The remaining models rank as follows: π0-FAST at 15.5% success (score 0.27), GR00T N1.6 at 7.2% success (score 0.17), π0 at 5.0% success (score 0.12), and PaliGemma at 3.4% success (score 0.10). The gap between success rate and normalized score is substantial and consistent across policies: on complex tasks, π0.5 achieves 13.5% success but a score of 0.44; π0-FAST achieves 2.9% success but a score of 0.22; both GR00T N1.6 and π0 achieve 0.0% success on complex tasks yet attain scores of 0.12 and 0.09 respectively (Table IV). The "Score (fail)" column in Table IV isolates progress on episodes that do not complete: for π0.5 on complex tasks, the fail-only score is 0.35, meaning that even within failed episodes, the policy reaches more than one-third of the milestones on average. This pattern—substantial partial progress despite low full-completion rates—holds across difficulty levels and competency axes, with the largest score-vs-success gaps appearing on complex tasks and procedural competency categories.

Trajectory-quality results (Table V) reveal additional differentiation between policies. π0.5 achieves the smoothest motion with SPARC −8.34 (±6.65) and the highest mean speed at 5.4 cm/s. PaliGemma shows substantially jerkier motion with SPARC −16.52 (±10.21) and the slowest speed at 1.9 cm/s. Across difficulty levels, SPARC degrades for all policies as task complexity increases: π0.5 goes from −7.38 on simple tasks to −10.83 on complex tasks; π0-FAST goes from −8.09 to −14.38. This degradation indicates that policies become more tentative and oscillatory when task demands exceed their comfortable operating regime.

Competency-Axis Breakdown (Table IV)

The competency-axis results reveal asymmetric generalization across skill dimensions. Relational tasks achieve the highest aggregate success rates (π0.5: 33.8%), followed by visual (23.8%), with procedural tasks substantially lower (21.8%). Within relational, performance varies dramatically: counting achieves 65.7% success for π0.5, conjunction achieves 43.8%, but spatial reasoning achieves only 21.0%. This indicates that policies can interpret object counts ("pick up three apples") and conjunctions ("put A and B on C") much more reliably than spatial prepositions ("put A to the left of B").

Within procedural competency, affordance tasks are most challenging (π0.5: 10.0% success, score 0.27), while stacking shows the highest procedural performance (π0.5: 35.0% success, score 0.57). Reorientation achieves π0.5: 28.3% success. The score-on-failure pattern persists strongly in procedural tasks: on affordance tasks, despite 10.0% success, the fail-only score is 0.19 for π0.5, meaning the policy consistently achieves initial milestones (typically grasping the correct object) but fails to complete the full manipulation sequence.

Within visual competency, all attribute types show low performance: color (23.8% for π0.5), semantics (23.2%), and size (30.0%). The relatively flat profile across visual attributes—compared to the high variance within relational—suggests that visual grounding is a uniformly challenging capability rather than being dominated by specific sub-dimensions.

Language Variation Sensitivity (Figure 8a, Figure 12a)

Figure 8a reports success rates under three levels of instruction specificity for the same underlying tasks and scenes. All policies exhibit substantial sensitivity to instruction phrasing: π0.5 drops from 28.0% on default instructions to 15.3% on vague instructions; π0-FAST drops from 15.5% to 7.6%; GR00T N1.6 drops from 7.2% to 2.8%. The degradation is monotonic with abstraction level across all policies. Figure 12a shows the same analysis using normalized scores instead of success rates—the degradation is more graceful (π0.5: vague score 0.29 vs. default score 0.43), but the trend persists, indicating that even partial progress suffers under vague instructions.

Scene Complexity Sensitivity (Figure 8b, Figure 12b)

Figure 8b isolates the effect of visual clutter by plotting success rate against the number of objects present on the table. Success rates degrade sharply as scene complexity increases for most policies. π0.5 shows some resilience—maintaining non-zero success even at high object counts—while GR00T N1.6 and π0 drop to near-zero beyond moderate clutter levels. PaliGemma performs near floor across all complexity levels. Figure 12b shows that normalized scores degrade more gradually than success rates, confirming that policies continue to make partial progress even when full task completion becomes rare in cluttered scenes.

Task Horizon Sensitivity (Figure 8c, Figure 12c)

Figure 8c shows success rate as a function of the number of sequential subtasks. Performance degrades as task horizon increases for all policies, with success rates approaching zero beyond 4–5 subtasks for most models. A non-monotonic spike appears at subtask=7 for π0.5 (~20% success). As discussed in Appendix A-D, this spike is driven by CubesAndBlocksInBinTask (Figure 11), a task that contains repetitive, independent pick-and-place sequences with geometrically simple cubes rather than a true causal long-horizon task. The paper explicitly cautions that this "recovery" reflects task composition—the subtask=7 bin is dominated by a task with low causal complexity—rather than improved long-horizon reasoning. Outside this confound, the trend is monotonically decreasing.

Sensitivity Analysis (Section IV-C, Table II, Figure 9)

Lighting robustness (Table II). Policies were evaluated under four lighting perturbation conditions: color temperature shifts, directional shadows, dim (reduced exposure), and overexposed (500× intensity increase). π0.5 maintains 90–100% success across all four lighting conditions on the two tested tasks (BananaInBowl, BananaAndCubeInBowl). π0-FAST shows similar robustness (70–100%) except under dim conditions (70%). π0 is more brittle: it achieves 70% under dim conditions but 0% under shadows and overexposed, and only 6.7% under color shifts. This reveals that while the strongest model is robust to lighting variation, mid-tier models retain sensitivity.

Visual appearance robustness (Table II). Variations over 10 background textures and 4 table textures produce minimal degradation for π0.5 (85.0% background, 87.5% table). π0-FAST shows larger drops (70.0% and 60.0% respectively). π0 degrades substantially (25.0% and 22.5%). This indicates that surface-level visual generalization to novel textures is achieved only by the strongest model.

Object pose sensitivity (Table II). Success degrades as object pose randomization radius increases. π0.5: 95.0% at both 10cm and 20cm, dropping to 62.5% at 30cm. π0-FAST: 55.0% at 10cm, 40.0% at 20cm, 35.0% at 30cm. π0: 22.5%, 20.0%, 17.5% respectively. The degradation pattern suggests that the 30cm radius pushes objects outside the comfortable reachability zone for most models.

Camera pose sensitivity (Table II, Figure 9). External camera randomization produces higher success across all policies (π0.5: 85.0%, π0-FAST: 45.0%, π0: 50.0%) compared to wrist camera randomization (π0.5: 60.0%, π0-FAST: 25.0%, π0: 10.0%). This asymmetry—external camera variation tolerated more than wrist camera variation—is consistent across policy tiers.

MNPE posterior analysis (Figure 9) formalizes this observation. The wrist-camera posterior is sharply concentrated near zero displacement for all policies, indicating that successful episodes required the wrist camera to remain close to its nominal calibration. The external-camera posterior is broader, indicating more tolerance to external-camera pose variation. The object-pose posterior peaks at approximately 0.5m from the robot's origin, suggesting that objects placed at this distance have the highest probability of success—interpreted by the paper as falling within a favorable reachability zone.

Real-World Correlation (Figure 10, Section IV-D)

Figure 10 plots RoboLab-120 success rate against RoboArena Elo score for the four policies where both measurements are available (π0.5, π0-FAST, π0, PaliGemma). The Spearman rank correlation is ρ = 1.00, indicating that the two benchmarks induce the identical ordering over policies. The Pearson correlation is r = 0.68, indicating a positive but imperfect linear relationship—consistent with the expectation that simulation success rates and real-world Elo scores should be correlated but not identical, since they measure performance in different environments with different metric structures. The paper presents this as evidence of "benchmark-level correlation," meaning RoboLab-120 preserves real-world policy rankings even if absolute success rates differ.

Ablation Studies and Robustness Checks

Scene generation method vs. baseline (Appendix C, Tables VI–VIII): The paper's iterative LLM-to-solver scene generation method is compared against a single-pass baseline that assigns objects to grid cells with uniform jitter and runs a physics pass without iterative feedback. Evaluated across 100 generated scenes and multiple metrics (VQA score, visual realism, functionality, layout correctness, completeness, quality, GPT preference), the method consistently outperforms the baseline. VQA score: 0.554 vs. 0.398. Visual realism: 8.755 vs. 6.889. GPT preference: 82% vs. 18%. The gap is largest on hard scenes (16–20 objects), where the baseline achieves VQA 0.326 vs. the method's 0.553 (Table VII), confirming that the iterative feedback loop is most valuable in dense, physically constrained configurations.
Scene generation across object counts (Table VII): Performance is stratified by scene density: easy (1–5 objects), medium (6–15 objects), and hard (16–20 objects). The method maintains strong performance across all density levels (VQA: 0.525, 0.561, 0.553 respectively), while the baseline degrades substantially on hard scenes (VQA: 0.326). This demonstrates that the hierarchical placement strategy—using containment and stacking to utilize vertical space, and clustering to group objects spatially—enables valid scene generation even at high object densities where the baseline's grid-based approach fails.
Scene generation across themes (Table VIII): Results are broken out across 10 scene themes (bathroom counter, classroom supplies, craft station, garage workstation, garden tools, kitchen cabinet, laundry sorting, office desk, storage room, tea time, workshop bench). The method outperforms the baseline on all themes and all metrics, with particularly large gaps on themes requiring containment and stacking (kitchen cabinet: VQA 0.554 vs. 0.327; storage room: VQA 0.552 vs. 0.324). This demonstrates robustness of the generation pipeline to diverse thematic contexts.
Task generation quality (Appendix D, Table IX): 812 automatically generated tasks across 59 scenes and 7 competency categories are evaluated using an LLM-as-judge framework. Overall alignment between natural language instructions and programmatic success conditions is 0.91, with 76% of tasks receiving "fully aligned" verdicts and 23% "partially aligned." Recognition tasks achieve highest alignment (0.96), conjunction tasks reach 0.97, while color-based tasks score lower (0.81), reflecting the difficulty of grounding color references to specific object instances in generated code. Instruction clarity is uniformly high (0.96 overall), and semantic match is strong (0.95), indicating that the generated termination conditions capture intended task semantics. Object coverage is 88% (fraction of manipulable objects in each scene that appear in at least one generated task), while predicate coverage is 29%, suggesting the generator conservatively favors a subset of reliable termination predicates. The paper explicitly notes that LLM-generated tasks require user review before deployment.
Score vs. success rate as complementary signals (Table IV, Figures 8 vs. 12): The normalized score consistently exceeds the success rate across all policy-difficulty-category combinations, with the gap largest on complex and procedural tasks. Figure 12 shows that the score-based analysis of complexity axes parallels the success-based analysis in Figure 8 but with more graceful degradation curves—the score declines more gradually than success as language becomes vaguer, scenes become more cluttered, or task horizon increases. This confirms that the score metric captures meaningful partial progress that binary success rates obscure, and that this signal is robust across all three complexity perturbation axes.
Anomalous long-horizon result (Appendix A-D, Figure 8c, Figure 11): The spike at subtask=7 in Figure 8c is identified and explained as a confound: it is dominated by CubesAndBlocksInBinTask, which has 7 subtasks but consists of repetitive, independent pick-and-place sequences with geometrically simple cubes rather than a true causal long-horizon task. The paper documents this transparently and cautions against interpreting Figure 8c as a clean horizon-vs-difficulty curve. This is an important negative finding regarding the interpretability of horizon-based aggregation.
PaliGemma exclusion from robustness tests (Table II): PaliGemma is excluded from the environmental variation experiments because it "fails to achieve meaningful results" on the two basic tasks used for those tests (BananaInBowl, BananaAndCubeInBowl). This is noted as a practical necessity but means the sensitivity analysis for PaliGemma is incomplete.

Critical Assessment

The experiments in this paper are designed to answer three questions posed in the introduction and explicitly addressed in Section IV: Q1: How well do SOTA models generalize given varying language instructions, scene complexity, and environment perturbations? Q2: When and why does a policy fail? Q3: Can a simulated benchmark be used to evaluate real-world models? The experimental design addresses each, but each answer comes with important qualifications.

Q1 is addressed through the central benchmark results (Table I, Figures 8a–8c) and the sensitivity analysis (Table II). The evidence clearly supports the claim that current SOTA policies exhibit limited generalization on a decoupled benchmark: π0.5 achieves 28.0% on 120 conceptually simple tasks, and success degrades under language abstraction, visual clutter, and task horizon. The language variation experiment (Figure 8a) is particularly well-designed because it holds the scenes and underlying task goals constant while varying only the instruction phrasing—this isolates language understanding from visual or motor factors.

However, the interpretation of these results requires care on several fronts. First, the benchmark's difficulty distribution is skewed: 65 simple, 38 moderate, 18 complex. The aggregate 28.0% success rate for π0.5 is a weighted average across this distribution, but the per-difficulty breakdown (Table I) reveals 29.7% on simple, 31.5% on moderate, and 13.5% on complex—rates that are low in absolute terms but follow the expected ordering. Second, the paper does not report what "chance" performance would be. For a pick-and-place task with N distractors on a cluttered table, what is the expected success rate of a random policy? Without this reference point, it is difficult to calibrate how much of the 28.0% reflects genuine capability versus fortunate noise in a task space where occasional accidental successes are possible. Third, the scene complexity experiment (Figure 8b) plots success vs. number of objects, but number of objects is not independent of task type—tasks with more objects tend to be more complex in other dimensions (e.g., longer horizon, more relational reasoning). The observed degradation may conflate visual clutter with task complexity rather than isolating clutter.

Q2 is the paper's strongest contribution, addressed through the normalized score, trajectory metrics, event tracking, and MNPE analysis. The score-vs-success gap (Table IV) provides genuine insight: policies make substantial partial progress on tasks they fail to complete, with the fail-only score revealing that failure tends to occur late in execution (after grasping) rather than early (before any object interaction). The trajectory metrics (Table V) independently corroborate this: SPARC degrades on complex tasks, consistent with the interpretation that policies become hesitant when facing challenging manipulation sequences.

However, the "when and why" diagnosis is limited by the granularity of the event decomposition. The subtask-event structure detects milestones (grasped, hovered, dropped, done) but cannot distinguish between qualitatively different failure modes at the same milestone. If a policy fails after Grasp but before Hover, was the grasp unstable (the object slipped) or was the trajectory planning incorrect (the robot moved to the wrong location)? The event system does not provide this resolution, limiting the specificity of the "why" diagnosis.

The MNPE analysis (Figure 9) demonstrates an intriguing methodology but is applied to only two basic tasks (BananaInBowl, BananaAndCubeInBowl) rather than the full RoboLab-120. The claim that policies are "critically dependent on wrist camera" is well-supported for these two tasks and three policies (π0.5, π0-FAST, π0), but whether this sensitivity generalizes across the benchmark's 120 tasks in diverse scenes is unknown. The paper treats MNPE as a proof-of-concept for the analytical approach, which is appropriate, but readers should not interpret the wrist-camera finding as a universal property of these policies across all task types.

The camera pose sensitivity result in Table II shows an interesting asymmetry: π0 achieves 50.0% success under external camera variation but only 10.0% under wrist camera variation, while π0.5 achieves 85.0% vs. 60.0% respectively. Both models are more tolerant of external camera perturbation, but π0 is almost completely incapacitated by wrist camera perturbation while π0.5 is only moderately affected. This suggests an interaction between model capability and perturbation sensitivity—stronger models have more robust visual representations—but the paper does not explore this interaction systematically across multiple perturbation types or difficulty levels.

Q3 is addressed through the RoboArena correlation (Figure 10) but on a very small sample. Four policies—π0.5, π0-FAST, π0, PaliGemma—have both RoboLab-120 success rates and RoboArena Elo scores. The Spearman ρ = 1.00 is the maximum possible value for four data points, meaning the rankings are perfectly preserved, but with only four policies, this is a weak statistical claim. A Spearman correlation of 1.00 with n = 4 can arise from chance with non-trivial probability, and the paper would benefit from including GR00T N1.6 (which was evaluated on RoboLab-120 but apparently lacks a RoboArena score) or additional policies to strengthen this result. The paper appropriately labels this as "benchmark-level correlation" rather than claiming validated real-world predictiveness.

A deeper concern about Q3 is what "correlation" means here. RoboArena Elo scores are derived from head-to-head comparisons between policies on diverse real-world tasks, while RoboLab-120 success rates are absolute performance measures on a fixed task set. The two metrics have different statistical properties and different sources of variance. The Spearman correlation captures rank agreement, which is a useful property for a benchmark (it correctly identifies which policies are better), but it does not address the question of whether the magnitudes of performance gaps in simulation correspond to magnitudes in the real world. A policy that is 2× better than another in simulation might be only 1.1× better in the real world, and the current analysis cannot detect this.

Unaddressed confound in the difficulty scoring. The difficulty formula is $\text{DifficultyScore} = N_{\text{subtasks}} + \max(w_{\text{skill}})$ . This conflates task length ( $N_{\text{subtasks}}$ ) with skill complexity ( $\max(w_{\text{skill}})$ ) in a specific linear combination with equal weighting. A 3-step visual task ( $3 + 0 = 3$ , moderate) and a 1-step reorientation task ( $1 + 3 = 4$ , also moderate) receive similar difficulty scores but demand fundamentally different capabilities. The paper does not validate whether this scoring produces difficulty bins that correspond to monotonic performance degradation—Figure 8c suggests a generally decreasing trend with horizon, but the confound at subtask=7 reveals that task composition can override difficulty scores. A validation experiment showing that success rate decreases monotonically with DifficultyScore (or explaining deviations) would strengthen the difficulty framework.

Missing experiments that would strengthen the claims. Several experiments are conspicuous by their absence: (1) Multiple random seeds per policy—the paper runs N = 10 episodes per task per policy but does not indicate whether policies are stochastic and whether different random seeds produce different results. If policies are deterministic given observations (which is true for some VLA architectures), then all variation comes from simulator stochasticity, and the policy behavior may not be representative of the full distribution. (2) Calibration analysis—the paper reports normalized scores on a 0–1 scale but does not validate whether a score of 0.5 genuinely corresponds to "half the task completed" in a way that correlates with human judgment of partial progress, which would validate the predicate-based scoring. (3) Inter-rater reliability for the competency-axis labeling—tasks can be labeled with multiple competency attributes, but the paper does not report whether multiple human labelers would agree on these assignments, which affects the reliability of the competency-axis breakdowns. (4) Training data overlap analysis—the paper evaluates policies trained on DROID, but DROID contains thousands of real-world manipulation demonstrations. If some RoboLab-120 tasks are semantically similar to specific DROID demonstrations (even if visually different), the generalization gap would be smaller than it appears. A nearest-neighbor analysis in embedding space would characterize this overlap. (5) Fine-tuning on RoboLab data—the paper argues that decoupling training and evaluation domains prevents memorization, but it does not demonstrate what happens if policies are fine-tuned on RoboLab-similar data. If success rates jump dramatically, that would validate the claim that low zero-shot performance reflects genuine generalization difficulty rather than an unfair simulation gap. If success rates remain low, it would suggest the simulation itself is problematic for policy execution.

The generalizability of the benchmark conclusions is limited by the single robot embodiment and task domain. All experiments use the DROID Franka Panda configuration with a specific camera setup, and all tasks are tabletop pick-and-place. The paper's claim that RoboLab is "robot- and policy-agnostic" (Section III-A) is architecturally true—the framework supports multiple robots—but empirically unvalidated, since only one robot is evaluated. Whether the observed competency-axis patterns (e.g., spatial reasoning substantially harder than conjunction reasoning) hold for mobile manipulators, bimanual setups, or non-tabletop environments is unknown. The paper does not claim these patterns are universal, but readers may over-interpret them as such given the framing around "generalist policy evaluation."

Statistical power caveats. The paper is admirably transparent about statistical limitations in Appendix A-A: with N = 10 trials per task, the 95% confidence interval on a per-task success rate is ±30% at p = 0.5. This means that the per-category success rates in Table IV—while aggregated over multiple tasks—are built from per-task estimates that are individually noisy. The aggregate trends (π0.5 > π0-FAST > GR00T N1 > π0 > PaliGemma) are supported by the 120-task averages, but fine-grained comparisons (e.g., whether π0.5's color recognition at 23.8% is genuinely better than its semantic recognition at 23.2%) are within the noise floor of the experimental design. The paper does not report confidence intervals or statistical tests for these comparisons, which is a limitation for readers who want to make precise claims about relative competency strengths.

The real-world correlation experiment is promising but preliminary. Four data points, one rank preserved, Pearson r = 0.68—this is a suggestive result that requires larger-scale validation. A more compelling experiment would evaluate a wider range of policies (including policy variants that differ only in specific architectural choices) to test whether RoboLab-120 can detect subtle capability differences, not just coarse rankings. If two policies with similar overall capability but different competency profiles (e.g., one better at visual tasks, one better at procedural tasks) showed corresponding differences in their RoboLab-120 competency-axis breakdowns and in real-world evaluation, that would provide much stronger evidence for the benchmark's diagnostic validity.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Claims, and the Benchmark Provides No Lightweight Alternative

The assumption or constraint. The RoboLab framework explicitly depends on a prior distribution over task difficulty to structure its analysis—the benchmark is designed to answer "how well does a policy generalize given known task difficulty?" The paper never claims to provide a method for estimating difficulty cheaply at deployment time (this is a different problem from the one RoboLab solves), but the benchmark's own construction implicitly consumes substantial compute in task authoring, validation, and difficulty scoring. Each of the 120 hand-curated tasks in RoboLab-120 required manual scene creation, language instruction authoring, and metadata annotation (competency axis labels, difficulty scores). The automated scene and task generation pipeline (Section III-A, Appendix C-D) is described as producing "hundreds of unique scenes and tasks," but the paper evaluates only the output quality of this pipeline (Tables VI–IX), not its computational cost. The LLM-to-solver feedback loop in scene generation (Stage I → Stage II → Stage III, with up to 4 progressively relaxed collision margins and a maximum of K_max collision resolution iterations in Algorithm 1) involves multiple LLM calls, physics simulation runs (300 timesteps each), and geometric constraint solving passes. Yet nowhere does the paper report wall-clock time, GPU hours, or API costs for this pipeline.

The consequence. A practitioner who reads the paper and wants to use RoboLab to evaluate their own policy on new tasks faces an unquantified cost. The abstract claims RoboLab "enables rapid extensibility" and the conclusion states "new tasks can be authored in minutes by arranging objects on a tabletop," but this refers only to the manual drag-and-drop workflow with pre-existing assets. The automated LLM-based generation—which is what would enable "continuous benchmark evolution"—is substantially more expensive, and its cost is completely opaque. If a single scene generation cycle requires 3–5 LLM calls at scale (one for initial plan generation, 2–4 for refinement after physics failures), and each LLM call processes a prompt containing the full asset catalog (hundreds of objects with dimensions), the cost per scene could easily reach dollars rather than cents. Generating 100 scenes at this cost adds hundreds of dollars to the benchmark construction budget. This cost must be amortized over the number of policies evaluated on those scenes—if you evaluate only one policy, the scene generation cost could dominate the total experimental budget. The paper's silence on this cost makes it impossible for practitioners to budget for benchmark expansion.

What evidence exists in the paper. None. The paper provides thorough quantitative evaluations of scene generation quality (Tables VI–VIII: VQA scores, realism, functionality, layout correctness, completeness, GPT preference) and task generation quality (Table IX: alignment, clarity, feasibility, match scores). It provides no evaluation of generation cost—no timing benchmarks, no LLM token counts, no GPU-hour estimates for physics simulation. Appendix C describes the algorithmic details (Algorithm 1, Algorithm 2) and the feedback loop structure, but the K_max constant (maximum collision resolution iterations) is unreported, and the number of LLM refinement rounds per scene is not summarized. This is a gap between the scalability narrative ("enables fast creation of hundreds of new tasks") and the empirical grounding of that narrative.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation. The scalability claims rest entirely on the architectural design (modular scene/task definitions, deferred binding, LLM integration) rather than on measured cost. A practitioner would need to replicate the pipeline and measure costs themselves to determine whether automated scene generation is economically viable for their use case.

2. The Benchmark Is Evaluated on a Single Robot Embodiment and a Single Task Family, Limiting Generality of Both the Framework and the Findings

The assumption or constraint. All experiments in RoboLab-120 use a single robot configuration: the DROID Franka Panda 7-DoF arm with Robotiq-2F-85 parallel-jaw gripper, ZED 2i external camera, and ZED mini wrist camera (Section IV-A). The paper explicitly claims RoboLab is "robot- and policy- agnostic" (Section III-A), and the formal environment definition $E = (T, \mathcal{R}, \mathcal{O}, \mathcal{A}, \xi)$ does parameterize the robot $\mathcal{R}$ as a configurable component. However, no experiments evaluate any robot other than DROID Franka, no tasks involve mobile manipulation, bimanual coordination, or non-parallel-jaw grippers, and all tasks are tabletop pick-and-place operations with rigid objects. The paper acknowledges in Section V that RoboLab "currently focuses on rigid-body tabletop scenes and does not fully capture the challenges of deformable object manipulation (e.g., cloth, cables, bags)" and that "many contact-rich skills that require precise force control, compliant interaction, or complex frictional dynamics are underrepresented." These are scope limitations—acknowledged but not experimentally characterized.

The consequence. The central empirical findings of the paper—the 28.0% success rate ceiling for π0.5, the asymmetric generalization across competency axes (e.g., counting 65.7% vs. affordance 10.0%), the sensitivity to wrist-camera displacement, the degradation under language abstraction and scene clutter—are all conditioned on a specific robot embodiment and task domain. We do not know whether any of these patterns generalize to other robots, other end-effectors, other camera configurations, or non-tabletop environments. A policy that achieves 28.0% on Franka tabletop pick-and-place might achieve 50% or 5% on a mobile manipulator doing door-opening tasks—not just because the tasks are different, but because the interaction between the policy's training data and the benchmark's robot embodiment is complex. DROID training data predominantly features the Franka Panda; policies fine-tuned on DROID may have inadvertently learned Franka-specific kinematic priors that transfer to RoboLab's Franka simulation but would fail on a Kinova or Universal Robots arm. The benchmark cannot distinguish between "policy generalizes across embodiments" and "policy has memorized Franka-specific behaviors," because the embodiment is never varied.

The "robot-agnostic" design claim is architecturally true—the framework supports swapping $\mathcal{R}$ —but empirically unvalidated. A benchmark that claims agnosticism but demonstrates it with only one robot is like a programming language that claims to be cross-platform but has only been tested on Linux. The architecture may be correct, but the claim cannot be relied upon without evidence.

What evidence exists in the paper. Zero cross-embodiment experiments. The paper evaluates five policies, all fine-tuned on DROID—which exclusively uses the Franka Panda—and all evaluated on the same simulated Franka Panda. The competency-axis breakdowns, difficulty scaling, and sensitivity analyses are all computed within this single-configuration setting. The paper does not even include a discussion experiment where a policy trained on one robot (e.g., Franka) is evaluated on a different simulated robot (e.g., Kinova) to characterize the embodiment gap. This is a missed opportunity to strengthen the "robot-agnostic" claim.

Mitigation status. Partially addressed through explicit scope acknowledgment in Section V, but no experiments or design provisions are included to characterize the embodiment dependence. The paper does not suggest future work on cross-embodiment evaluation, nor does it provide guidance on what would need to change to support a different robot (different camera calibration? different action space normalization? different workspace bounds?). A practitioner wanting to evaluate policies on a non-DROID robot would need to do substantial integration work—adding a new robot model to Isaac Sim, configuring cameras, calibrating the observation space to match the policy's expected inputs—without any validated template from the paper.

3. The Real-World Correlation Evidence Is Based on Only Four Data Points, Making the Benchmark's External Validity Weakly Supported

The assumption or constraint. Section IV-D presents Figure 10, which plots RoboLab-120 success rates against RoboArena Elo scores for the four policies where both measurements are available: π0.5, π0-FAST, π0, and PaliGemma. The Spearman rank correlation is reported as ρ = 1.00 and the Pearson correlation as r = 0.68. The paper interprets this as evidence that "RoboLab achieves benchmark-level correlation with real-world performance." This is the only external validation of the benchmark's relationship to real-world policy quality. All other analyses in the paper—competency breakdowns, sensitivity analysis, trajectory metrics—are internal to the simulation.

The consequence. A Spearman correlation of 1.00 with n = 4 data points is statistically fragile. To understand why: Spearman's ρ measures whether the rankings match perfectly. With four policies, there are 4! = 24 possible rankings. The null hypothesis (no correlation) would produce a perfect ranking by chance with probability 1/24 ≈ 0.042, which is above the conventional 0.05 significance threshold. This means the observed perfect rank correlation would not be statistically significant at the p < 0.05 level even under generous assumptions about measurement independence—and in practice, the four policies are not independent (π0-FAST is a variant of π0, sharing the same base architecture and training distribution), making the effective sample size even smaller. The paper does not report a p-value for the Spearman correlation, nor does it discuss the statistical power of this comparison.

The Pearson r = 0.68 is more informative about linear relationship strength, but with n = 4 data points, the confidence interval on r is extremely wide. With n = 4, the 95% confidence interval for a true Pearson correlation of 0.68 spans roughly from −0.4 to +0.98 (depending on the distributional assumptions). This means the data are consistent with a true correlation anywhere from moderately negative to near-perfect positive—the evidence simply does not constrain the relationship strongly. A practitioner cannot conclude from this analysis that RoboLab-120 success rates will reliably predict real-world performance for a new, unseen policy.

What evidence exists in the paper. Figure 10, plus the accompanying text reporting Spearman ρ = 1.00 and Pearson r = 0.68. The paper does not include confidence intervals for these correlations, does not report p-values, and does not discuss the sample size limitation. GR00T N1.6 is evaluated on RoboLab-120 (Table I: 7.2% success) but is absent from Figure 10, suggesting it lacks a RoboArena Elo score—if it had one, the sample would be n = 5, which would improve (but not resolve) the statistical power issue.

Mitigation status. Not addressed. The paper presents the correlation as positive evidence without caveats about sample size. The term "benchmark-level correlation" is vague and does not clarify whether it means "policies are ranked in the same order" (which is what is shown) or "performance magnitudes correspond" (which is not shown). Section V does not mention the small sample size as a limitation, focusing instead on domain coverage (rigid bodies, deformables) and residual sim2real visual gap.

4. The MNPE Sensitivity Analysis Is a Proof-of-Concept Applied to Only Two Simple Tasks, Not a Deployed Evaluation Layer

The assumption or constraint. The Bayesian sensitivity analysis via Mixed Neural Posterior Estimation (Section III-D, Section IV-C, Appendix B) is presented as a core contribution: "We introduce a suite of analysis tools that gives insight into the model performance beyond binary success rates" (Section I, contribution 3), and the MNPE framework is described in substantial mathematical detail. However, the experimental application of MNPE in Section IV-C is limited to two simple tasks (BananaInBowl, BananaAndCubeInBowl) and three policies (π0.5, π0-FAST, π0; PaliGemma is excluded because it "fails to achieve meaningful results"). This is a dramatically narrower scope than the full RoboLab-120 benchmark (120 tasks, 5 policies). The camera pose sensitivity posterior (Figure 9) and the object pose sensitivity posterior are inferred from a dataset of randomized episodes on these two tasks only. Table II reports robustness results (success percentages under lighting, visual, object pose, and camera pose variations) on the same two tasks—not on the 120-task benchmark.

The consequence. The sensitivity findings—that policies are highly sensitive to wrist-camera displacement from nominal pose, that external camera displacement is better tolerated, and that optimal object placement is at approximately 0.5m from the robot base—are task-conditional. We do not know whether these findings generalize to the other 118 tasks in the benchmark. A task that requires precise visual servoing to insert a peg into a hole might show stronger wrist-camera sensitivity (because precise alignment is critical). A task that involves broad-area search ("find the apple somewhere on the table") might show weaker wrist-camera sensitivity (because the robot moves to scan the table, changing the wrist camera's viewpoint anyway). The MNPE posterior is learned from a specific task distribution, and its shape—where it is sharp vs. broad—is a fingerprint of the task-policy interaction, not a pure property of the policy. Reporting it as a finding about "policies" rather than about "policy π0.5 on task BananaInBowl" risks overgeneralization.

This limitation is particularly consequential because the MNPE analysis is one of the paper's most distinctive technical contributions—it differentiates RoboLab from prior benchmarks that report only robustness percentages. If the analysis is not practically deployable at scale, its value as a benchmarking tool is substantially diminished.

What evidence exists in the paper. The MNPE experimental design and results are described in Section IV-C, Table II, and Figure 9. The task scope is stated explicitly: the experiments are "given two basic tasks" (Section IV-C text preceding Table II). The paper never claims the MNPE results apply to all 120 tasks. However, the presentation in the abstract and introduction ("we introduce a suite of analysis tools...") frames MNPE as a general capability of the framework, and readers who do not carefully check the experimental scope may assume the sensitivity findings are benchmark-wide.

Mitigation status. The paper treats MNPE as a demonstrated methodology rather than a fully deployed evaluation layer—the section introducing it describes what the framework can do, and the experiments demonstrate it on a subset. The method itself is mathematically sound and the implementation is described in sufficient detail for replication. The missing piece is scaling: applying MNPE to all 120 tasks, across all policies and perturbation types, would require substantially more data collection (randomized episodes per task-policy-perturbation combination) and training (one density estimator per policy-perturbation pair, or a single amortized estimator). The paper does not discuss the computational cost of scaling MNPE, nor does it provide guidance on minimum sample sizes for reliable posterior inference with the 2–3 continuous parameters used in the experiments.

5. Ten Episodes per Task Provides Insufficient Statistical Power for Per-Task and Fine-Grained Comparisons, Yet the Competency-Axis Breakdowns Depend on Per-Task Aggregation

The assumption or constraint. Each policy is evaluated on each task with N = 10 episodes (Section IV-A). The paper acknowledges this limitation explicitly in Appendix A-A:

"With only 10 trials, the 95% confidence interval on a single per-task success rate is approximately ±30% near p=0.5 and ±19% near p=0.9, meaning per-task numbers should be interpreted as coarse indicators rather than precise estimates."

The paper argues that aggregate scores across all tasks are "considerably tighter, since the effective sample size scales with the number of tasks," and recommends N ≥ 100 episodes per task for "per-task conclusions or resolving differences smaller than 10% between policies."

The consequence. The paper's own recommendation (N ≥ 100) is not followed for the main benchmark results. This creates a tension between the paper's analytical ambitions and its statistical foundation. The competency-axis breakdowns in Table IV report success rates for categories with as few as 6 tasks (size, stacking, reorientation). With N = 10 episodes per task, the per-category success rates are averages over 60 episodes for 6-task categories—an improvement over per-task rates, but still subject to substantial sampling variance. A 95% confidence interval on a 6-task category with p ≈ 0.3 (typical for these policies) and 60 total trials is approximately ±12 percentage points, meaning that the reported differences between closely spaced categories (e.g., π0.5 color recognition 23.8% vs. semantic recognition 23.2%) are well within the noise floor. The paper does not report confidence intervals or conduct statistical tests for any of these comparisons, leaving readers to interpret small differences as meaningful when they may reflect sampling noise.

The trajectory-quality metrics (Table V) further compound this issue. SPARC is reported as a mean with standard deviation across episodes. For categories with 6 tasks × 10 episodes = 60 observations, the reported standard deviation (e.g., π0.5 on procedural tasks: SPARC −10.08 ± 4.93) reflects episode-to-episode variation, but the standard error of the mean (which determines confidence in the estimated SPARC) is the standard deviation divided by √60 ≈ 7.75, yielding a standard error of roughly 0.64. Differences between categories (e.g., π0.5 SPARC −7.50 on relational vs. −10.08 on procedural) are large relative to this standard error and are likely statistically significant, but the paper does not perform this calculation or report it, leaving readers to guess at which comparisons are reliable.

What evidence exists in the paper. Appendix A-A acknowledges the issue in general terms. The recommendation for N ≥ 100 is provided but not followed. No confidence intervals, standard errors, or statistical tests appear in any of the main results tables or figures.

Mitigation status. Partially addressed through transparency about the limitation (Appendix A-A) and through aggregate reporting (120-task averages in Table I have an effective sample size of 1200 episodes per policy, which is reasonably well-powered for mean success rate estimation). The paper's primary claims—the overall success rates, the policy ranking, the broad competency-axis patterns—are supported by aggregate statistics. However, the granular competency profiles and per-category comparisons that the paper uses to support its diagnostic claims (e.g., "counting > conjunction > spatial" within relational) are underpowered for the precision with which they are presented. A practitioner who wants to make design decisions based on per-category performance differences (e.g., "improve spatial reasoning because it is the bottleneck") would need to collect substantially more data to confirm those differences with statistical confidence.

6. The Benchmark Cannot Distinguish Between Genuine Task Generalization Failure and an Unfavorable Simulation Domain Gap, and the Paper Provides Only Preliminary Evidence That This Distinction Matters

The assumption or constraint. The central design principle of RoboLab is training-evaluation domain decoupling: policies are trained on real-world DROID data and evaluated in Isaac Sim, ensuring that the benchmark measures generalization to novel visual and physical conditions rather than memorization of simulator-specific patterns (Section II, "What RoboLab does differently"). This design relies on the implicit assumption that the Isaac Sim environment is a fair proxy for real-world evaluation—that it preserves the relative ordering of policies and does not introduce systematic artifacts that affect some policies more than others.

The paper acknowledges in Section V that "a residual visual distribution shift remains" and that "this gap needs to be characterized further both by analyzing the behavior and robustness of the visual perception stack and through extensive validation on real-world deployments." This is an honest acknowledgment, but it leaves open a critical question: when a policy fails on RoboLab-120, is it failing because it genuinely cannot generalize to the task, or because the simulation introduces a visual or physical domain shift that disproportionately affects that specific policy architecture?

The consequence. The benchmark's diagnostic signals—competency-axis breakdowns, sensitivity analyses, normalized scores—are only interpretable as signals about task generalization if the simulation domain gap affects all policies and all task types roughly uniformly. If the gap is non-uniform—if, say, PaliGemma's visual encoder is more brittle to simulation-specific rendering artifacts than π0.5's visual encoder—then the observed performance differences confound task generalization capability with simulator robustness. A policy that scores 3.4% on RoboLab-120 (PaliGemma) might genuinely be worse at task generalization than a policy that scores 28.0% (π0.5), or it might be roughly comparable at task reasoning but much more sensitive to the specific visual appearance of Isaac Sim's rendering. The benchmark cannot distinguish between these explanations because it provides no analysis of why the simulation gap affects each policy differently.

This is not a hypothetical concern. Table II shows that under dim lighting, π0 achieves 70.0% success (its best lighting condition) while under color shifts it achieves only 6.7%. This 10× variation for the same policy across lighting conditions that are all simulated in Isaac Sim—not real-world—suggests that the simulator's lighting model interacts with policy-specific visual encoders in ways that produce large performance swings. If different policies have different sensitivity profiles to simulation-specific lighting artifacts, then the aggregate RoboLab-120 ranking reflects a mixture of task generalization and simulator robustness, and the competency-axis profiles are contaminated by task × simulator interaction effects that cannot be isolated.

What evidence exists in the paper. The real-world correlation experiment (Figure 10) provides the only direct evidence on this question, and it is limited to 4 policies with one rank correlation coefficient. The paper does not conduct any experiment that would characterize the uniformity of the domain gap: there is no analysis of whether policies that perform similarly on RoboArena also perform similarly on RoboLab-120 in terms of absolute success rate (beyond rank ordering), no comparison of failure modes in simulation vs. real world for the same task (e.g., does the policy fail at the same subtask milestone in both domains), and no experiment that incrementally bridges the simulation-to-real gap (e.g., by using the Gaussian Splat + Mesh hybrid scenes described in Appendix C, Figure 13) to measure how performance changes as visual fidelity increases.

Mitigation status. The paper acknowledges the gap (Section V) and provides the preliminary RoboArena correlation (Figure 10), but treats these as initial evidence rather than a systematic characterization. The hybrid Gaussian Splat + Mesh scenes are described as an extension (Appendix C) but are not used in any experiment, missing an opportunity to test whether closing the visual fidelity gap changes policy rankings or absolute performance. The paper frames further characterization as future work ("This gap needs to be characterized further"), which is appropriate for a benchmark paper but leaves practitioners with an important uncertainty when interpreting RoboLab-120 results.

7. Implications and Future Directions

How This Work Changes the Landscape

RoboLab does not introduce a new model architecture, a new training algorithm, or a new theoretical framework for robot learning. What it introduces is a methodological standard for what rigorous evaluation of generalist robot policies should look like—and in doing so, it reshapes what evidence the field should demand before accepting claims of generalization.

This is not a paradigm shift; it is a diagnostic intervention. The paradigm—training large-scale VLA models on diverse real-world datasets and evaluating them on held-out tasks—remains intact. But RoboLab demonstrates, through concrete empirical results, that this paradigm's evaluation practices are systematically overestimating generalization. The paper's central empirical finding—that π0.5, a model described as exhibiting "open-world generalization" [9], achieves 28.0% success on 120 conceptually simple pick-and-place tasks when training and evaluation domains are decoupled (Table I)—is not a critique of any specific policy. It is evidence that the field's default evaluation methodology conflates domain adaptation (performing well on tasks that look like training data) with task generalization (performing well on tasks that require the same underlying competencies but in visually and procedurally novel contexts). RoboLab makes this conflation legible.

The conceptual shift, if it takes hold, is that benchmarks for generalist policies must have evaluation domains that are genuinely out-of-distribution relative to training data. This sounds obvious—it is what machine learning benchmarks are supposed to do—but the paper documents that prior robot manipulation benchmarks (LIBERO, RLBench, MetaWorld, CALVIN) violate this principle by training and evaluating in the identical simulator, creating a domain overlap that makes benchmark performance a weak proxy for generalization. RoboLab's architectural decision to decouple training (real-world DROID data) and evaluation (Isaac Sim) transforms the benchmark from a test of familiarity into a test of transfer. The 28.0% success rate, far from being a disappointing result, is evidence that the benchmark is functioning as designed—it is discriminating between policies in a regime where prior benchmarks would show saturation.

The paper also resolves a tension that pervades recent robotics research. On one hand, generalist policies are described in increasingly ambitious terms: "open-world generalization," "zero-shot task execution," "foundation models for robot control." On the other hand, systematic evaluations—including this very paper—show sharp performance degradation under seemingly mild distribution shifts: vaguer language, more visual clutter, longer task horizons, displaced cameras. RoboLab provides a framework for reconciling these narratives. The competency-axis breakdown (Table IV) shows that generalization is not monolithic: policies are genuinely strong at relational counting (π0.5: 65.7%) and genuinely weak at procedural affordance reasoning (π0.5: 10.0%). A demonstration video can cherry-pick the former; a rigorous benchmark exposes the latter. The gap between the narrative and the data is not a failure of the policies—it is a failure of evaluation practices to capture the full distribution of policy behavior.

Research directions that become more attractive:

Robust visual representations for robot manipulation. The MNPE analysis (Figure 9) identifies wrist-camera sensitivity as a critical failure mode—successful episodes require the wrist camera to remain within a narrow range of its nominal calibration. This makes improving visual encoder robustness across camera viewpoints a high-priority research target, especially given that the wrist camera moves continuously during manipulation. Techniques from multi-view representation learning, test-time viewpoint adaptation, and data augmentation with aggressive camera pose randomization become directly motivated by a benchmark that quantifies the sensitivity.
Difficulty-adaptive policy architectures. Figure 8c shows that all policies degrade with task horizon, and the competency-axis breakdown (Table IV) shows that procedural reasoning lags far behind relational language understanding. This suggests architectures that explicitly model task structure—decomposing long-horizon instructions into subgoals, maintaining plan representations across steps, or using separate pathways for language grounding and action sequencing—would be more testable with a benchmark that isolates these competencies.
Verifier-guided policy improvement for robotics. The normalized score metric (Section III-C) is effectively a learned verifier—a set of predicate functions that evaluate partial progress. The observation that policies often make substantial partial progress before failing (e.g., π0.5 scores 0.44 on complex tasks despite 13.5% success) suggests an obvious follow-up: use these verifier signals as reward functions for fine-tuning. This is the robotics analog of using PRM-based search to improve LLM reasoning, and RoboLab provides the infrastructure (predicate library, automatic scoring) to make this feasible without human annotation.

Research directions that become less attractive:

Scaling model size without improving evaluation rigor. If the generalization gap documented here (28.0% success ceiling for SOTA models on simple tasks in a decoupled domain) persists as models scale, then simply training larger VLA models on the same data distributions will produce diminishing returns on generalization even if it improves training-distribution performance. RoboLab provides the tool to measure this distinction, making it harder to claim generalization improvements without evidence from a decoupled benchmark.
Real-world-only evaluation at scale. The RoboArena correlation (Figure 10, Spearman ρ = 1.00 across 4 policies) provides initial evidence that a well-designed simulated benchmark can preserve real-world policy rankings at a fraction of the cost. If this correlation holds as more policies are evaluated, the case for running expensive real-world evaluations to rank policies becomes weaker—simulation can handle the ranking, and real-world evaluation can focus on calibration (validating that the ranking holds) and on edge cases where simulation fidelity is insufficient.

Follow-Up Research This Work Enables

Extending the RoboArena correlation to a larger, more diverse policy set with statistical rigor. Figure 10 establishes a perfect Spearman rank correlation across 4 policies, but with n = 4, this is statistically fragile—perfect ranking could occur by chance with probability 1/24 ≈ 0.042, above the conventional 0.05 significance threshold. The most important immediate follow-up is to evaluate 10–20 policies—including policy variants that differ only in specific architectural choices (e.g., vision encoder backbone, action prediction head, training data mixture)—on both RoboLab-120 and RoboArena, and compute not just the rank correlation but the full regression of real-world performance on simulated performance, with confidence intervals. This would answer the critical calibration question: if a new policy achieves X% on RoboLab-120, what is its expected RoboArena Elo, and what is the uncertainty? The paper's existing infrastructure (RoboLab-120, the DROID robot configuration, support for multiple policies) makes this straightforward to execute—the bottleneck is running the RoboArena real-world evaluations for each policy. A strong result would show that the correlation holds across a diverse policy set (Spearman ρ > 0.9 with n ≥ 10), while a weak result—e.g., policies that rank differently in simulation vs. real world—would identify specific competency dimensions where the simulation gap is non-uniform, guiding targeted improvements to the benchmark.

Scaling the MNPE sensitivity analysis to all 120 tasks to produce per-competency sensitivity profiles. The MNPE analysis in Section IV-C demonstrates that posterior inference over environmental parameters conditioned on task success can reveal why policies fail, but it is applied to only two simple tasks (BananaInBowl, BananaAndCubeInBowl). The natural extension is to run the same camera pose, object pose, lighting, and visual variation experiments across a representative subset of RoboLab-120—say, 20 tasks spanning all three competency axes and all three difficulty levels—and train per-task or amortized MNPE posteriors for π0.5 and π0-FAST. This would produce a sensitivity matrix: for each task, which environmental parameter has the narrowest success-conditional posterior? Are visual competency tasks primarily sensitive to camera pose (because the policy needs clear views to identify objects by color) while procedural tasks are primarily sensitive to object pose (because reorientation requires precise approach angles)? If this pattern holds, it would validate the competency-axis framework by showing that different competency dimensions have qualitatively different sensitivity signatures. If no pattern emerges, it would indicate that sensitivity is task-idiosyncratic rather than competency-structured, refining our understanding of what the competency axes actually capture.

Training a difficulty predictor from scene and task metadata to enable compute-adaptive evaluation. The paper's difficulty scoring (DifficultyScore = N_subtasks + max(w_skill)) is a coarse heuristic, and the benchmark provides no mechanism for predicting task difficulty before running policy evaluation. A practical extension would train a regression model that takes scene features (number of objects, number of containers, spatial clutter metrics), task features (instruction length, number of subtasks, competency-axis labels), and policy features (model architecture, training data size) as input, and predicts the expected success rate or normalized score. Training data would come from the RoboLab-120 results across all five policies (or from running additional policies to increase the sample). A strong predictor would enable compute-adaptive benchmarking: allocate more evaluation episodes to task-policy combinations where the predicted success rate is in the discriminative range (20–80%), and fewer to combinations where the policy is either near-perfect or near-floor. This is the test-time compute allocation problem studied in the LLM scaling literature, applied to robot policy evaluation—a natural cross-pollination enabled by RoboLab's large-scale task framework.

Closing the simulation-to-real visual gap using the Gaussian Splat + Mesh hybrid scenes and measuring whether it changes policy rankings. Appendix C (Figure 13) describes a scene variant where the background is a 3D Gaussian Splat reconstruction with collision mesh, while foreground objects remain standard mesh assets. This hybrid approach is not used in any experiment. A critical follow-up would select 10–20 RoboLab-120 tasks, reconstruct their scenes using the hybrid pipeline (which trades off generation speed for visual fidelity), re-evaluate π0.5 and π0-FAST on these high-fidelity variants, and compare success rates and failure modes to the mesh-only versions. If success rates increase significantly on the hybrid scenes for both policies, the residual visual gap matters for absolute performance but not for relative ranking—this would strengthen the case that mesh-only RoboLab-120 is sufficient for policy comparison. If success rates change differently for different policies (e.g., π0-FAST improves more than π0.5), it would indicate that the simulation gap is architecturally non-uniform and that the benchmark's policy ranking is partially an artifact of simulator fidelity rather than genuine generalization capability. Either outcome is informative, and both are testable with the existing infrastructure.

Using the normalized score and predicate library as a reward signal for fine-tuning policies to improve procedural reasoning. The paper's most striking diagnostic finding is that policies achieve substantial partial credit (normalized scores of 0.3–0.5) on tasks they do not complete, particularly in procedural categories like affordance and reorientation where success rates are as low as 10.0% for π0.5 (Table IV). The predicate functions that compute these scores—Grasp(obj), Hover(target), Drop, Done, and their compositional combinations—define a dense reward signal that is automatically available for every episode without human annotation. A follow-up would fine-tune π0.5 (or a smaller open-source VLA) on RoboLab scenes using reinforcement learning or behavioral cloning with this predicate-based reward, and measure whether the normalized score on held-out tasks improves. The hypothesis: partial credit signals can guide the policy toward completing the final stages of execution that currently cause failure, converting near-success episodes into full successes. If this works, it would transform RoboLab from an evaluation-only benchmark into a self-improvement platform—policies improve by practicing on RoboLab tasks and receiving automatic feedback. If it fails (policies overfit to the predicate structure without genuinely improving manipulation), that negative result would reveal a fundamental limitation of verifier-guided training in robotics: the gap between "satisfying a predicate function" and "robustly executing a physical skill."

Systematically characterizing the embodiment gap by evaluating the same policy on multiple simulated robot morphologies. The paper claims RoboLab is "robot-agnostic" (Section III-A), but all experiments use the DROID Franka Panda. A stress-test of this claim would select one or two RoboLab-120 tasks that do not require Franka-specific kinematics—simple pick-and-place on a centrally located object—and evaluate π0.5 on three simulated robots: the Franka Panda (same as training data), a Kinova Gen3 (different kinematics, similar 7-DoF arm), and a Universal Robots UR5 (different kinematics, 6-DoF). If π0.5 achieves similar success rates across embodiments, the policy has genuinely learned embodiment-agnostic manipulation. If performance degrades sharply on non-Franka arms, the policy's apparent generalization is partly an artifact of embodiment overfitting to the training distribution—and RoboLab's claim of robot-agnosticism, while architecturally valid, would not translate to fair cross-embodiment evaluation without additional calibration (e.g., action space normalization, observation adaptation). This experiment is feasible because Isaac Sim already supports multiple robot models, and π0.5's action space (7-DoF joint positions) can be mapped to other arms with minimal engineering.

Practical Applications and Downstream Use Cases

Pre-deployment safety and robustness screening for generalist policies in commercial robotics. A robotics company developing a VLA-based generalist policy for warehouse pick-and-place or home assistance could integrate RoboLab into their CI/CD pipeline: before deploying a new model checkpoint to physical robots, evaluate it on a representative subset of RoboLab tasks with controlled environmental perturbations (lighting, camera pose, background clutter, object pose). The normalized score provides a more granular pass/fail signal than binary success rates—a policy that achieves 80% success but with a normalized score of 0.95 on failed episodes (indicating near-completion) is safer to deploy than a policy with 80% success but a score of 0.3 (indicating sharp failures after initial progress). The sensitivity analysis identifies which environmental conditions the policy is brittle to, enabling targeted data collection before deployment. The key practical benefit is cost: running 120 tasks × 10 episodes in simulation costs GPU hours, not physical robot technician time, making safety screening economically viable at the cadence of model retraining (potentially daily or weekly).

Academic benchmarking with reduced variance and improved signal-to-noise for VLA research. The current state of academic VLA evaluation—each lab evaluates on different real-world tasks with different objects, different robots, and different success criteria—makes it nearly impossible to compare results across papers. RoboLab-120 provides a standardized evaluation set with automated scoring, consistent environment parameters, and fine-grained competency-axis breakdowns. A research group developing a new VLA architecture could evaluate on RoboLab-120 and report not just an aggregate success rate but a competency profile (Table IV) and a trajectory-quality profile (Table V), enabling direct comparison with published results for π0.5, π0-FAST, and other models. The key practical benefit is comparability: a reader can see at a glance that a new model improves affordance reasoning from 10.0% to 15.0% (relative to π0.5) while maintaining counting performance, rather than comparing vaguely described "manipulation success rates" from different papers with different evaluation protocols. This is the role that standardized benchmarks like ImageNet and GLUE played in their respective fields—reducing evaluation variance to accelerate the pace of methodological comparison.

Data curation for targeted policy improvement in industry. A company that has deployed a generalist policy and observes that it struggles with specific failure modes (e.g., dropping objects during reorientation, misidentifying objects by color under warm lighting) can use RoboLab's scene and task generation pipeline to automatically generate training data targeting those failure modes. The competency-axis templates (Section III-B) enable specifying "generate 50 tasks that exercise color-based visual recognition under warm color-temperature lighting with 10–15 distractor objects," and the LLM-to-solver loop produces executable, validated scenes without manual authoring. The policy can then be fine-tuned on demonstrations collected in these scenes. The key practical benefit is targeted data generation at scale: rather than collecting expensive real-world demonstrations for specific failure cases, a company can generate hundreds of simulated scenarios, fine-tune the policy, and verify improvement on held-out RoboLab tasks before deploying to physical robots. The paper's finding that π0.5 only achieves 23.8% on color recognition tasks (Table IV) and drops from 96.7% to 90.0% under dim lighting on the sensitivity tasks (Table II) provides concrete baseline numbers against which improvement from such targeted fine-tuning can be measured.

Verification that "generalist" claims in published research are not artifacts of narrow evaluation. When a new VLA paper claims "open-world generalization" or "zero-shot task performance," RoboLab-120 provides a standardized third-party audit for these claims. A reviewer, meta-analysis, or competitor can evaluate the published model checkpoint on RoboLab-120 without any cooperation from the original authors, because the benchmark uses only simulation and automated scoring. If a model that claims 90% success on its own evaluation tasks achieves 30% on RoboLab-120, that gap—and its structure across competency axes—becomes part of the public evidence about what "generalization" actually means for that model. The key practical benefit is accountability: claims of generalization become empirically testable by independent parties, creating a incentive for authors to characterize their models' limitations more honestly and for the community to converge on a shared understanding of what current models can and cannot do. This is the function that rigorous benchmarks serve in mature scientific fields—they make claims falsifiable.