SAGE: Scalable Agentic 3D Scene Generation for Embodied AI

🎯 Pitch

Embodied AI policies trained solely in simulation need physically plausible scenes, yet even state-of-the-art generators produce environments where objects float in midair or clip through furniture nearly a third of the time. SAGE flips this by baking physics simulation directly into the generation loop, yielding 99.9% physically stable 3D scenes—and policies trained on this data show clear scaling laws for real-world generalization.

1. Executive Summary

SAGE introduces an agentic framework that converts open-vocabulary text prompts into simulation-ready 3D environments by adaptively orchestrating multiple generators—scene initialization, asset placement, movement, and removal—under the Model Context Protocol (MCP), while two complementary critics provide iterative feedback: a visual critic for semantic and spatial coherence (e.g., detecting missing or misplaced objects from multi-view renderings) and a physics critic with simulator-in-the-loop validation (e.g., checking stability under gravity in Isaac Sim and rejecting placements that cause collisions). Evaluated on three common indoor scene types (Bedroom, Kitchen, Living Room) against Holodeck and SceneWeaver, SAGE achieves 99.9% physical stability (versus 63.8% and 67.7% for baselines), reduces collision rates to 1.9% (versus 22.0% and 32.8%), and scores highest across all visual quality metrics—realism (8.8 vs. 7.5/8.6), functionality (9.5 vs. 6.7/8.9), layout (7.9 vs. 5.0/7.3), and completeness (8.2 vs. 6.5/7.2). Policies trained on SAGE-generated data exhibit clear scaling trends—success rates climbing with scene diversity and demonstration count on Pick-and-Place and Mobile Manipulation tasks—establishing that simulation-validated, agentically-generated environments can drive effective embodied policy learning while generalizing to unseen objects and layouts, though the current scope is limited to indoor rigid-body scenes.

2. Context and Motivation

The Core Problem: Generating Simulation-Ready 3D Environments at Scale for Embodied AI

The fundamental problem this paper addresses is that embodied AI suffers from an acute data shortage, and existing approaches to generating training data—particularly 3D simulation environments—cannot simultaneously satisfy the four requirements of realism, diversity, simulation-readiness, and task-awareness at scale.

This is not a niche concern. Embodied agents—robots that interact with the physical world—need to practice tasks millions of times to develop reliable behaviors. But doing this in the real world is prohibitively expensive, slow, and sometimes dangerous. As the authors state in the opening paragraph:

"Real-world embodied data collection is slow and costly, and is fundamentally constrained by the need for interactive environments."

The economics are brutal: physically setting up a kitchen scene, having a robot attempt a pick-and-place task thousands of times, resetting objects between trials, and ensuring safety would consume enormous human labor and calendar time. Moreover, certain tasks—manipulating fragile objects, operating in hazardous environments like fire scenes—are inherently unsafe for iterative trial-and-error learning on physical hardware.

Simulation therefore emerges as the natural alternative. It is inherently scalable (you can run thousands of parallel instances on GPU clusters), deterministic and repeatable (you can reset to the exact same initial conditions for fair comparisons), and safe (there are no real-world consequences when the robot fails). The paper cites a rich ecosystem of simulation platforms—Isaac Sim, MuJoCo, Habitat, PyBullet, SAPIEN, Genesis—all testimony to the field's recognition that simulation is essential.

But simulators are only as good as the scenes you put inside them. This is where the data bottleneck shifts: you can simulate, but what exactly do you simulate in? The environments themselves need to exist first, and creating them is the crux of the problem.

The Four Desiderata and Why Simultaneous Satisfaction Is Hard

The paper lays out four requirements that a simulation environment must satisfy to be useful for embodied AI (Section 1):

Realism: The geometry, appearance, semantics, and physics in simulation must resemble the real world closely enough that policies learned in simulation can transfer to reality. If the simulated apple looks nothing like a real apple, or slips unrealistically on the table, the learned grasping policy won't work on a physical robot. This is the classic "sim-to-real" transfer problem.
Diversity: Simulation environments must encompass a wide range of assets, environments, and tasks. If a robot is trained on only one specific kitchen with one specific mug, it will be brittle—unable to generalize to different mug shapes, different table heights, or different room layouts. Diversity prevents overfitting and supports generalization to unseen conditions.
Simulation-readiness: Objects and scenes must be physically stable, interactable, and directly compatible with modern simulators. This means objects must have appropriate mass properties, collision geometries, and material parameters (e.g., friction, metallic/roughness for physics-based rendering). A scene that looks beautiful in a static renderer but collapses under gravity when loaded into a physics engine is useless for robot training. The paper is explicit about this in the introduction:

"There are still major gaps in (1) physical grounding for interaction and (2) compatibility with robot simulators"
Task-awareness: Environments should be adaptable to specific embodied tasks. A data engine should be able to generate kitchens for household robots, fire-hazard scenes for rescue robots, or surgical rooms for medical robots, all from user-specified task descriptions.

The challenge is that existing approaches optimize for some of these at the expense of others. This is the fundamental tension that motivates the paper: no prior method achieves all four simultaneously at scale.

Where Prior Approaches Fall Short

The paper systematizes existing methods into four broad families, each with characteristic strengths and failures. Table 1 in the paper provides a structured comparison; the narrative in Sections 1 and 2 explains why each family falls short.

Real2Sim: High Realism, Poor Scalability

Real2Sim approaches—represented by works like DRAWER, Video2Game, HoloScene, and PhysTwin—reconstruct digital twins from real-world data (e.g., scanning a real kitchen with cameras and reconstructing it as a 3D model). These methods achieve the highest realism because they directly capture real geometry, textures, and object arrangements.

Why they fail for scalable training data generation: The process of capturing, processing, and reconstructing a single real-world environment is extraordinarily labor-intensive. It requires physical access to the space, specialized hardware and software pipelines, and extensive manual cleanup to fix reconstruction artifacts. The paper states:

"the high costs of data capture and reconstruction make them difficult to scale."

To generate thousands of diverse kitchens, you'd need access to thousands of real kitchens—an obviously infeasible proposition. Real2Sim is excellent for creating high-fidelity digital twins of specific environments, but it cannot produce the volume and diversity of scenes needed for generalizable policy learning.

Furthermore, Real2Sim only replicates existing environments—it does not enable the creative generation of novel, task-specific environments. You can't ask a Real2Sim pipeline to "generate a cyberpunk game den" or "a kitchen with fire hazards for rescue robot training" because it can only reproduce what exists.

Rule-Based/Procedural Systems: Physical Plausibility, Limited Flexibility

Procedural systems like Infinigen Indoors and ProcTHOR generate scenes by following hand-crafted rules and grammars. For example, a rule might specify that "in a kitchen, a stove must be placed against a wall, a sink must be near the stove, and a table should be in the center of the room with chairs around it." These systems can produce physically plausible layouts because the rules encode common-sense spatial constraints, and they scale well because the rules can be applied programmatically to generate many variations.

Why they fail: The rules are fixed. This limitation manifests in several critical ways:

No open vocabulary: The system can only generate room types and object categories that are explicitly encoded in its rules. You cannot prompt a procedural system with "rusty and dusty restroom" or "starry-night bedroom" and expect it to understand the aesthetic intent and produce an appropriate scene. The vocabulary is closed at design time.
No fine-grained control: If you want to specify that "the mug should be on the left side of the table, near the window," a procedural system typically does not expose that level of per-object control. The rules are global, not instance-specific.
No self-improvement: Because the procedural pipeline is a fixed "computational graph" executing a predetermined sequence of operations, it cannot adapt based on feedback. If it places a chair that clips through a table, there's no mechanism to detect this failure and try a different placement. The paper explicitly calls this out:

"fixed recipes constrain open vocabulary, fine-grained control, and self-improvement, and rarely expose physics-rich attributes beyond hand-coded rules."
No physics validation: Procedural systems typically do not run physics simulations during scene construction. They place objects according to geometric rules but do not verify that those placements are stable under gravity. The resulting scenes may have floating objects, objects resting on surfaces at angles that would cause them to slide, or objects interpenetrating in ways that would generate explosive forces when the physics engine is initialized.

Data-Driven Approaches: Realistic Layouts, Closed Taxonomies

Data-driven methods—including ATISS, DiffuScene, CommonScenes, and EchoScene—learn spatial priors from datasets of human-designed 3D scenes (e.g., 3D-FRONT, which contains professionally designed interior layouts). These models learn the statistical patterns of how objects co-occur and are arranged relative to each other, and can then generate new layouts that follow similar patterns. ATISS uses autoregressive transformers to predict object placements sequentially; DiffuScene uses a diffusion model to generate the full scene layout in one shot; CommonScenes and EchoScene incorporate scene graph structure to enforce relational constraints.

Why they fail for embodied AI data generation:

Closed taxonomies: These models are trained on datasets with fixed sets of object categories (e.g., the 3D-FRONT dataset has a specific vocabulary of furniture types). They cannot generate objects outside this vocabulary. If the training data contains only modern furniture, the model cannot generate "cyberpunk" or "fairy-tale princess room" aesthetics.
Limited object-level levers: You typically cannot specify detailed per-object properties (e.g., "make this chair wooden with a blue cushion"). The model draws from a fixed asset library—it can choose which chair to place but cannot generate a novel chair matching a specific description.
No physical properties: The models operate on geometric representations (bounding boxes, poses) without attaching mass, friction coefficients, or collision meshes. The generated scenes are static 3D models, not physics-ready simulation environments.
No physics validation: Similar to procedural systems, these methods cannot verify that the generated placement is stable or interaction-ready. An ATISS-generated living room might look realistic to a human but collapse when loaded into Isaac Sim because the lamp is balanced precariously on the edge of a table with physically implausible support.
Data hunger with weak generalization: Because these models learn from specific datasets, their ability to generalize to novel room types or stylistic prompts is bounded by the coverage of the training data. The paper notes:

"their limited 3D training data prevents them from generalizing to new room types, handling open-vocabulary prompts, or supporting fine-grained layout control."

LLM/Foundation Model Pipelines: Open Vocabulary, Weak 3D Grounding

More recent approaches leverage the world knowledge and reasoning capabilities of large language models (LLMs) and vision-language models (VLMs) to generate scenes from text prompts. LayoutGPT uses LLMs to plan object placements by generating bounding box coordinates in natural language. LayoutVLM uses differentiable optimization guided by VLM scores to refine layouts. Holodeck—one of the two primary baselines in this paper—combines an LLM for high-level planning (what objects to place) with retrieval from asset libraries and a fixed placement pipeline.

Why they fail for simulation-ready generation:

Static, unidirectional computation: These systems follow a fixed sequence: LLM plans → objects are retrieved/placed → output. There is no feedback loop, no mechanism for the system to look at what it just generated, identify that a chair is floating or that the room is sparse, and correct itself. The paper characterizes this as:

"these systems are static: their 'computational graph' is fixed, preventing adaptive reasoning and self-correction."
Sparse 3D grounding: LLMs reason about space through text, not through 3D coordinates. When an LLM says "place the chair near the table," it has no 3D understanding of what "near" means in metric space, what the table's dimensions are, or whether the chair will collide with the wall. This results in physically invalid placements: objects floating in mid-air, intersecting walls or other objects, or arranged in geometrically impossible configurations.
No physics validation: Critically, these pipelines do not run physics simulations. The Holodeck paper, for instance, generates beautifully arranged rooms that look correct in static renders—but they have never been tested under gravity. The SAGE paper demonstrates this concretely in Figure 5: when Holodeck scenes are loaded into Isaac Sim, objects shift, fall, or fly apart because their placements were physically unstable. The paper quantifies this: Holodeck achieves only 63.8% stability in bedrooms (meaning over a third of objects fall or shift when physics is applied), 73.8% in kitchens, and 66.5% in living rooms, with collision rates of 29.1%, 16.0%, and 20.8% respectively.
Limited or no object generation: Most LLM-based systems retrieve pre-existing 3D assets from a library rather than generating new objects. This means the diversity of scenes is bounded by the library size and coverage. If the library lacks "a rusty metal desk with peeling paint," the system cannot create one. In contrast, SAGE uses TRELLIS for text-to-3D generation, enabling open-vocabulary object creation that is not limited by an asset library.

Agentic Systems: A Step Forward, but Still Not Simulation-Ready

The most recent development in this space—and the closest antecedent to SAGE—is the emergence of agentic scene generation systems that use LLMs not just as planners but as agents that can reason about their outputs and take corrective actions. SceneWeaver, a concurrent work cited in the paper, represents the state of the art in this category. It is agent-based, supports self-reflection (the agent can assess its own output and decide to refine it), and handles collision avoidance during placement.

Why SceneWeaver is insufficient:

Despite being a significant advance over static pipelines, SceneWeaver still falls short of producing simulation-ready scenes. The paper identifies two specific gaps:

Missing physical attributes: SceneWeaver does not systematically attach physical properties (mass, friction coefficients, collision geometry) to objects. Without these, the scene cannot be meaningfully simulated—the physics engine has no information about how objects should behave.
No simulator-in-the-loop verification: SceneWeaver does not run a physics simulation during the generation process. While it checks for geometric collisions during placement, it does not test whether objects are stable under gravity. As Figure 5 demonstrates, SceneWeaver scenes exhibit objects that displace or fall when loaded into Isaac Sim. Quantitatively: 67.7% stability averaged across room types, with 32.8% collision rates—better than Holodeck but still far from the near-perfect stability required for reliable robot training.

The paper summarizes this gap succinctly in the introduction:

"there are still major gaps in (1) physical grounding for interaction and (2) compatibility with robot simulators"

SceneWeaver produces semantically plausible scenes, but not physically valid ones. For embodied AI, this distinction is critical. A robot trained in a scene where objects float or slide unpredictably will learn behaviors that are useless (or dangerous) in the real world.

The Broader Importance: Simulation-Driven Scaling for Embodied AI

The motivation for this work extends beyond fixing technical limitations in prior approaches. The paper frames its contribution within a larger vision of simulation-driven scaling for embodied AI—the idea that, just as web-scale data powered the scaling of language and vision models, simulation-generated data can power the scaling of robotic policies.

This is not a new vision (the paper cites a rich literature of simulation-based RL and imitation learning), but it has been bottlenecked by the difficulty of creating simulation environments. If creating a single realistic, diverse, physics-valid kitchen takes days or weeks of manual effort, the vision of training on thousands or millions of scenes remains impractical. SAGE's contribution—if it works—is to eliminate this bottleneck: making scene generation as easy as writing a text prompt, while automatically ensuring physical validity.

The paper also positions itself within a broader recognition that the web cannot solve embodied AI's data problem. Unlike language or images, embodied interaction data cannot be passively scraped from the internet. You cannot download a dataset of a robot successfully performing 10,000 different pick-and-place tasks in diverse kitchens—no such dataset exists. The environments themselves, and the robot interactions within them, must be generated.

The paper cites the growing investment in simulation platforms (Isaac Sim, Habitat, ManiSkill, RoboCasa, Genesis) and the emergence of simulation-to-reality transfer techniques as evidence that the field is converging on simulation as the primary engine for embodied AI data. The missing piece—and what SAGE aims to provide—is the scene generation engine that makes these simulators productive at scale.

How This Paper Positions Itself Relative to Existing Work

SAGE explicitly positions itself at the intersection of two previously separate research threads:

3D scene generation: The line of work from rule-based systems through data-driven methods to LLM-based pipelines, all of which focus on producing visually and semantically plausible scenes but have not prioritized physical validity or simulator compatibility.
Simulated environments for embodied AI: The ecosystem of physics engines, robot simulators, and benchmark suites that provide the infrastructure for robot training but have relied on manually-created or procedurally-retrieved scenes that limit diversity.

SAGE's claim is that it unifies these two threads by making scene generation simulator-aware. The key mechanisms that enable this unification are:

Agentic orchestration under MCP: Rather than a fixed pipeline, SAGE uses an LLM agent that dynamically decides which tool to invoke next based on the current state of the scene and the feedback it has received. This enables self-correction: if the scene is sparse, the agent can add objects; if objects are colliding, it can move or remove them; if placements are unstable, it can try alternative locations or smaller objects.
Complementary visual and physics critics: The visual critic (using multi-view VLM reasoning) addresses the semantic shortcomings of LLM-only systems—it actually looks at the generated scene and can identify missing objects, implausible arrangements, or aesthetic inconsistencies. The physics critic (with Isaac Sim in the loop) addresses the physical shortcomings—it actually tests whether the scene is stable and rejects placements that fail.
Text-to-3D generation with physical property estimation: Rather than relying on a fixed asset library, SAGE uses TRELLIS to generate novel 3D objects from text descriptions and a VLM to estimate their physical properties (mass, dimensions, material parameters). This enables open-vocabulary scene creation and unbounded diversity.

The paper's Table 1 makes this positioning explicit by comparing SAGE against a comprehensive list of prior methods across multiple axes: open-modality vocabulary, 3D grounding, self-improvement, fine-grained control, collision checking, physical attributes, and simulator validation. SAGE is the only method that checks every box. The table is not merely a taxonomy—it is an argument that prior work has been solving sub-problems in isolation, and what is needed is a system that integrates all of these capabilities into a single framework.

The paper's title—Scalable Agentic 3D Scene Generation for Embodied AI—encapsulates this positioning. It is not merely a scene generation paper (that would be "3D Scene Generation"), nor merely an embodied AI paper (that would be "Embodied AI Environments"). It is a paper about making scene generation scalable (through automation and augmentation), agentic (through adaptive tool use and self-correction), and directly useful for embodied AI (through simulation-validation and downstream policy learning). The combination of these three adjectives defines the gap the paper aims to fill.

3. Technical Approach

3.1 Reader Orientation

SAGE is an automated system that turns a text description of a desired robot task into a fully interactive, physically stable 3D simulation environment, ready for a robot to practice in. The core problem it solves is the bottleneck in embodied AI training data—you need thousands of diverse, realistic, physics-valid 3D scenes to train generalizable robot policies, but creating even one such scene manually takes hours or days. The shape of the solution is an LLM agent that adaptively calls specialized generator tools to build and refine a scene, guided by visual and physics critics that detect problems and suggest corrections, with the entire process looping until the scene is both visually coherent and physically stable.

3.2 Big-Picture Architecture (Diagram in Words)

The SAGE system has five major components connected in a closed feedback loop:

The Agent LLM — the central reasoning engine. It receives the user's task description, maintains the current state of the scene, decides which tool to call next, interprets feedback from critics, and declares when generation is complete. It operates under the Model Context Protocol (MCP), a standardized interface for tool interaction.
Generator Tools — a set of four executable operations exposed as MCP servers. The agent invokes them by sending structured requests:
- Scene Initializer: generates the floor plan, walls, and proposes an initial object list.
- Asset Placer: synthesizes 3D objects from text descriptions using TRELLIS, estimates their physical properties via a VLM, and places them in the scene using depth-first search with collision avoidance.
- Asset Mover: relocates an existing object to a new position.
- Asset Remover: deletes an object from the scene.
Visual Critic — a feedback component that renders the current scene from multiple viewpoints (top-down and four corner views), analyzes the images, and produces structured feedback: it identifies missing objects, suggests natural object combinations (e.g., "add chairs around the table"), flags empty areas that need filler objects, and verifies that user-specified task objects are present.
Physics Critic — a simulator-in-the-loop validator that loads the scene into Isaac Sim after each object operation and tests physical stability under gravity. It measures the pose change of each object after simulation, rejects placements that cause instability or collisions, and reports failures to the agent with suggestions (e.g., "try a smaller object" or "use a different support surface").
Scene Augmentation Module — a post-generation scaling component that takes a single generated scene and produces diverse variants by randomizing object configurations, synthesizing new object categories from augmented text descriptions, or regenerating the entire background layout while preserving task-relevant objects.

Information flows as follows: user prompt → agent analyzes requirements → agent calls Scene Initializer to get floor plan and object list → agent iteratively calls Asset Placer for groups of objects → Visual Critic renders and analyzes the scene → Visual Critic suggests additions/movements/removals → agent acts on suggestions → Physics Critic validates each placement in Isaac Sim → agent receives pass/fail feedback → agent continues iterating until the scene satisfies all criteria → declared complete → (optionally) augmentation module generates variants → action generation module creates robot demonstrations.

3.3 Roadmap for the Deep Dive

First, the Model Context Protocol (MCP), since it is the communication substrate that all components speak—the agent is the client, the generators and critics are servers, and understanding the request-response structure is prerequisite to everything else.
Second, the four generator tools in detail, because they physically construct the scene: how floor plans are created, how objects are synthesized and placed, and how placement constraints are enforced.
Third, the visual critic, since it provides the semantic feedback that drives iterative refinement—what images it sees, how it reasons about them, and what kind of suggestions it makes.
Fourth, the physics critic, because it is the novel component that makes scenes simulation-ready—how stability is tested, what constitutes a failure, and how failures are communicated back to the agent.
Fifth, the full iterative self-improvement loop, showing how generators and critics interact over multiple rounds to converge on a valid scene.
Sixth, the scene augmentation strategies, which take a single scene and scale it into hundreds of diverse variants for policy training.
Seventh, the action generation and policy learning pipeline, showing the end-to-end flow from scene to robot demonstration to trained policy.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building paper whose core idea is that adaptive, critic-guided, simulator-validated scene generation can produce environments that are simultaneously diverse, realistic, and directly deployable for robot policy learning.

The Model Context Protocol (MCP) as Integration Substrate

SAGE is not a monolithic program with a fixed execution order. Instead, it operates under the Model Context Protocol (MCP), a standardized protocol for how applications provide context to LLMs and how LLMs call external tools. In the SAGE architecture, the agent LLM acts as the MCP client—it initiates requests—while each generator tool and critic is hosted behind an MCP server—it responds to requests. The agent and tools communicate through structured messages that include a tool name, input arguments, and a JSON-formatted return value.

Why MCP rather than a fixed pipeline? The fundamental limitation of prior systems like Holodeck is that their "computational graph" is static. The system always does A, then B, then C, regardless of whether B produced a good result. If B places a table that blocks a doorway, there is no mechanism to notice and call a different tool to fix the problem. MCP allows the agent to dynamically decide which tool to call next based on the current scene state and the feedback it has received. The sequence of tool calls is not predetermined—it emerges from the agent's reasoning about what the scene needs.

Tool discovery and invocation. Each tool (generator or critic) registers with the MCP system by providing a description of its function, its input argument types (specified as Python string types), and its output format. The agent uses these descriptions to understand what capabilities are available and how to invoke them. At each iteration, the agent either specifies the next tool to call with its input arguments (as a structured request), or declares that scene generation is complete. The MCP server for that tool executes the requested operation and returns the result as a JSON dictionary string, which the agent can parse and interpret. This setup is described in Appendix B.1.1:

"For the agent to understand each tool, we provide descriptions that include the tool's function, input argument types (specified as Python strings), and output format. We formulate all tool outputs as JSON dictionary strings, which the agent can easily parse and interpret."

Iteration structure. The agent operates in a loop: it receives the current scene state and any feedback from the previous step, reasons about what is needed, selects and calls a tool, receives the tool's output, and decides whether to continue or stop. The stopping condition is explicitly declared by the agent: when it determines that the scene meets the visual and physical quality criteria, it signals completion. There is no fixed number of iterations—complex scenes with many objects and difficult placement constraints may require more rounds of generation and refinement than simple scenes.

Generator Tools: The Four Scene Construction Operations

The scene is physically constructed through a set of four generator tools, each exposed as an MCP server that the agent can invoke. These tools are the only way the agent can modify the scene state—there is no direct manipulation of the scene representation outside of these four operations. This constraint is important because it means all scene modifications are mediated through the physics critic (which validates placements) and are logged in a structured format that the visual critic can analyze.

Scene Initializer

Purpose and responsibilities. The Scene Initializer takes the scene specification (derived from the user's prompt) as input and is responsible for two outputs: (1) an empty 3D room with only floor and walls, and (2) a list of proposed objects, each with a text description, estimated physical attributes, and placement constraints.

Floor plan generation. The process begins with floor plan generation, which proceeds in two steps. First, the LLM is prompted with the scene description and returns the room types and sizes. This is the agent's first use of language model reasoning—it translates the user's high-level intent ("a student apartment with one bedroom" or "a rustic kitchen") into concrete spatial specifications (a room of type "kitchen" with dimensions 4m × 5m, a connected room of type "living room" with dimensions 5m × 6m, etc.). For single-room scenes, this produces a single room specification. For multi-room layouts, a second step generates connectivity between rooms—which rooms are adjacent, where doors connect them—and places connecting doors in the shared walls. This is illustrated in Figure 7 of the paper, which shows multi-room examples like "Multilingual teacher's apartment" and "Mid-century modern family home."

Wall and floor material generation. Once the room geometry is established, the system generates textures for the floor and walls. The LLM provides a text description of the desired materials (e.g., "hardwood floor," "white painted walls"), and these text descriptions are passed to MatFuse, a diffusion-based material generation model that synthesizes physically-based rendering (PBR) material maps (diffuse, metallic, roughness, normal) from text prompts. The paper notes in Appendix B.1.2 that for supplementary material visuals, they also use Flux.1-dev and its kontext variant for more detailed texture generation, but MatFuse is the primary material generator for the core pipeline.

Object list generation. In addition to the room geometry, the Scene Initializer outputs a list of proposed objects that should populate the scene. Each object in this list has a text description that will later be used for text-to-3D generation. The object list is generated to reflect the user's task demands and the room type's semantic expectations. Critically, if the user specifies a robot task (e.g., "pick up a bowl and place it on the table"), the generated object list will explicitly include those task-relevant objects (bowl, table) so that the scene contains the necessary elements for the downstream embodied task. The paper states:

"For example, if the user asks about learning a task of 'pick an apple and place it to a bowl', the generator will include the required apple and bowl in the object list to be returned to the agent."

Physical attribute estimation. For each proposed object, the LLM estimates approximate physical attributes including the object's dimensions, an initial estimate of mass, and PBR material parameters (metallic and roughness values). These estimates serve as starting points that will be refined later by the VLM during actual object placement.

Output structure. The output of the Scene Initializer is returned to the agent as a structured JSON containing: the room geometry (type, dimensions, wall positions, floor plane), references to the generated material maps, the list of proposed objects with their text descriptions and initial physical estimates, and placement constraints for each object (e.g., "this object should be placed near the wall," "this object must be on top of the desk").

Asset Placer

Purpose and responsibilities. The Asset Placer is the workhorse generator—it takes a text string describing the placement requirements for one or more objects and is responsible for three tasks: (1) generating the 3D mesh for each object from its text description, (2) estimating accurate physical properties for the generated mesh, and (3) finding a valid placement in the scene that satisfies semantic constraints, avoids collisions, and is physically stable.

Text-to-3D object generation. Objects are generated using TRELLIS, a text-to-3D diffusion model that produces a 3D mesh from a text prompt. This is a critical architectural choice: rather than retrieving pre-existing assets from a library (as Holodeck and most LLM-based systems do), SAGE synthesizes novel objects on demand. This enables open-vocabulary scene generation—the system can create a "cyberpunk computer chair with neon accents" or a "rustic wooden bowl with a cracked finish" even if no such object exists in any asset library. TRELLIS generates unit-sized meshes; the actual scale is applied later based on the VLM's height estimate.

Mesh post-processing. The raw meshes from TRELLIS undergo a series of post-processing operations to ensure they are suitable for physics simulation: decimation (reducing polygon count for simulation efficiency), non-manifold correction (fixing geometric defects where edges are shared by more than two faces), and hole filling (closing gaps in the mesh surface). These operations ensure the "watertight" property—the mesh is a closed, manifold surface that a physics engine can reliably use for collision detection. Without these corrections, the physics engine might incorrectly compute penetration depths or miss collisions entirely.

Physical property estimation via VLM. After mesh generation, a Vision-Language Model (VLM) is queried to estimate the object's physical properties. Specifically, the VLM takes the text description and (presumably) a rendering of the generated mesh and outputs:

Height: the actual physical height of the object in meters, used to rescale the unit-sized TRELLIS output to real-world dimensions.
Mass: the object's mass in kilograms, used by the physics engine for gravity, inertia, and collision response calculations.
Metallic and roughness values: PBR material parameters for rendering.

These properties are critical for simulation-readiness. Without mass estimates, the physics engine cannot compute realistic gravitational forces or momentum transfer during collisions. Without metallic/roughness, the renderer cannot produce realistic lighting.

Placement constraint analysis. For each object to be placed, an LLM analyzes the input placement condition and classifies it into one of three categories:

Floor: the object rests on the ground plane (e.g., tables, chairs, floor lamps, rugs).
Wall: the object is mounted on a wall surface (e.g., paintings, shelves, wall lights).
On-top: the object rests on top of another object (e.g., a book on a desk, a bowl on a table, a lamp on a nightstand).

This classification determines the placement strategy used in the next step.

Floor object placement. For objects classified as "floor," the placement algorithm first uses the LLM to generate placement constraints expressed in natural language, which are then translated into geometric constraints. These include: a global position preference (e.g., "near the center of the room," "in the corner"), relative position to existing objects (e.g., "2 meters from the sofa," "facing the television"), and relative orientation (e.g., "facing toward the center of the room"). The system then samples candidate positions using a grid-based approach over the floor area, scores each candidate against the constraints, and applies depth-first search with collision checking to find the highest-scoring position and orientation that is collision-free. Collision checking uses the mesh geometries of all existing objects.

Wall object placement. For wall-mounted objects, a similar grid-based sampling is performed along the wall surfaces (sampling positions at regular intervals along each wall segment). Depth-first search is used to find valid positions that avoid collisions with both floor objects (which might block wall placement) and other wall objects. Wall objects are treated as static (immovable) during later physics simulation since they are attached to walls.

On-top placement with multi-layer support. For objects that must rest on top of other objects, the placement algorithm samples candidate locations by computing surface normals on the supporting object's mesh—specifically, selecting faces whose normal vectors are approximately aligned with the room's upward axis (within some angular tolerance). This identifies horizontal surfaces that could physically support another object. The paper highlights a key advantage over Holodeck:

"unlike [Holodeck] which supports only single-layer relationships, our method enables multi-layer scene graphs."

This means SAGE can place a book on a shelf, which rests on a bookshelf, which rests on the floor—the system handles nested supporting relationships. After identifying candidate support surfaces, the placer samples positions on those surfaces, checks for collisions with the supporting object and nearby objects, and produces a set of candidate placements.

Physics validation during placement. Each candidate placement is individually validated using Isaac Sim, the physics simulator. The scene with the proposed placement is loaded into the simulator, and physics is applied. The system measures the pose change of each object after a short simulation period (a few time steps, typically 120 steps as specified in the metrics section):

Stability criterion: An object is considered unstable if its relative translation exceeds 0.2 meters or its rotation exceeds 8 degrees after simulation. These thresholds are specified in Section 4.1.1 under Metrics.
Collision detection: The trimesh library is used to detect mesh interpenetrations between objects. If the candidate placement causes collisions, it is rejected.

If a placement candidate is unstable—for example, a pillow placed upright on a bed that falls over when physics is applied—the system records the post-simulation pose (where the pillow ended up after falling) and re-simulates with this adjusted pose. If the second simulation is stable (the pillow now rests flat on the bed in its fallen position), the placement is accepted. If the object falls through the supporting surface, flies away, or otherwise never reaches stability, the placement is rejected. This two-step validation is a pragmatic engineering choice: it allows the system to "discover" the natural resting pose of an object without requiring precise initial placement predictions.

Early-stopping optimization. To control computational cost, the placer adopts an early-stopping strategy: it evaluates candidate placements in order of their constraint-satisfaction score and accepts the first candidate that passes both collision checking and physics validation. The paper notes in Appendix A.3:

"This greatly reduces computation time, as we typically find stable placements within a few trials despite having 30-50 candidate locations."

This is efficient because physics simulation is the most expensive step (1-2 seconds per candidate according to runtime analysis), and most placements that pass collision checking are also stable for simple floor and wall objects. On-top placements are more frequently rejected, but the search typically finds a valid placement quickly.

Iterative placement across tool calls. A crucial aspect of the agentic design is that the Asset Placer is not called once for all objects in the scene. Instead, the agent calls it multiple times throughout the generation process, each time with a different group of objects to place. This allows the agent to: (1) place critical objects first and then add complementary objects around them, (2) respond to visual critic feedback by placing additional objects that were missing, and (3) handle placement failures by adjusting the placement request (e.g., asking for a smaller object or a different location). The paper emphasizes this contrast with Holodeck:

"unlike [Holodeck], which supports only single-iteration placement, our agentic framework enables iterative, adaptive placements across multiple tool calls to fully realize the scene and satisfy user requirements."

How the agent communicates placement requirements. The agent sends a text string to the Asset Placer that describes what to place and any constraints. The LLM within the placer parses this string into structured placement instructions. The text can reference multiple objects, specify relationships between them, and include location preferences. The exact format of this communication is not detailed in the paper but is embedded in the MCP tool's input argument specification.

Asset Mover

Purpose and operation. The Asset Mover locates an existing object in the scene (specified by its text description in the input string) and repositions it to a new location. Its operation has three steps:

Object localization: An integrated LLM parses the input text to identify which object to move. The system locates the object in the current scene representation by matching the text description to the metadata of placed objects (each object has a name, description, and other attributes from the generation process).
Temporary removal: The identified object is removed from the scene. This is necessary because the placement algorithm requires a clean state to evaluate candidate positions without the original placement interfering with collision checking.
Replacement: The same placement logic used by the Asset Placer is applied to find a new valid position for the object. If the placement succeeds (a collision-free, stable position is found), the object is placed at the new location. If placement fails (because no suitable position exists, for example due to insufficient space), the object is restored to its original location and the failure is reported back to the agent.

Source of movement instructions. Movement instructions typically come from the visual critic. For example, the critic might observe in a rendered view that "the chair is blocking the doorway" or "the lamp is too close to the edge of the table and looks unstable." The agent, upon receiving this feedback, decides to invoke the Asset Mover with the appropriate parameters.

Why a separate tool instead of remove-then-place? The Asset Mover could theoretically be implemented as a composition of Asset Remover and Asset Placer calls. However, having it as a distinct tool simplifies the agent's reasoning—the agent expresses the intent "move this chair" rather than "remove the chair, then place a chair near the table." The Mover also handles the failure case (restore original position) atomically, which would be error-prone if implemented as separate tool calls where the agent would need to remember to restore the object on failure.

Asset Remover

Purpose and operation. The Asset Remover deletes an object from the scene. It has a simple interface: it takes a text description of the object to remove, uses LLM reasoning to locate the matching object in the scene representation, and removes it. The paper describes it tersely:

"It is usually called when the critics (described later) gives feedback to remove an object."

Typical usage scenarios. The Remover is invoked in several situations: (1) the visual critic determines that an object is semantically inconsistent with the scene (e.g., a beach ball in a formal dining room), (2) the physics critic reports repeated placement failures for a large object and suggests removing a conflicting object to free up space, (3) the agent decides that an object placed earlier in the generation process no longer fits the evolving scene composition, or (4) the scene is over-cluttered and needs simplification.

The Visual Critic: Semantic and Spatial Coherence Feedback

Purpose and operation. The Visual Critic is the component that gives the agent "eyes"—it actually looks at the scene and provides structured feedback about what is wrong or missing. This addresses the fundamental weakness of LLM-only pipeline systems: an LLM can plan what objects should be in a room, but it cannot see whether those objects are correctly placed, whether the room looks sparse or cluttered, or whether aesthetic choices have produced a coherent visual result.

Input data. The critic takes as input the complete current scene configuration, specifically: (1) the list of all placed objects with their positions, orientations, and text descriptions, and (2) multi-view renderings of the current scene state. The renderings include a top-down orthographic view (showing the spatial layout from above) and four perspective corner views (showing how the room looks from different angles). These images are rendered using the same PBR materials and lighting that will be used in the final simulation, so the critic sees a photorealistic representation of the scene. The multi-view approach ensures that the critic can assess the scene from all angles—an object that looks well-placed from one viewpoint might appear to be floating or clipping from another.

Reasoning process. The critic processes these images using a VLM (specifically Qwen3-VL-30B-A3B-Instruct). The VLM is not just describing the scene—it is tasked with specific analytical judgments:

Identifying natural object combinations: The critic looks for semantic relationships that should exist but are missing. For example, if there is a table with no chairs around it, the critic notes that chairs should be added. The paper mentions that example combinations are provided to guide the critic's reasoning:

"it identifies natural object combinations—for example, placing chairs around tables or books on bookshelves—using example combinations we provide to guide its reasoning."
Detecting sparseness: When the room appears empty or under-furnished, the critic suggests background filler objects such as floor plants, decorative items, rugs, or wall art that belong in the room type. This prevents the common failure mode where an LLM-generated object list is too minimal (e.g., a living room with only a sofa and a TV, no coffee table, no side tables, no decorations).
Verifying task-relevant objects: The critic checks whether user-specified task objects have been placed and notifies the agent if any are missing. This ensures that the downstream embodied task is actually executable—if the user asked for a scene to practice "pick up a mug and place it in a bowl," and the mug was never placed, the critic catches this.
Proposing adjustments to existing placements: By analyzing the multi-view renderings, the critic can identify objects that are poorly positioned—too close to walls, blocking pathways, creating visual imbalances, or placed in physically improbable configurations.

Output format. The critic produces structured feedback: a list of suggested additions (with text descriptions of new objects to place), a list of suggested movements (which objects to relocate and where), and a list of suggested removals (which objects are inappropriate or cluttering the scene). This feedback is returned to the agent as a JSON string.

How feedback drives tool selection. The agent receives the critic's feedback and decides which generator to invoke next. If the critic says "add chairs around the dining table," the agent calls the Asset Placer with appropriate placement instructions. If the critic says "the floor lamp is too close to the sofa and looks cramped," the agent calls the Asset Mover. If the critic says "the garbage can doesn't belong in a bedroom," the agent calls the Asset Remover. There is no hard-coded mapping from feedback to action—the agent reasons about the most appropriate response.

Why a VLM-based visual critic? An alternative approach would be purely rule-based checks (e.g., "if there is a table with seating surface, ensure there are chairs within X meters"). But such rules quickly become combinatorially complex and cannot capture the nuanced, context-dependent judgments that a VLM can make. A VLM can recognize that "this room feels sparse and needs additional furniture" or that "the color scheme is inconsistent" in ways that would be impossible to encode as explicit rules. The trade-off is reliability—VLM judgments can be noisy or inconsistent—but the iterative nature of the SAGE loop (the critic can be called multiple times as the scene evolves) mitigates this by allowing the agent to converge on a scene that satisfies the critic across multiple rounds.

The Physics Critic: Simulator-in-the-Loop Stability Validation

Purpose and operation. The Physics Critic is the component that makes SAGE scenes genuinely "simulation-ready"—it validates that the generated scene is physically stable by actually running a physics simulation and checking whether objects maintain their positions. This is the component that distinguishes SAGE from all prior scene generation systems, which either did not check physics at all (Holodeck) or checked only geometric collision without simulating dynamics (SceneWeaver).

When validation occurs. The physics critic operates during every stage of every generator, not as a post-hoc check. Specifically: after each object addition (Asset Placer places a new object), after each object movement (Asset Mover relocates an object), and after each object removal (Asset Remover deletes an object). The paper states in Section 3.1.2:

"After each object addition, movement, or removal, the scene is loaded into Isaac Sim to test its physical stability."

Simulation setup. The scene is loaded into Isaac Sim, NVIDIA's GPU-accelerated robotics simulator built on PhysX. Key configuration details:

Wall objects are treated as static: Any object placed against a wall (paintings, shelves, mounted lights) has its rigid body set to static (immovable) during simulation. This is a physically correct approximation—wall-mounted objects are fixed in the real world—and it prevents the physics engine from incorrectly computing gravitational forces on them.
All other objects are dynamic: Floor objects and on-top objects respond to gravity, collisions, and contact forces. The physics engine computes their trajectories based on their estimated masses, the collision geometries from their meshes, and the material properties (which affect friction and restitution).
Simulation duration: The simulation runs for 120 steps, as specified in the metrics definition. This is long enough for objects to settle under gravity—objects that start in unstable configurations (e.g., a tall vase balanced on a small base) will tip over or slide off surfaces within this time window.
Stability criterion: An object is considered unstable if its relative translation (the distance between its initial and final position, relative to its own coordinate frame) exceeds 0.2 meters or its relative rotation (the angular difference between its initial and final orientation) exceeds 8 degrees after the 120 simulation steps.

Validation at the placement level. The physics critic operates at the level of individual placements during the Asset Placer's operation. When the placer identifies candidate positions, each candidate is individually validated. This is described in detail in Appendix B.1.2:

"We simulate each candidate placement—if unstable (e.g., a pillow standing on a bed), we record the post-simulation pose and re-simulate with this adjusted pose. If the second simulation is stable, we accept the placement; otherwise, we reject it."

This two-pass approach for unstable placements is a clever engineering hack: rather than trying to predict the final resting configuration of an object that starts in an unstable state, the system lets physics determine the resting state, records it, and then verifies that the resting state is stable. For the pillow example: the initial placement has the pillow standing upright on its edge (unstable); the first simulation causes it to fall flat on the bed; the system records the fallen pose and re-simulates; the pillow now rests stably in its new pose; placement is accepted with the adjusted pose.

What happens when all candidates fail. If the Asset Placer evaluates all candidate placements and none pass both collision and stability checks, the physics critic reports the failure to the agent. The failure message includes suggestions for alternative actions:

"If the generator fails to find a stable configuration, the critic reports the failure to the agent, suggesting alternative actions such as using smaller objects or adjusting placement target locations."

The agent can then respond by: calling the Asset Placer again with a request for a smaller version of the object, calling the Asset Remover to clear space by deleting a conflicting object, calling the Asset Mover to relocate an existing object that is blocking viable placement positions, or deciding that the object is optional and proceeding without it.

Batch simulation optimization. In production, validating each placement individually would be extremely expensive. SAGE employs a two-tier strategy to reduce simulation overhead:

Floor and wall objects: These are batched—the system places all floor and wall objects first, then simulates once with all of them present. If any object is unstable after the batch simulation, it is removed. The paper states:

"we simulate once after placing all floor and wall objects, then remove any unstable objects. This significantly improves efficiency by consolidating multiple simulations into one."
On-top objects: These are simulated individually during placement because they are typically smaller, more prone to instability, and their stability depends on the specific supporting surface and neighborhood of nearby objects. The paper notes:

"For on-top objects, however, we simulate each placement individually since these objects are typically smaller and more prone to instability."

Runtime cost. Each simulation validation takes 1-2 seconds per placement candidate according to Appendix A.3. With the early-stopping strategy (accepting the first stable, collision-free candidate) and typically finding valid placements within a few trials, the simulation overhead per object is manageable—on the order of a few seconds per object.

Comparison with baselines. The quantitative impact of the physics critic is dramatic. In Table 2, SAGE achieves 99.9% average stability across three room types, compared to 63.8% for Holodeck and 67.7% for SceneWeaver. Collision rates are 1.9% for SAGE versus 22.0% and 32.8% respectively. The ablation study in Table 3 further isolates the effect: with only the visual critic and no physics critic, stability drops to 80.3%; with the physics critic alone, stability jumps to 99.6% but visual quality metrics suffer. The full system (both critics) achieves the best of both worlds.

The Iterative Self-Improvement Loop

The generators and critics are not independent modules called in a fixed sequence. They interact through a closed-loop process where the agent continuously evaluates the scene state and decides the next action. Understanding this loop is essential to understanding why SAGE works where static pipelines fail.

Initialization phase. The agent begins by calling the Scene Initializer with the interpreted user requirements. This produces the floor plan and an initial object list. The object list is a proposal, not a commitment—the agent may add, remove, or modify objects as generation proceeds based on critic feedback.

Construction phase. The agent iteratively calls the Asset Placer for groups of objects. The grouping is determined by the agent: it might place all large furniture first (sofa, table, bed), then add smaller complementary objects (lamps, decorations), then fill gaps with background objects (plants, rugs). The agent's choice of ordering matters because later placements can reference earlier ones as constraints (e.g., "place chairs facing the sofa that was placed in round 1").

Critique phase. After each round of placements (or after significant scene changes), the agent may call the Visual Critic to get an assessment. The critic's feedback is specific and actionable ("the coffee table is missing," "the painting is too high on the wall"). The agent uses this feedback to plan the next round of tool calls.

Refinement phase. Based on critic feedback, the agent invokes the appropriate generators: Asset Placer for missing objects, Asset Mover for poorly positioned objects, Asset Remover for inappropriate objects. After each operation, the Physics Critic automatically validates stability—the agent does not need to explicitly call it; it is embedded in the generator tools' execution.

Termination phase. The loop continues until the agent determines that the scene meets all requirements. The stopping condition is not based on a metric threshold—it is a high-level judgment by the agent that the scene is "complete": all user-specified task objects are present, the visual critic no longer reports significant issues, and all physics validations have passed. The agent explicitly signals completion to the MCP framework.

Why the loop is necessary. Consider a failure scenario that would break a static pipeline: The Asset Placer is asked to place a large dining table in the center of a kitchen. The placement succeeds (stable, collision-free). The Visual Critic is then called and notes that the room looks sparse—chairs are missing. The agent calls Asset Placer for chairs around the table. But the first chair placement candidate collides with a cabinet that was fine before. The placer tries alternative positions and finds one that works. Now the visual critic notes that the room is still missing wall decorations. The agent calls Asset Placer for wall art. One painting placement is rejected by the physics critic because the wall surface at that position is slightly uneven. The placer tries an adjacent position and succeeds. At each step, failures are caught and corrected immediately, preventing error accumulation. In a static pipeline, the table's placement might have blocked chair placement entirely, resulting in a scene with a table but no chairs—or worse, chairs placed inside the table geometry.

Error modes the loop cannot fix. It is important to note what the loop does not address. The loop can correct placement errors and semantic omissions, but it cannot fix fundamentally poor object generation. If TRELLIS produces a distorted mesh, the VLM estimates impossible physical properties, or the LLM plans an illogical room layout, the iterative loop has no mechanism to regenerate from scratch with better parameters. The quality ceiling is set by the individual components, not by the orchestration.

Scene Augmentation: Scaling One Scene into Many

Generating a single scene is not sufficient for training generalizable robot policies—the policy needs to see many variations of the same task to avoid overfitting. SAGE provides three complementary augmentation strategies that systematically expand a single base scene into diverse variants while preserving the task semantics that the policy needs to learn.

Object Configuration-Level Augmentation

Purpose. This is the simplest form of augmentation: take the existing scene and randomize the poses of task-relevant objects within reasonable bounds, creating spatial variation while keeping the scene composition identical.

Mechanism. For each task-relevant object (e.g., the target object to be picked up, the destination object to be placed on), its pose is resampled within the same scene. The new pose must still satisfy the semantic constraints (a mug must remain on a surface, not in midair) and pass physics validation. The paper describes it tersely:

"The pose of each task-relevant object (e.g., the target to be picked, placed on, or navigated toward) is resampled within the current scene to create variations in object placement."

What stays constant. The room geometry, the identity and positions of all non-task objects, and the task definition remain unchanged. Only the positions of the objects directly involved in the task vary.

Why this matters for policy learning. A policy trained only on a single mug-at-the-center-of-the-table configuration will likely fail when the mug is at the edge of the table or on the left side instead of the right. Configuration-level augmentation exposes the policy to this spatial variation, forcing it to learn to locate the object visually rather than relying on a memorized position.

Object Category-Level Augmentation

Purpose. This form of augmentation changes the appearance and geometry of task-relevant objects while keeping their functional category the same. The robot always needs to grasp "a mug," but across training episodes, the mug can be tall, short, wide, ceramic, metallic, blue, red, etc.

Two-step mechanism. The augmentation proceeds in two phases:

Text augmentation via LLM: Given the original text description of a task-relevant object (e.g., "a ceramic coffee mug"), an LLM generates variations that change the object's geometry and texture while maintaining the object category. The paper specifies:

"we employ an LLM-based text augmentation to produce variations in geometry and texture (e.g., shape, color, material, or finish) while maintaining the original object category."

Example outputs might include: "a tall stainless steel travel mug," "a wide ceramic mug with a blue glaze," "a small espresso cup with a saucer."
Text-to-3D generation: Each augmented text description is passed to TRELLIS, which synthesizes a new 3D asset. The new asset replaces the original object in the scene, using the same placement constraints (supporting surface, approximate location) but with the new geometry.

Physics re-validation. After each replacement, the physics critic validates the new placement, since the new object may have different dimensions or mass that affect stability.

What stays constant. The room layout, non-task objects, and the task semantics (the robot is always grasping a mug and placing it in a bowl) remain unchanged. Only the visual and physical properties of the task objects vary.

Why this matters for policy learning. This is the most powerful form of augmentation for sim-to-real transfer. If the policy has only seen one specific mug during training, it will likely fail when presented with a visually different mug at test time because the visual features it learned to associate with "graspable object" are too narrow. Category-level augmentation forces the policy to learn category-invariant features: a mug is graspable regardless of its color or exact shape, and the policy must generalize from the training distribution to novel instances.

Figure 1 in the Appendix shows examples of category-level augmentation across four different base scenes, with three augmented variants per scene. The visual diversity is substantial—objects change color, shape, material, and scale—while the scene structure remains semantically identical.

Physical property re-estimation. When a new object is generated with a different shape and size, its physical properties change. The VLM re-estimates mass and dimensions for each augmented variant, and these updated properties are used for physics simulation. This ensures that the policy learns to handle objects with different inertial properties—a heavy ceramic mug requires different grasping force than a lightweight plastic one.

Scene Layout-Level Augmentation

Purpose. While the previous two augmentations modify task-relevant objects within a fixed background, layout-level augmentation changes the entire background environment—the room geometry, the floor plan, and all task-irrelevant objects—while keeping the task-relevant objects and their relationships intact.

Mechanism. The paper describes the process as regenerating the scene from scratch, but with a crucial modification to the generation pipeline:

"the background scene, including room geometry and all task-irrelevant objects, is regenerated through the agent-driven scene generation, while previously generated task-relevant objects are preserved and reused."

To implement this, a special stage is added to the Asset Placer that excludes pre-generated task-relevant objects from the generation list using LLM-based reasoning:

"we add a stage in the Asset Placer generator that excludes pre-generated objects from the generation list using LLM-based reasoning, preventing them from being regenerated."

In other words: the agent generates a new floor plan, new walls, new non-task furniture, new decorations, and new layout, but reuses the existing task-relevant objects. The task objects are repositioned according to the new scene layout (placed on surfaces in the new room, subject to the same semantic constraints), but their identities and geometries are preserved.

What stays constant across variants. The task-relevant objects (their descriptions, 3D meshes, physical properties) and the task specification ("pick up the mug and place it in the bowl") remain constant. Everything else—room type, dimensions, furniture, decorations, wall textures—changes.

Why this matters for policy learning. This augmentation teaches the policy to perform the task regardless of the surrounding environment. A robot trained only in a modern kitchen with white cabinets may fail when deployed in a rustic kitchen with wooden countertops because the visual context is entirely different. Layout-level augmentation ensures the policy sees the same task across diverse backgrounds and learns to ignore irrelevant visual features.

Figure 2 in the Appendix shows examples across four scene types (Bedroom, Living Room, Office, Meeting Room), with the base scene and three layout variants each. The constant task-relevant objects (nightstand, coffee table, desk, meeting table) appear in all variants, while everything else changes dramatically.

Composability. The paper explicitly notes that augmentations can be composed:

"scenes from layout-level augmentation can undergo further augmentation. For example, we can combine them with another round of category-level augmentation to create even greater diversity."

This means the final training dataset can contain scenes that vary along all three axes simultaneously: different room layouts, different object configurations within each layout, and different object instances within each configuration. The total diversity grows multiplicatively.

Simulation-Ready Validation During Augmentation

All three augmentation strategies are subject to the same physics validation as the base scene generation:

"After each augmentation step, we call the physics critic to ensure the stability and physical plausibility of all placements. Object-level physical properties such as mass and PBR parameters are estimated by a VLM as before."

This means that augmented scenes are not just visual variations—they are fully simulation-validated environments that can be immediately used for policy training without manual cleanup.

Action Generation: From Scenes to Robot Demonstrations

Once the scenes are generated and augmented, the next step in the pipeline is to produce robot action data—trajectories of robot joint positions, end-effector poses, and gripper states—that can be used to train an imitation learning policy. SAGE uses motion planning techniques to automatically generate these demonstrations without human teleoperation.

Pick-and-Place Action Generation

Task specification. The task is standard: pick up one object (e.g., a mug) and place it into or onto another object (e.g., a bowl). The robot is a Franka Emika Panda arm with a parallel-jaw gripper.

Grasp pose generation. The system uses M2T2 (Multi-Task Masked Transformer), a neural network that predicts grasp pose candidates from depth images. The process:

A depth image is rendered from a camera view (the wrist camera or an external camera).
M2T2 takes the depth image as input and outputs a set of candidate grasp poses—6-DOF positions and orientations of the gripper that are predicted to result in successful grasps.
These grasp poses are in camera coordinates. They are transformed to world coordinates using the known camera extrinsic parameters to obtain the actual 3D grasp pose.

Motion planning pipeline. Given a grasp pose, the robot's trajectory is decomposed into segments, each solved using inverse kinematics (IK) with collision avoidance:

Approach: The end-effector moves from its starting position to a point directly above the grasp pose. This is an IK problem: find a collision-free joint configuration that places the gripper at a specific position and orientation.
Grasp: The gripper lowers to the grasp pose, closes its fingers, and grasps the object.
Lift: The gripper lifts the object straight up to clear the supporting surface.
Transport: The gripper moves to a point above the placement target.
Place: The gripper lowers to place the object. The paper notes that placement is simplified to a "drop"—the gripper opens above the target location. The paper acknowledges this could be improved:

"We simplify placement to a drop, which could be improved with additional motion planning steps to gently place the object down."

Collision avoidance and IK solver. The system uses Curobo, a GPU-parallelized collision-free motion planning library:

"Curobo is integrated into the motion planning and IK pipeline by incorporating mesh geometries into its collision checking, ensuring feasible and stable grasp execution."

Curobo computes IK solutions while checking for collisions between the robot arm and the scene geometry (table, objects, walls). This ensures the generated trajectories are physically feasible—the robot does not pass through solid objects.

Data format. Each demonstration consists of a sequence of end-effector poses (position and orientation quaternion) and gripper open/close commands at each timestep.

Mobile Manipulation Action Generation

Task specification. This is a composite task: the robot (a Franka Emika Panda arm mounted on an Omron LD-60 mobile base) must navigate to a location, pick up an object, navigate to another location, and place the object. The decomposition into subtasks is:

Navigate from spawn location to picking location (near a table with a target object).
Pick up the object using the arm.
Navigate from picking location to placing location (near a different table/desk).
Place the object on the target surface.

Navigation motion planning. For navigation (subtasks 1 and 3), the system uses RRT (Rapidly-exploring Random Trees):

"we adopt RRT for robot path planning, generating collision-free trajectories between designated start and target positions."

RRT is a sampling-based motion planning algorithm that incrementally builds a tree of collision-free configurations by randomly sampling points in the configuration space and connecting them to the nearest node in the tree. The algorithm grows from both the start and target positions (bidirectional RRT) and terminates when the two trees connect.

Collision checking for navigation. For computational efficiency, collision checking during navigation uses a 2D occupancy grid rather than 3D mesh collision checking:

"Collision checking is implemented using a 2D occupancy grid, which is faster and saves memory compared to 3D explicit mesh collision checking."

The mobile base operates on the ground plane, so 2D collision checking is sufficient—the grid encodes which floor cells are occupied by furniture or walls. This is faster than performing full 3D mesh-mesh collision queries at every RRT expansion step.

Arm manipulation during mobile manipulation. The pick-and-place segments (subtasks 2 and 4) use the same M2T2 + Curobo pipeline as the standalone pick-and-place task, but with the added complexity that the arm's base is now mobile. The arm's IK calculations must account for the base's position at the time of grasping.

Policy decomposition. Unlike the standalone pick-and-place task where a single policy is trained end-to-end, the long-horizon mobile manipulation task is decomposed into four sequential policies, as described in Section 4.2.1:

"the long-horizon Mobile Manipulation is decomposed into four sequential policies. Each policy is responsible for one stage of the task and also predicts a termination signal for stage transition."

Each policy handles one stage (navigate-to-pick, pick, navigate-to-place, place) and outputs a termination signal when its stage is complete. The termination signal triggers the switch to the next policy. This decomposition simplifies the learning problem: instead of learning a single policy for a complex, long-horizon task, the system learns four simpler policies that are composed sequentially.

Failure Filtering

Motion planning is not always successful. The paper identifies specific failure modes:

"Motion planning is not always successful, often due to inaccurate grasp pose predictions, unreachable target configurations, or unexpected collisions during trajectory execution."

Failed demonstrations are filtered out:

"We filter out failed examples by performing collision checks and verifying whether each manipulated object reaches its expected location."

The filtering checks: (1) whether every planned trajectory segment is collision-free (did the robot arm pass through solid objects during execution?), and (2) whether the task outcome was achieved (did the grasped object end up at the target location? did the robot navigate to the correct positions?). Only successful demonstrations are retained for policy training.

Parallelization

To generate demonstrations at scale, the system leverages GPU parallelism:

"Parallelized action generation is performed with 8 environments for Pick-and-Place task and 2 environments for Mobile Manipulation task in parallel per GPU in the IsaacSim simulator."

Pick-and-Place uses more parallel environments (8 per GPU) because each environment is simpler (a tabletop scene with a robot arm) and requires less memory. Mobile Manipulation uses fewer (2 per GPU) because each environment is larger (full room with mobile base, arm, furniture) and the navigation planning is more computationally intensive. The paper notes that this "scales easily to multiple GPUs in a cluster" for even larger-scale generation.

Runtime metrics. From Appendix A.3:

Pick-and-Place: 8-10 seconds per demonstration when run sequentially; with 8 parallel environments, total time is 15-20 seconds, giving an effective rate of 2-3 seconds per demonstration.
Mobile Manipulation: 8-10 seconds per demonstration in parallel (with 2 environments per GPU).
The scale of the generated dataset: over 28,000 demonstrations for Pick-and-Place with 264 unique objects, and nearly 50,000 demonstrations for Mobile Manipulation across 50 diverse scenes.

Policy Learning: From Demonstrations to Executable Policies

Policy architecture. The paper uses Diffusion Policy, a state-of-the-art imitation learning approach based on denoising diffusion probabilistic models. Diffusion Policy learns to generate action sequences by iteratively denoising random noise into coherent action trajectories, conditioned on visual observations.

Input observations. The policy takes as input RGB and depth images from multiple camera views along with end-effector state information. For Pick-and-Place:

"the camera views input includes 3 perspective cameras located at left, right, and wrist of the Franka Emika Panda robot."

For Mobile Manipulation:

"the camera views input includes 5 cameras. Three of them are perspective cameras located at left, right, and wrist of the composed mobile manipulator robot. The extra two cameras are fisheye cameras located on top of the robot, looking front and back separately for navigation purposes."

The additional fisheye cameras provide a wider field of view for navigation tasks where the robot needs to see obstacles and pathways in multiple directions.

Image resolution. Both RGB and depth images are provided at 128×128 pixel resolution to all cameras. This is a relatively low resolution chosen for computational efficiency—higher resolutions would increase GPU memory and training time, and may not provide significantly more useful information for the types of manipulation and navigation tasks evaluated.

Output actions. The policy outputs continuous action vectors that specify the next-step end-effector movement (6-DOF delta pose or absolute pose) and gripper command (open/close).

Training framework. Training is done using the Robomimic framework, a standard benchmark and implementation toolkit for imitation learning in robotics. Robomimic provides implementations of several imitation learning algorithms and handles data loading, normalization, and evaluation.

Training time. From Appendix A.3:

"We train a diffusion policy using the Robomimic framework, which takes several hours to converge on our robot action data."

The paper notes that diffusion policy was chosen "for its simplicity" and that "other methods, such as fine-tuning a VLA, may prove more efficient and we leave for future work." A VLA (Vision-Language-Action model) is a more recent paradigm that directly predicts actions from language instructions and visual observations using large pretrained transformer models.

Evaluation protocol. Policies are evaluated on held-out test scenes that were not used during training. The paper reports:

"We report success rates averaged across three held-out scenes, each evaluated with 90 random robot spawning poses and object configurations."

This is a rigorous evaluation: 3 scenes × 90 configurations = 270 trials per reported success rate. The random spawning poses and object configurations test whether the policy can generalize to novel initial conditions within the same scene geometry.

Privileged agent baseline. The paper compares the learned policy against a privileged agent—the motion planner itself, which has full access to the 3D scene state (exact object positions, perfect depth, complete collision information). The learned policy only sees partial visual observations (RGB and depth from a limited number of camera views). The gap between the learned policy and the privileged agent measures how much performance is lost due to partial observability and the difficulty of learning from visual inputs. The scaling curves in Figures 10 and 11 show that the learned policy approaches but does not exceed the privileged agent, as expected.

Scaling evaluation. The key experiment in the paper's embodied AI section demonstrates scaling behavior: as the number of training scenes (or demonstrations) increases, policy success rate increases monotonically, eventually approaching the privileged agent's performance. This is the central empirical claim of the paper's second contribution—that SAGE's generated data is of sufficient quality and diversity to drive meaningful policy improvement through scaling alone.

Summary of Design Choices and Their Justifications

MCP over fixed pipeline: Enables adaptive tool selection and self-correction rather than static error accumulation. The same LLM that understands the user's intent also decides runtime execution order based on feedback.
Text-to-3D asset generation over asset retrieval: Enables true open-vocabulary scene creation and unbounded visual diversity. TRELLIS can synthesize objects for which no pre-existing 3D model exists.
VLM-estimated physical properties over manual specification: Automates the assignment of mass, dimensions, and material parameters without requiring domain expertise. This is essential for scaling—a human annotator cannot estimate properties for 565,000 unique objects (the scale of the SAGE-10k dataset).
Two-pass physics validation (simulate, record fallen pose, re-simulate) over predictive stability estimation: Simpler and more reliable than trying to predict stable configurations analytically. Accepts the physically correct resting state even when the initial placement was unstable.
Multi-view visual critic over single-view or no visual feedback: Gives the agent spatial awareness that text-based LLM reasoning alone cannot provide. Corner views reveal problems (floating objects, collisions, sparseness) invisible in a top-down view.
Depth-first search with early stopping over exhaustive evaluation: Accepting the first valid placement dramatically reduces simulation overhead (from potentially 30-50 sims per object to typically 1-3).
Batch simulation for floor/wall objects, individual for on-top: Optimizes the trade-off between validation thoroughness and computational cost. Floor and wall objects are less likely to be unstable (large base, low center of mass) so batch simulation is sufficient.
Decomposed sequential policies for long-horizon tasks over single end-to-end policy: Simplifies the learning problem and allows each subtask to be optimized independently. Each policy can use a different observation space and action parameterization appropriate to its stage.
2D occupancy grid for navigation collision checking over 3D mesh collisions: Sufficient for ground-plane navigation and much faster, enabling RRT to explore thousands of configurations in reasonable time.
Diffusion Policy over alternative imitation learning methods: Chosen for simplicity and demonstrated effectiveness on manipulation tasks. The paper acknowledges that other methods (particularly VLA models) may be more efficient and leaves this as future work.

4. Key Insights and Innovations

Innovation 1: Simulation-Readiness as a First-Class Generation Objective, Not an Afterthought

The dominant paradigm in 3D scene generation—from rule-based systems through data-driven methods to LLM-based pipelines like Holodeck—treats the problem as one of producing visually and semantically plausible static arrangements of objects. The output is a 3D model: a set of meshes with positions and orientations. Whether that arrangement is physically stable under gravity, whether objects interpenetrate in ways that would cause explosive forces in a physics engine, or whether objects have the mass and material properties needed for dynamic simulation—these questions are simply not part of the generation objective. They are, at best, addressed manually after generation, and at worst, ignored entirely.

SAGE makes a fundamental conceptual shift: simulation-readiness is not a post-processing step but a core generation constraint, enforced continuously through a closed feedback loop. The key mechanism is the physics critic—simulator-in-the-loop validation that checks every object placement, addition, movement, and removal by actually running a physics simulation and verifying stability. But the innovation is not the mechanism itself; it is the reframing of the problem. Prior work asked, "Does this scene look right?" SAGE asks, "Does this scene behave right under physics?" This changes what counts as a valid output.

The empirical evidence for why this matters is stark. Table 2 shows that Holodeck—a state-of-the-art LLM-driven system that produces visually appealing scenes—achieves only 63.8% average stability, meaning more than one in three objects shifts or falls when physics is applied. SceneWeaver, the most advanced agentic system, reaches 67.7%. SAGE achieves 99.9%. Figure 5 visualizes this concretely: baseline scenes show displaced and fallen objects after simulation; SAGE scenes remain stable. The ablation in Table 3 isolates the effect: adding the physics critic alone raises stability from 80.3% (visual critic only) to 99.6%, with collision rates dropping from 7.8% to 1.9%.

The significance of this reframing extends beyond the metric. It fundamentally changes the relationship between scene generation and embodied AI. In the old paradigm, generated scenes and simulation environments were separate artifacts—you would generate a scene, then (hopefully) make it simulation-ready through manual effort. In SAGE's paradigm, scene generation and simulation validation are a single integrated process. This matters because it makes the scaling vision feasible: you cannot manually fix physics violations in 10,000 scenes, but you can run a physics critic as part of the generation loop at scale. The SAGE-10k dataset—10,000 scenes with 565,000 uniquely generated objects, all physics-validated—is evidence that this integration unlocks scalability that manual approaches cannot achieve.

This is not an incremental improvement over prior work; it is a fundamental shift in the objective function of scene generation for embodied AI. It changes the output from a static asset to a dynamic, interactable, simulator-ready environment. The downstream policy learning results (Figures 10, 11) demonstrate that this matters for the ultimate goal: policies trained on physics-validated SAGE data outperform those trained on baseline-generated data by substantial margins, even when controlling for demonstration count.

Innovation 2: Critic-Guided Self-Improvement as a General Pattern for Generative Systems with Complex Constraints

Many generative systems face a common problem: the outputs of a generator contain errors—missing elements, incorrect placements, physically impossible configurations—but the generator itself has no mechanism to detect or correct these errors. The standard solution is to improve the generator (train a better model, hand-craft more rules, increase capacity) so that its outputs are correct on average. SAGE takes a different approach that represents a general architectural pattern: pair a generator with multiple complementary critics that provide structured feedback, and let an agent use that feedback to iteratively improve the output.

This is not the same as training with a loss function or reinforcement learning reward. Those mechanisms operate at training time and shape the generator's distribution. SAGE's critics operate at inference time and shape a specific output. The distinction matters because it allows the system to handle errors that are too sparse, too context-dependent, or too physically complex to be captured in a training signal. A VLM can notice that a particular room arrangement feels "off" without needing a labeled dataset of millions of room arrangements labeled "off" vs. "not off"; a physics simulator can detect instability in a specific placement without needing a learned stability predictor.

The key design insight is the division of critical labor between visual and physics critics. The visual critic addresses semantic errors—missing objects, implausible combinations, aesthetic imbalances—that require world knowledge and visual reasoning. The physics critic addresses physical errors—instability, collisions—that require explicit simulation. Neither critic alone suffices (Table 3: visual-only gives good realism but poor stability; physics-only gives good stability but poor visual quality), but together they cover the space of possible errors. This division is not arbitrary; it mirrors the natural categories of what can go wrong in scene generation (the scene doesn't look right, or the scene doesn't behave right) and assigns each category to the critic best equipped to handle it.

The agent's role in this architecture—dynamically selecting which generator to invoke in response to critic feedback—is equally important. The agent is not just executing a fixed sequence of generate-critique-revise; it is deciding what kind of revision is needed based on the specific feedback. A "missing chairs" critique triggers an Asset Placer call, not an Asset Mover call. This makes the system more efficient than a brute-force generate-and-test loop and more flexible than a pipeline with hard-coded revision rules.

This pattern—generator + specialized critics + agent for adaptive correction—is not specific to scene generation. It could apply to any generative domain where outputs must satisfy multiple, qualitatively different constraints: code generation (functional correctness critic + style/readability critic), document generation (factual accuracy critic + coherence critic), design generation (structural integrity critic + aesthetic critic). SAGE provides a worked example and empirical validation of a pattern that has broader significance for generative AI systems.

Innovation 3: Open-Vocabulary Object Synthesis Enables Unbounded Scene Diversity Without a Pre-Existing Asset Library

Prior scene generation systems—both procedural (ProcTHOR, Infinigen Indoors) and data-driven (ATISS, DiffuScene)—are fundamentally limited by the assets they can access. Procedural systems have a fixed set of object types defined by their rules; data-driven systems are trained on datasets with closed taxonomies; LLM-based systems like Holodeck retrieve from existing asset libraries. In all cases, the diversity of the generated scenes is bounded by the diversity of the available assets. You cannot generate a "cyberpunk game den" if no cyberpunk furniture exists in your library.

SAGE's decision to use text-to-3D generation (TRELLIS) rather than asset retrieval is not merely a technical substitution; it is a fundamental architectural choice that removes the diversity ceiling. Any object that can be described in text can be synthesized as a 3D mesh, regardless of whether a similar object exists in any training dataset or asset library. This enables truly open-vocabulary scene generation: a "fairy-tale princess room" with ornate, fantasy-style furniture (Figure 3), a "rusty and dusty restroom" with corroded fixtures (Figure 3), a "starry-night bedroom" with celestial-themed decor (Figure 4)—these are not variations on a few base templates; they involve synthesizing objects that likely have no direct analogs in standard furniture datasets.

This choice has downstream implications that go beyond visual novelty. For embodied AI, diversity of object geometry and appearance is critical for generalization. A policy trained only on one specific mug will not generalize to mugs of different shapes, sizes, or materials. Category-level augmentation (Section 3.2.1), powered by TRELLIS, produces diverse object instances from augmented text descriptions—different shapes, colors, materials—while maintaining the same semantic category. This is not possible with retrieval because the diversity is limited to whatever variants exist in the library. With synthesis, the diversity is unbounded: the LLM can generate arbitrarily many textual variations ("tall ceramic mug with blue glaze," "wide stainless steel travel mug," "small espresso cup with floral pattern"), and TRELLIS can synthesize a corresponding 3D asset for each.

The empirical evidence for this choice's importance comes from the embodied AI experiments. Figure 10 (right) shows that increasing the number of unique objects in the training data (via category-level augmentation) improves policy success rate, demonstrating that object diversity directly translates to better generalization. The cross-evaluation in Table 5 provides indirect evidence: SAGE-trained policies achieve higher success rates on baseline-generated scenes than policies trained on those baselines, suggesting that the diversity and quality of SAGE's synthesized assets produces more robust visual representations.

This is a fundamental rather than incremental contribution. It changes the scaling dynamics of scene generation from linear (you can generate as many scenes as you have assets × time) to multiplicative (you can generate scenes, and within each scene, you can generate novel object instances, each of which can be further varied). The SAGE-10k dataset with 565,000 uniquely generated objects is evidence of this multiplicative scaling in practice.

Innovation 4: Empirical Demonstration That Simulation-Generated Data Can Drive Policy Scaling for Embodied AI

The vision of using simulation to generate unlimited training data for robots is not new—it has been a motivating force behind the development of physics simulators, reinforcement learning for robotics, and sim-to-real transfer research for decades. However, the practical realization of this vision has been bottlenecked by the difficulty of creating the simulation environments themselves. Most prior demonstrations of policy learning from simulation data use manually created scenes (iBEHAVIOR, RoboCasa, LIBERO) or procedurally generated scenes with limited diversity (ProcTHOR). There has been no demonstration—prior to SAGE—that automatically generated, open-vocabulary, physics-validated scenes can serve as effective training data for generalizable robot policies.

SAGE provides this demonstration, and the results are not obvious a priori. It would be entirely reasonable to worry that automatically generated scenes contain subtle artifacts—mesh defects, unrealistic textures, physically implausible object arrangements that survive physics validation—that cause policies trained on them to learn spurious correlations rather than robust manipulation skills. The scaling curves in Figures 10 and 11 directly address this concern: as the number of SAGE-generated training scenes increases, policy success rate increases monotonically, approaching the performance of the privileged agent (which has access to perfect state information). If the generated data were fundamentally flawed, we would expect the scaling curves to plateau at a low ceiling or even degrade. They do not.

The cross-evaluation experiment (Table 5) provides particularly strong evidence. Policies trained on SAGE data are evaluated on scenes generated by the baseline systems (Holodeck and SceneWeaver)—scenes the policy has never seen during training, generated by different methods with different visual characteristics. SAGE-trained policies achieve 39.1% success on their own SAGE test scenes, 24.7% on SceneWeaver scenes, and 13.1% on Holodeck scenes. While performance drops on out-of-distribution scenes (as expected), SAGE-trained policies outperform policies trained directly on those baselines' data: policies trained on Holodeck data achieve only 16.2% on Holodeck test scenes; policies trained on SceneWeaver data achieve 13.2% on SceneWeaver test scenes. This means SAGE-generated data produces policies that are more robust—even on scenes from other generation systems—than policies trained on those systems' own data.

The significance of this finding is that it validates the central premise on which SAGE, and the broader simulation-driven scaling agenda, rests: synthetic data, if generated with sufficient quality controls (physics validation, visual critique, diverse augmentation), can substitute for manually curated training environments and drive meaningful scaling behavior. This is not a theoretical contribution but an empirical one with direct practical implications: it means the investment in building automated scene generation systems like SAGE is justified by measurable downstream gains in policy performance. This is a fundamental contribution because it shifts the burden of proof—prior to this work, one could reasonably doubt whether automatically generated scenes would be "good enough" for policy learning; after this work, the question becomes how to make them even better.

5. Experimental Analysis

Evaluation Methodology

Dataset. The scene generation evaluation uses the authors' own protocol: 10 generated scenes per room type (Bedroom, Kitchen, Living Room) for each method, with scores averaged across these 10 scenes. The embodied AI evaluation uses 3 held-out test scenes for each task (Pick-and-Place and Mobile Manipulation), each evaluated with 90 random robot spawning poses and object configurations, yielding 270 trials per reported success rate. There is no pre-existing benchmark dataset for this task; the evaluation protocol is defined by the paper itself.
Base model(s). SAGE integrates multiple foundation models. The agent LLM and the integrated LLM used within generator tools is gpt-oss-120b. The VLM for visual reasoning (both the visual critic and physical property estimation) is Qwen3-VL-30B-A3B-Instruct. The text-to-3D generator is TRELLIS. The material generator for floor and wall textures is MatFuse (with Flux.1-dev used for supplementary visual results). The choice of gpt-oss-120b is described as using "open-source models hosted via self-managed APIs to ensure reproducibility."
Metrics. Scene generation is evaluated along two axes. Visual quality metrics (Realism, Functionality, Layout, Completeness) are scored by GPT-4.1 on an unspecified scale (the paper does not state the maximum, but Table 2 values range from roughly 5 to 10, suggesting a 1–10 Likert scale). Physical validity metrics include Collision % (the ratio of 3D object meshes that interpenetrate, measured using the trimesh library) and Stability % (the fraction of objects that remain stable after 120 simulation steps in Isaac Sim, where an object is unstable if its relative translation exceeds 0.2 meters or its rotation exceeds 8 degrees). Embodied AI evaluation uses success rate (%): for Pick-and-Place, whether the object was successfully grasped and placed at the target; for Mobile Manipulation, whether all four sequential stages completed successfully.
Baselines. Two baselines from prior work are evaluated using their official codebases. Holodeck (Yang et al., 2024) is an LLM-driven pipeline that generates scenes from text prompts using a fixed sequence of modules (LLM planning → asset retrieval → constrained placement). It lacks self-improvement and physics validation. SceneWeaver (Yang et al., 2025) is a concurrent agent-based system that supports self-reflection and collision-aware placement, but omits simulator-in-the-loop validation and systematic physical property assignment. For the embodied AI experiments, two additional baselines are constructed by ablating SAGE components: Baseline 1 mimics SceneWeaver by removing the physics critic and replacing text-to-3D synthesis with asset retrieval; Baseline 2 further replaces the agent with a fixed pipeline, resembling Holodeck.
Generation budget / compute accounting. Scene generation is not evaluated under a fixed compute budget—the metric is output quality regardless of generation time. The paper reports that generating a scene with approximately 20 objects takes roughly 10 minutes, with time scaling linearly with object count. For action generation, parallelization across 8 environments per GPU yields effective rates of 2–3 seconds per demonstration for Pick-and-Place and 8–10 seconds per demonstration for Mobile Manipulation. Policy training takes "several hours to converge." There is no FLOPs-matched comparison or compute-normalized evaluation across methods.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported for the scene generation results. The 10-scene-per-room-type evaluation is a fixed sample without confidence intervals. For embodied AI, success rates are averaged across 3 held-out scenes with 90 configurations each (270 trials total), but no standard deviations or confidence intervals are reported. The cross-evaluation in Table 5 reports success rates on held-out scenes from different generation methods, but again without uncertainty quantification.

Main Quantitative Results

Scene Generation Quality (Table 2, Figures 3–5)

Common room types. Table 2 presents the headline comparison across three room types (Bedroom, Kitchen, Living Room) averaged over 10 scenes each. SAGE achieves the highest scores on every metric:

Object count (#Obj): SAGE places substantially more objects (48.3, 47.6, 48.8 across room types; average 48.2) compared to Holodeck (28.5, 28.5, 34.0; average 30.3) and SceneWeaver (17.5, 37.5, 18.1; average 24.4). This reflects SAGE's visual critic detecting sparseness and suggesting additional objects.
Visual quality (Realism, Functionality, Layout, Completeness): SAGE leads on all four metrics. On average: Realism 8.8 vs. 7.5 (Holodeck) and 8.6 (SceneWeaver); Functionality 9.5 vs. 6.7 and 8.9; Layout 7.9 vs. 5.0 and 7.3; Completeness 8.2 vs. 6.5 and 7.2. The gap is largest on Functionality (SAGE scores 2.8 points higher than Holodeck) and Layout (SAGE scores 2.9 points higher than Holodeck), suggesting that the visual critic and iterative placement particularly improve the functional coherence and spatial arrangement of objects.
Physical validity: The gap is dramatic. SAGE achieves 1.9% collision rate versus 22.0% (Holodeck) and 32.8% (SceneWeaver). Stability is 99.9% for SAGE versus 63.8% (Holodeck) and 67.7% (SceneWeaver). More than one-third of objects in baseline scenes are physically unstable. Figure 5 visualizes this: baseline scenes show displaced and fallen objects post-simulation; SAGE scenes remain intact.

Open-vocabulary generation. Figures 3 and 4 show qualitative results for stylistically diverse scenes ("Fairy-tale princess room," "Rusty and dusty restroom," "Cyberpunk game den," "Starry-night bedroom"). No quantitative metrics are reported for these open-vocabulary scenes—the evaluation is purely qualitative, relying on visual inspection of the generated outputs. The paper claims SAGE "produces more complete scenes with more realistic layouts on common room types, while following the style prompts more faithfully on open-vocabulary queries," but this latter claim has no numerical evidence.

Physical stability visualization. Figure 5 provides a direct visual comparison of stability. Three scenes (one per method) are shown before and after physics simulation in Isaac Sim. Both Holodeck and SceneWeaver scenes exhibit objects that have shifted or fallen (displaced chairs, fallen decorations), while the SAGE scene remains visually identical before and after simulation. This is a compelling visual demonstration but only a single qualitative example per method—it does not represent the distribution of failure modes.

Ablation Study: Critic Components (Table 3)

Table 3 isolates the contribution of the visual and physics critics by ablating them from the full SAGE system. Results are averaged over five scenes across three room types (15 total scenes). The four configurations are:

Neither critic: 35.3 objects, Realism 8.5, Functionality 9.2, Layout 7.5, Completeness 7.3, Collision 7.8%, Stability 80.3%. This is the baseline generator-only system without any critic feedback.
Visual critic only: 50.1 objects, Realism 8.9, Functionality 9.5, Layout 8.1, Completeness 8.1, Collision 3.7%, Stability 84.1%. The visual critic substantially improves visual quality metrics (especially object count, which nearly doubles from 35.3 to 50.1, and completeness, from 7.3 to 8.1) and modestly improves collision rate (7.8% → 3.7%) but only marginally improves stability (80.3% → 84.1%).
Physics critic only: 36.8 objects, Realism 8.8, Functionality 9.3, Layout 7.7, Completeness 7.8, Collision 1.9%, Stability 99.6%. The physics critic dramatically improves stability (80.3% → 99.6%) and collision rate (7.8% → 1.9%) with relatively small impact on visual quality metrics (modest improvements in realism and functionality, minor decrease in layout and completeness vs. the visual critic).
Both critics (full SAGE): 53.7 objects, Realism 8.9, Functionality 9.6, Layout 8.0, Completeness 8.2, Collision 0.8%, Stability 100.0%. The combination achieves the best of both: highest object count, best visual metrics, near-zero collisions, and perfect stability.

The key takeaway is that the two critics are complementary, not redundant. The visual critic adds semantic richness (more objects, better completeness) while the physics critic adds physical validity. Each alone addresses a different class of errors, and the combination yields a scene that is both visually rich and physically stable.

Embodied AI Policy Learning (Figures 10–11, Tables 4–5)

Scaling trends. Figures 10 and 11 demonstrate monotonic improvements in policy success rate as the number of training scenes (or demonstrations) increases. For Pick-and-Place (Figure 10, right): as the number of demonstrations increases from approximately 2,000 to 28,000, success rate rises from roughly 20% to approximately 63%, approaching the privileged agent's performance (roughly 65% for motion planning). A second curve on the same plot shows that increasing the number of unique objects (via category-level augmentation) similarly improves success rate, with a curve that tracks closely with the demonstration-count curve. For Mobile Manipulation (Figure 11, right): as the number of scenes increases, SAGE-trained policy success rises from roughly 10% (at 5 scenes) to approximately 46% (at 50 scenes), while the privileged agent achieves roughly 55% (motion planning on training scenes) and approximately 53% (motion planning on test scenes). Both baseline curves (Baseline 1 without physics critic, Baseline 2 with fixed pipeline) plateau at much lower success rates—roughly 10–15% at maximum scale.

Policy rollout vs. motion planning. Table 4 compares the success rates of the learned policy rollout against the privileged motion planning oracle. For Pick-and-Place: motion planning achieves 65.3% (train) and 57.7% (test); policy rollout achieves 63.4% (train) and 50.0% (test). For Mobile Manipulation: motion planning achieves 68.4% (train) and 52.8% (test); policy rollout achieves 54.6% (train) and 46.0% (test). The policy closely approaches motion planning performance on training scenes and shows a moderate generalization gap on test scenes. This confirms that the generated demonstrations are learnable and that imitation learning can recover most of the motion planner's capability.

Cross-evaluation generalization (Table 5). Policies trained on data from each method are evaluated on held-out test scenes from all three methods. The key findings:

SAGE-trained policies achieve 46.0% on SAGE test scenes, 39.1% on SceneWeaver test scenes, and 24.7% on Holodeck test scenes.
SceneWeaver-trained policies achieve only 13.2% on their own test scenes and 14.4% on SAGE test scenes.
Holodeck-trained policies achieve 16.2% on their own test scenes and 13.5% on SAGE test scenes.
SAGE-trained policies outperform baselines on their own test scenes: SAGE gets 39.1% on SceneWeaver scenes vs. SceneWeaver's 13.2% on its own scenes; SAGE gets 24.7% on Holodeck scenes vs. Holodeck's 16.2% on its own scenes.

This is strong evidence that SAGE's physics validation and diverse augmentation produce more robust training data that generalizes better, even to out-of-distribution scenes generated by other methods.

Ablation Studies and Robustness Checks

Visual critic alone vs. physics critic alone vs. both (Table 3): As discussed in the main results, the two critics are complementary. The visual critic alone improves visual metrics and object count but leaves stability at only 84.1%. The physics critic alone achieves 99.6% stability but reduces object count to 36.8 (versus 50.1 with visual critic). Only the combination achieves both high visual quality and near-perfect stability. This is the most important ablation, confirming that both critics are necessary and addressing different error modes.
Baseline degradation in embodied AI (Figure 11, right): Removing the physics critic and text-to-3D synthesis (Baseline 1, mimicking SceneWeaver) and further replacing the agent with a fixed pipeline (Baseline 2, mimicking Holodeck) both dramatically reduce policy success rates. At maximum scale (50 scenes), SAGE achieves ~46% while Baselines 1 and 2 plateau at roughly 15% and 10% respectively. This ablation demonstrates that all three components—agentic orchestration, physics validation, and generative object synthesis—contribute meaningfully to downstream policy performance.
Cross-evaluation robustness (Table 5): The cross-evaluation experiment serves as a robustness check on the generality of SAGE-trained policies. The finding that SAGE policies outperform baselines even on baseline-generated scenes provides evidence that SAGE's training data produces more generalizable visual representations. This is a non-obvious result—one might expect policies to perform best on the data distribution they were trained on, regardless of quality. That SAGE policies transfer better suggests that physics validation and diverse augmentation produce training data that covers a more generalizable manifold.
SAGE-10k dataset (Figure 6): The paper reports generating a 10,000-scene dataset across 50 room types and 50 styles, containing 565,000 uniquely generated 3D objects. This serves as an existence proof for scalability—the system can produce scenes at volume—but no downstream policy experiments are reported on this dataset, so its quality for training is unverified beyond the smaller-scale experiments in Section 4.2.
Multi-room extension (Figure 7): The paper demonstrates multi-room generation ("Multilingual teacher's apartment," "Mid-century modern family home") as an extension. This is qualitative only—no metrics are reported for multi-room scenes, and these are not included in the quantitative evaluations.
Image-conditioned generation (Figure 8): The paper shows that SAGE can be extended to accept reference images as input by using Qwen3-VL to extract style and object attributes. Again, this is purely qualitative with no quantitative evaluation.
Articulated objects (Figure 9): SAGE can incorporate articulated objects from PartNet-Mobility and generate robot actions involving them (e.g., "open drawer, place bowl, close drawer"). This is a qualitative demonstration of extensibility rather than a controlled experiment.

Negative results. The paper does not report any explicit negative results in the sense of SAGE components failing to improve metrics. All ablations show improvements from adding components. However, there are implicit negative findings:

Without the visual critic, the system generates scenes that are visually sparse (35.3 objects vs. 53.7) and less complete (7.3 vs. 8.2).
Without the physics critic, stability is poor (80.3%) and collision rates are high (7.8%).
Baseline embodied AI policies fail to scale effectively, plateauing at low success rates.
The policy learning results show a gap between train and test performance (Table 4), indicating that generalization to novel scenes is imperfect despite augmentation.

Critical Assessment

Claim 1: SAGE produces simulation-ready scenes with state-of-the-art realism and physical validity.

What the experiments demonstrate. Table 2 convincingly shows that SAGE outperforms Holodeck and SceneWeaver on both visual and physical metrics for three common room types. The stability metric (99.9% vs. 63.8% and 67.7%) and collision rate (1.9% vs. 22.0% and 32.8%) demonstrate a dramatic improvement in physical validity. Figure 5 provides qualitative evidence that this metric difference corresponds to meaningful behavioral differences (baseline scenes collapse; SAGE scenes remain stable).

What the experiments do not demonstrate. Several limitations qualify this claim:

GPT-4.1 as the evaluator for visual quality. The visual metrics (Realism, Functionality, Layout, Completeness) are scored by GPT-4.1, a language model that evaluates scenes from rendered images or scene descriptions. The paper does not validate whether GPT-4.1's scores correlate with human judgments of visual quality, nor does it report inter-rater reliability or human evaluation. It is possible that GPT-4.1's scoring is biased toward scenes that match its training distribution (which likely includes many well-furnished, conventional interior designs), potentially favoring SAGE's tendency to add more objects. The paper does not report a human evaluation study to ground-truth the GPT-4.1 scores.
Only 10 scenes per room type. The evaluation uses 30 scenes total (10 per room type). With no confidence intervals reported, it is impossible to assess whether SAGE's advantage is statistically significant or could be explained by sampling variance. A 10-sample evaluation is small enough that a few lucky or unlucky generations could shift the average meaningfully.
No ablation on the VLM for visual critic quality. The visual critic uses Qwen3-VL. There is no experiment testing whether a different VLM (or a non-VLM heuristic) would produce comparable results, leaving open the question of how dependent SAGE's visual quality is on the specific VLM choice.
Open-vocabulary scenes are not quantitatively evaluated. The claim of "open-vocabulary generation" is supported only by qualitative examples (Figures 3, 4, 7, 8). There are no metrics, no user study, and no comparison against baselines on non-standard room types. The paper does not establish that SAGE maintains its quality advantage on out-of-distribution prompts.

Claim 2: The agentic framework with visual and physics critics enables adaptive tool use and self-improvement.

What the experiments demonstrate. Table 3 shows that adding the visual critic substantially increases object count and visual quality, while adding the physics critic dramatically improves stability. The combination achieves the best results. This demonstrates that the critics improve scene quality, which is consistent with the claim that they enable self-improvement.

What the experiments do not demonstrate. The paper does not provide direct evidence for adaptive tool use:

No ablation comparing agentic vs. fixed tool sequencing. The claim that MCP-based adaptive orchestration is superior to a fixed pipeline is supported by the comparison against Holodeck (which uses a fixed pipeline) and SceneWeaver (which is agent-based but lacks physics validation). However, these baselines differ in multiple ways simultaneously—Holodeck uses asset retrieval, not text-to-3D; SceneWeaver lacks physics validation. The difference in performance cannot be attributed specifically to adaptive tool selection versus other confounding factors. A clean ablation would compare SAGE's agent-based orchestration against an otherwise identical system that calls tools in a fixed, pre-determined order. This ablation is not performed.
No analysis of agent decisions. The paper does not report what fraction of tool calls are driven by critic feedback versus the agent's initial plan, how many iterative cycles are typically required, what types of errors the agent most commonly corrects, or whether the agent's decisions are consistent across runs. Without such analysis, the claim of "adaptive tool use" remains a description of the system architecture rather than a demonstrated capability.
Self-improvement vs. just having critics. The ablation in Table 3 shows that critics improve quality, but it does not demonstrate that iterative improvement (multiple rounds of generation and critique) is superior to a single round. A system could call the visual critic once after initial generation, apply all suggested fixes, and stop—it is unclear from the experiments whether additional iteration provides marginal benefit.

Claim 3: Scene augmentation strategies produce diverse scenes that improve policy generalization.

What the experiments demonstrate. Figure 10 (right) shows that increasing the number of unique objects (via category-level augmentation) improves policy success rate for Pick-and-Place. Figure 11 (right) shows that increasing the number of scenes (via layout-level augmentation) improves success rate for Mobile Manipulation. The cross-evaluation in Table 5 shows SAGE-trained policies generalize better to baseline-generated scenes than baselines do to their own scenes.

What the experiments do not demonstrate.

No ablation on augmentation type vs. raw data quantity. The scaling curves (Figures 10, 11) show that more data improves performance, but they do not isolate whether the augmentation is providing benefit beyond simply having more demonstrations from the same scenes. A key ablation would compare: (a) N demonstrations from 1 scene with no augmentation, (b) N demonstrations from 1 scene with configuration-level augmentation, (c) N demonstrations from 1 scene with category-level augmentation, and (d) N demonstrations from M different base scenes with no augmentation. Without this, it is unclear whether augmentation adds value beyond what could be achieved by simply generating more base scenes.
No evaluation of augmentation quality. The paper does not report whether augmented objects are physically stable (do they pass physics validation at the same rate as base objects?), visually coherent (does a "rustic wooden bowl" generated from an augmented text prompt actually look like a bowl and fit the scene?), or semantically appropriate. A failure mode of LLM-based text augmentation would be generating descriptions that TRELLIS cannot faithfully render or that produce objects inconsistent with their category.
The cross-evaluation (Table 5) has a data scale confound. SAGE-trained policies are trained on SAGE data, which contains more diverse objects and physics-validated scenes. The baselines are trained on less diverse, non-physics-validated data. The fact that SAGE policies generalize better to baseline scenes could be because SAGE data is higher quality, more diverse, or both—the experiment does not disentangle these factors.

Claim 4: Policies trained on SAGE data exhibit clear scaling trends and generalize to unseen objects and layouts.

What the experiments demonstrate. The scaling curves in Figures 10 and 11 show monotonic improvements with more data, and the policy approaches the privileged agent's performance. This is evidence of scaling behavior and suggests the data is learnable.

What the experiments do not demonstrate.

No demonstration of generalization to truly novel object categories. Category-level augmentation varies objects within the same category (different mugs, different bowls). The experiments do not test whether the policy can handle a completely novel object category not seen during training (e.g., training on mugs and bowls, testing on cups and plates). The claim of "generalizing to unseen objects" is supported only for within-category variation.
No sim-to-real transfer. All policy evaluation is in simulation. The paper does not demonstrate that policies trained in SAGE-generated scenes transfer to a physical robot. This is a common limitation of simulation-based policy learning papers, but it means the ultimate test of the generated data's quality—whether it enables real-world task performance—is absent.
No comparison against human-designed scenes. The privileged agent baseline uses motion planning on the SAGE-generated scenes. There is no comparison against policies trained on manually designed scenes (e.g., scenes from RoboCasa or LIBERO) to establish whether SAGE-generated scenes are as effective as human-curated ones for policy learning.
Only two tasks, one robot morphology. All embodied AI experiments use a Franka Emika Panda arm (and Omron LD-60 base for mobile manipulation). There is no evidence that the findings generalize to other robot morphologies, gripper types, or task families.
Potential data leakage. The paper states that SAGE generates task-relevant objects based on the user's task description (e.g., "pick up a bowl and place it on the table" causes SAGE to explicitly include a bowl and a table). If the policy training data is generated from the same task specification, there may be unintended correlations between the scene generation and the task specification that make the task easier than it would be in a truly independent scene. This is not discussed as a limitation.

Overall Assessment

The experiments provide strong evidence for the paper's primary technical contribution: that simulator-in-the-loop physics validation, combined with visual critique, produces scenes that are more physically stable and visually complete than prior methods. The stability improvement (63.8% → 99.9%) is dramatic and well-supported by both quantitative metrics and qualitative visualization. The ablation study (Table 3) cleanly isolates the contribution of each critic.

The experiments provide moderate evidence for the secondary contribution: that SAGE-generated data can serve as effective training data for embodied policies. The scaling curves are convincing, and the cross-evaluation shows SAGE policies transfer better than baselines. However, the lack of sim-to-real transfer, the limited task diversity, and the absence of comparisons against human-designed scenes leave open the question of how SAGE-generated data compares to the best available alternatives for policy training.

The weakest empirical support is for the "agentic" and "adaptive" aspects of the system. The paper demonstrates that the full system (with critics) outperforms ablations (without critics), but it does not demonstrate that the agent's dynamic decision-making—as opposed to simply having critic feedback available in a fixed pipeline—is responsible for the improvement. The MCP-based architecture is described in detail but never experimentally validated against a non-agentic alternative that has access to the same generators and critics.

Several important experiments that would strengthen the paper are absent: a human evaluation of visual quality (to validate GPT-4.1 scores), an ablation comparing agentic vs. fixed tool sequencing, a controlled experiment isolating augmentation strategy benefits from raw data quantity, a sim-to-real transfer experiment, and evaluation on a wider range of robot morphologies and tasks. The paper acknowledges some of these limitations (e.g., indoor rigid-body scope only, no RL integration) but not all of them.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted for in the Headline Numbers

The SAGE system relies on an iterative loop where the agent calls generator tools, receives feedback from visual and physics critics, and adapts its actions accordingly. However, the paper does not report what fraction of generation attempts fail and require retry, how many agent-tool interaction rounds are typically needed per scene, or what the failure rate is for individual tool calls. The runtime analysis in Appendix A.3 mentions early-stopping during placement ("we typically find stable placements within a few trials despite having 30-50 candidate locations") but gives no distributional statistics. Without this information, a practitioner cannot estimate the expected cost or latency of generating a scene of a given complexity.

The consequence is that the system's practical deployability for time-sensitive applications is unknown. If the agent requires many rounds of critique and revision, or if a non-trivial fraction of tool calls fail and need retry, the effective cost per scene could be substantially higher than the ~10 minutes reported for a 20-object scene. More importantly, the "99.9% stability" figure reflects the final stability of scenes that successfully complete the generation loop. It does not account for scenes that fail entirely—where the agent cannot find a valid configuration and must restart from scratch or where the generation process diverges into an unrecoverable state. The paper does not report a completion rate or abandonment rate. A practitioner deploying this system needs to know: how often does it produce a usable output within a time budget? The current experiments provide no answer.

The paper provides no measurement of this limitation. There is no analysis of the distribution of iteration counts, no breakdown of tool call success rates, no reporting of mean vs. variance in generation time, and no mention of scenes that failed to complete. The runtime estimates in Appendix A.3 are point estimates without variance. The mitigation status is nil: this limitation is not acknowledged as such, and no diagnostics or monitoring mechanisms are proposed. This is a significant gap for a system whose primary value proposition is scalability—scalability requires predictable, reliable operation, not just occasional successful outputs.

The Agentic Orchestration Claim Is Undersupported by Controlled Experiments

The paper's central architectural claim—reinforced in the title ("Agentic"), the method description, and the comparison table—is that adaptive, agent-driven tool orchestration under MCP is superior to fixed-pipeline approaches. The agent is described as dynamically selecting which tool to invoke next based on critic feedback, reasoning about the scene state, and deciding when generation is complete. This is positioned as a key advantage over Holodeck's "static" computational graph and SceneWeaver's more limited feedback mechanisms.

However, the experiments do not isolate the contribution of agentic orchestration from other confounded variables. The primary baselines (Holodeck, SceneWeaver) differ from SAGE on multiple dimensions simultaneously: asset generation method (retrieval vs. text-to-3D synthesis), presence of physics validation, visual critique mechanism, and the agentic orchestration itself. A reader cannot determine whether SAGE's performance advantage over Holodeck comes from its agent-based decision-making or from its use of TRELLIS for object synthesis and its physics critic for stability validation. The ablation study (Table 3) tests the presence/absence of critics but keeps the agent constant—it does not compare agentic vs. fixed-order tool calling. The embodied AI baselines (Figure 11) remove the physics critic and text-to-3D synthesis but also replace the agent with a fixed pipeline, conflating multiple changes.

The consequence is that a core claim of the paper—the value of agentic adaptation—remains a design description rather than an empirically validated property. A practitioner choosing between an agentic architecture and a simpler fixed-pipeline architecture (with the same generators and critics) has no evidence from this paper to guide that decision. The agentic design adds complexity (LLM reasoning at every step, MCP protocol overhead, unpredictable execution paths) that may not be justified if a well-engineered fixed pipeline with critic feedback achieves similar quality.

The evidence for this limitation is the absence of a specific ablation: SAGE with agentic orchestration vs. SAGE with a fixed, pre-determined sequence of tool calls (e.g., initialize → place all floor objects → place all on-top objects → run visual critic → apply all fixes → run physics critic → repeat once). Such an ablation would be straightforward to implement and would directly test the agentic claim. The paper does not report it. The mitigation status is that the paper makes the agentic claim explicitly but provides no experimental evidence that isolates this factor from the many other differences between SAGE and its baselines. This is a methodological gap that weakens a headline contribution.

The Evaluation Is Narrow in Scope: Single Domain, Single Simulator, No Real-World Validation

All quantitative experiments—both scene generation quality and embodied policy learning—are conducted within a narrow scope: indoor scenes only, rigid-body physics only, the MATH benchmark-equivalent for this domain (three common room types at 10 scenes each), a single physics simulator (Isaac Sim/PhysX), and a single robot morphology (Franka Emika Panda arm, with Omron LD-60 base for mobile manipulation). The paper is transparent about the indoor and rigid-body scope, stating in Section 5:

"Our current scope emphasizes indoor scenes and rigid-body physics, and extending to outdoor settings as well as articulated and deformable objects is promising."

However, it does not discuss the potential sensitivity of results to the specific simulator, physics engine, or robot used.

The consequence is that a practitioner cannot predict how SAGE would perform in a different simulation environment (e.g., MuJoCo, PyBullet, SAPIEN), with a different robot morphology (e.g., a mobile manipulator with a different kinematic structure, a dual-arm setup, a quadcopter), or on tasks that fall outside the Pick-and-Place and Mobile Manipulation templates. Moreover, the absence of sim-to-real transfer—the ultimate test of whether SAGE-generated scenes produce policies that work on physical hardware—leaves the most important practical question unanswered. The paper's motivation emphasizes that real-world data collection is "costly and unsafe" and that simulation should substitute for it, but it never closes the loop by demonstrating that a policy trained purely in SAGE-generated simulation can successfully execute a task on a physical robot.

The evidence is the scope of the experimental sections. All scene metrics (Table 2, Table 3) are from the same simulator. All policy results (Figures 10–11, Tables 4–5) are in Isaac Sim/Lab. The paper provides qualitative demonstrations of extensions (multi-room, image-conditioned, articulated objects) but no quantitative metrics for any of them. The diversity of the SAGE-10k dataset (50 room types, 50 styles) suggests the system can generate varied outputs, but no downstream experiments are reported on this dataset. The mitigation status is that the paper acknowledges the indoor and rigid-body scope limitation and frames extensions as "promising" future work, but does not acknowledge the absence of sim-to-real validation as a gap given the paper's motivating rhetoric about replacing real-world data collection.

Visual Quality Metrics Rely on Unvalidated LLM-as-Judge Scoring

The paper's visual quality metrics—Realism, Functionality, Layout, and Completeness—are scored entirely by GPT-4.1, a large language model. The paper does not report any validation of whether GPT-4.1's scores correlate with human judgments of scene quality. The evaluation protocol involves showing GPT-4.1 either rendered images of the scene or a structured description of the scene configuration (the paper does not specify which), and asking it to rate each dimension. No human evaluation study, no inter-rater reliability metrics, and no comparison against alternative automated metrics are reported.

The consequence is that the reported visual quality advantages of SAGE (e.g., Realism 8.8 vs. 7.5 for Holodeck, Functionality 9.5 vs. 6.7) may not reflect genuine perceptual quality differences. LLM-as-judge approaches are known to be sensitive to prompt phrasing, output format, and training data biases. GPT-4.1 may systematically prefer scenes that match its internal representation of what a "well-furnished room" looks like—which could favor SAGE's tendency to place more objects (48.2 average vs. 30.3 for Holodeck), regardless of whether those additional objects genuinely improve the scene's realism or functionality. A practitioner concerned with the actual visual quality of generated scenes—as opposed to the proxy score—has no basis for interpreting the reported numbers.

The evidence is the complete absence of human evaluation or metric validation in the paper. Section 4.1.1 states:

"Visual metrics cover Realism, Functionality, Layout, and Completeness based on GPT-4.1."

The word "based" is the full extent of the methodology description. There is no discussion of prompt design, no calibration against human judgments, and no acknowledgment of the limitations of LLM-based evaluation for perceptual quality assessment. This is a significant methodological weakness given that visual quality is one of the two primary axes on which SAGE claims superiority. The mitigation status is nil: the paper does not acknowledge this as a limitation and does not propose future validation.

The System Does Not Generalize to Hard Problems: Articulated Objects, Outdoor Scenes, and Complex Physics

The paper is explicit that its scope is limited to indoor scenes and rigid-body physics. Section 5 states this as a limitation:

"Our current scope emphasizes indoor scenes and rigid-body physics, and extending to outdoor settings as well as articulated and deformable objects is promising."

However, the paper also demonstrates extensions (Figures 7, 8, 9) that might suggest these limitations are easily surmountable—multi-room scenes, image-conditioned generation, and a single qualitative example with articulated objects. A practitioner might misinterpret these qualitative demonstrations as evidence that the system already handles these extensions, when in fact they are cherry-picked examples with no quantitative validation.

The consequence is that SAGE provides no path forward for generating scenes for a large class of embodied AI tasks that are critical in practice. Outdoor navigation (delivery robots, autonomous vehicles), manipulation of articulated objects (opening doors, drawers, cabinets—which is essential for most household manipulation), deformable object manipulation (folding cloth, handling rope), and tasks involving fluids or granular materials are all outside SAGE's demonstrated capabilities. The physics critic validates stability under gravity for rigid bodies, but it cannot validate whether a drawer's joint limits are correct, whether a cloth drapes realistically, or whether a door can be opened without colliding with nearby furniture. The articulated objects extension (Figure 9) shows retrieval from PartNet-Mobility but does not demonstrate that the physics critic validates articulated object interactions or that policies can be trained effectively on scenes containing them.

The evidence is the explicit scope statement in the conclusion and the purely qualitative nature of the extensions. There are no experiments measuring SAGE's performance on articulated object scenes, outdoor scenes, or any task beyond Pick-and-Place and Mobile Manipulation with rigid objects. The mitigation status is that the paper acknowledges the scope limitation and frames extensions as future work, but the qualitative demonstrations risk creating a misleading impression of capability that the quantitative experiments do not support. The paper would be stronger if it explicitly stated what SAGE cannot currently do rather than showcasing carefully selected examples of extensions.

Policy Training Does Not Leverage Online Interaction or Reinforcement Learning

The embodied AI pipeline uses a strictly offline approach: generate scenes, use motion planning to create demonstrations, filter failures, train an imitation learning policy via Diffusion Policy. The paper acknowledges this limitation implicitly in its conclusion:

"Beyond imitation, coupling the generator with online RL and real-robot closed-loop validation could further boost performance."

However, the paper does not explore whether SAGE's simulation-ready scenes could support online policy learning—where the agent collects its own experience by interacting with the environment, learning from both successes and failures. Imitation learning from motion-planned demonstrations has a well-known limitation: the policy can only learn to imitate what the motion planner does well. If the motion planner fails on certain configurations (e.g., grasp poses in cluttered scenes, navigation through tight passages), those configurations are simply absent from the training data, and the policy never learns to handle them. The paper's failure filtering (Section 3.2.2) explicitly removes failed motion planning attempts, which means the training data is biased toward configurations where motion planning succeeds—a form of survivorship bias.

The consequence is that the learned policy's performance ceiling is bounded by the motion planner's success rate. Table 4 confirms this: policy rollout success (50.0% test for Pick-and-Place) does not exceed motion planning success (57.7% test). The policy cannot learn to outperform its teacher. For tasks where motion planning is imperfect—which is the realistic case—imitation learning from filtered demonstrations will inherit the planner's weaknesses. Online RL, by contrast, could potentially learn policies that succeed in configurations where the motion planner fails, because the agent can explore and learn from its own mistakes.

The evidence is the explicit architecture choice and the success rate numbers in Table 4. The policy never exceeds the motion planning oracle on test scenes. The mitigation status is that the paper acknowledges RL as future work but provides no experiments or analysis of what benefits online learning might provide over the current offline pipeline. Given that SAGE's scenes are fully interactive and simulation-ready—which would make them excellent environments for RL—this omission is a missed opportunity to demonstrate the full value of simulation-readiness. A practitioner considering SAGE for policy training might reasonably ask: is the infrastructure investment justified if the resulting policies cannot outperform what motion planning alone achieves?

7. Implications and Future Directions

How This Work Changes the Landscape

SAGE introduces a fundamental reframing of what it means to "generate a 3D scene" for embodied AI. Prior to this work, the field operated under an implicit separation of concerns: scene generation was about producing visually plausible static arrangements, while simulation readiness was a separate, largely manual step handled by domain experts who would fix collisions, adjust physical properties, and tune stability by hand. SAGE collapses this distinction by making simulation validation a continuous, in-the-loop constraint during generation rather than a post-hoc correction. This is not merely a technical improvement—it changes the objective function of scene generation from "looks right" to "behaves right under physics," and in doing so, it aligns the generation process with the downstream needs of embodied AI for the first time.

The significance of this reframing is best understood by comparison to what it displaces. Prior systems—whether procedural (ProcTHOR, Infinigen Indoors), data-driven (ATISS, DiffuScene), or LLM-based (Holodeck)—all implicitly treat physics as someone else's problem. The fact that Holodeck scenes achieve only 63.8% stability and that SceneWeaver scenes reach only 67.7% stability (Table 2) is not an implementation bug that could be patched with minor fixes; it is a structural consequence of architectures that never test whether their outputs are physically valid. SAGE's 99.9% stability and 1.9% collision rate demonstrate that the gap between "semantically plausible" and "simulation-ready" is large—and that closing it requires explicitly optimizing for physical validity, not hoping it emerges from better visual generation.

The paper also resolves a latent tension in the embodied AI community about the scalability and quality of synthetic training data. There has been a persistent, reasonable doubt: can automatically generated 3D scenes—synthesized by foundation models without human curation—serve as effective training environments for robot policies? The concern is that subtle artifacts (mesh defects, unrealistic textures, physically implausible arrangements that survive superficial validation) would cause policies to learn spurious correlations rather than robust manipulation skills. The scaling curves in Figures 10 and 11 provide a partial resolution: SAGE-generated data exhibits clear, monotonic scaling behavior, with policy success rates approaching those of the privileged motion planning agent. The cross-evaluation in Table 5 provides stronger evidence: policies trained on SAGE data outperform policies trained on baseline data even when evaluated on baseline-generated scenes, suggesting that SAGE's physics validation and diverse augmentation produce training data that covers a more generalizable region of the scene distribution. This is not a proof that synthetic data is universally sufficient—the experiments are limited to two tasks, one robot morphology, and rigid-body indoor scenes—but it shifts the burden of proof. Prior to this work, one could reasonably doubt whether automatically generated scenes would be useful at all; after this work, the question becomes how much synthetic data is needed, what quality controls are essential, and where the remaining failure modes lie.

Methodologically, SAGE demonstrates a generator-critic architecture that has broader implications beyond scene generation. The pattern—pair a generative system with multiple specialized critics that provide structured feedback, and use an agent to adaptively route corrective actions—addresses a fundamental challenge in generative AI: how to enforce complex, multi-faceted constraints that cannot be captured by a single loss function or reward signal. The division of critical labor between a visual critic (semantic coherence) and a physics critic (physical validity) is particularly instructive because it mirrors the natural categories of error in many generative domains—code generation needs both functional correctness and style/readability checks; document generation needs factual accuracy and coherence checks; design generation needs structural integrity and aesthetic checks. SAGE provides a worked example of this architecture with quantitative validation (Table 3: both critics are necessary; neither alone suffices), which may influence how researchers approach constraint satisfaction in other generative domains.

This work also redirects attention toward simulation infrastructure as a first-class component of generative pipelines. Prior scene generation research treated the simulator as a downstream consumer of generated assets—the generator produced a 3D model, and the simulator loaded it (often with manual fixes). SAGE treats the simulator as an active participant in the generation loop, called hundreds or thousands of times per scene to validate each placement candidate. This makes simulation speed and reliability bottlenecks for the entire pipeline, not just for downstream training. The paper's efficiency optimizations—batch simulation for floor/wall objects, early-stopping during placement, 2D occupancy grids for navigation collision checking—are not incidental engineering details; they are architectural decisions that make the closed-loop approach computationally feasible. This suggests that future generative systems that incorporate physics, dynamics, or other forms of explicit simulation will need to co-design the generation algorithm and the simulation backend rather than treating them as independent modules.

Follow-Up Research This Work Enables

Decoupling the agentic orchestration contribution from the critic and generator contributions. The paper claims that MCP-based adaptive tool orchestration is a key advantage over fixed pipelines, but this claim is confounded with the simultaneous introduction of physics validation and text-to-3D synthesis. A clean ablation experiment would fix the generators (TRELLIS, same placer logic) and critics (same visual and physics validation), then compare two conditions: (1) SAGE's agent-driven orchestration where the LLM dynamically decides which tool to call next, and (2) a fixed, pre-determined sequence (initialize → place all floor objects → place all on-top objects → run visual critic → apply all suggested additions/movements/removals → run physics critic → retry failed placements → stop). If the agentic version produces meaningfully better scenes (more objects, higher visual quality, fewer remaining issues), this would validate the agentic claim. If the fixed pipeline achieves comparable quality, it would suggest that the critics and generators—not the adaptive orchestration—are responsible for SAGE's performance, and the agentic architecture adds complexity without benefit. A strong experiment would also measure iteration count distributions for the agentic version and report how often the agent's dynamic decisions differ from the fixed sequence.

Sim-to-real transfer of policies trained on SAGE-generated scenes. The paper's motivating narrative emphasizes that simulation should substitute for costly and unsafe real-world data collection, but it never closes the loop by demonstrating that policies trained purely in SAGE-generated simulation transfer to a physical robot. A strong follow-up would: deploy the same Franka Emika Panda robot used in simulation to a physical lab setup, create a physical test scene mirroring one of the Pick-and-Place task configurations (a real table, several real mugs and bowls), and evaluate the zero-shot sim-to-real transfer success rate of a policy trained exclusively on SAGE-generated data. The primary metric would be task success rate on the physical robot compared to simulation performance. Key additional measurements: (1) does the transfer gap correlate with object category diversity (are policies trained with more category-level augmentation—and thus more diverse object appearances—more robust to the visual domain shift?), and (2) does physics validation during scene generation (which ensures stable object placements) translate to more predictable physical behavior that the policy can handle at test time? A negative result—policies that fail catastrophically on the physical robot despite high simulation success rates—would indicate that SAGE-generated scenes contain systematic domain gaps not captured by current visual fidelity metrics, and would motivate research into what additional realism properties are necessary for transfer.

Scaling laws for synthetic data quality vs. quantity in embodied policy learning. The paper demonstrates that more SAGE-generated data improves policy performance (Figures 10, 11), but it does not characterize the shape of this scaling relationship or its dependence on data quality. A systematic study would generate training datasets at multiple quality levels and multiple quantities, then fit scaling law parameters to predict test performance. Quality levels could be defined by: (a) full SAGE with both critics, (b) SAGE with physics critic only, (c) SAGE with visual critic only, (d) baseline scenes without any critics, and (e) manually curated scenes (as an upper bound). For each quality level, generate datasets at multiple sizes (e.g., 100, 500, 1,000, 5,000, 20,000 demonstrations) and train policies. This would produce a family of scaling curves from which one could read off practical questions: how many extra synthetic demonstrations are needed to match the performance of a smaller set of manually curated demonstrations? Is there a quality threshold below which scaling saturates (i.e., more low-quality data never catches up to less high-quality data)? The paper's existing data—28,000 Pick-and-Place demonstrations with 264 unique objects, 50,000 Mobile Manipulation demonstrations across 50 scenes—provides a starting point, but the quality axis is unexplored.

Extending SAGE to handle articulated objects and multi-step interaction policies. The paper's qualitative demonstration with articulated objects (Figure 9) and the mention of a task sequence ("pick up the bowl, place it in the drawer, and close the drawer") suggests a natural extension. A rigorous follow-up would: (1) integrate articulated object retrieval from PartNet-Mobility (or a generation pipeline for articulated assets) as a first-class object type in SAGE, (2) extend the physics critic to validate articulated joint dynamics (do drawers open/close without interpenetrating the cabinet? do doors swing through their full range without colliding with nearby objects? are joint forces realistic?), (3) extend motion planning to generate demonstrations for tasks involving articulated object interaction (open drawer → place object → close drawer), and (4) evaluate whether policies trained on such data can generalize to unseen articulated object instances. The key challenge is that articulated objects introduce temporal dependencies—the state of a drawer at time t depends on whether it was opened at time t-1—which neither the static scene generation pipeline nor the current motion planning framework address. A strong experiment would measure success rates on tasks requiring multiple sequential articulated object interactions and compare against a baseline that treats the task as a sequence of independent rigid-body subtasks.

Online reinforcement learning in SAGE-generated environments. The current pipeline uses offline imitation learning from motion-planned demonstrations, which bounds policy performance by the motion planner's success rate (confirmed in Table 4: policy never exceeds motion planning on test scenes). SAGE's simulation-ready scenes are ideal environments for online RL because they are fully interactive, physically stable, and can be trivially reset. A compelling follow-up would: (1) take the same scenes and task specifications used in the paper, (2) use the motion-planned demonstrations to bootstrap an initial policy, (3) then continue training with online RL where the agent collects its own experience through trial and error, and (4) measure whether the policy can exceed the motion planner's success rate by learning strategies that the planner cannot discover (e.g., exploiting dynamics, using contact-rich maneuvers, recovering from near-failure states). The key research question is whether SAGE's physics-validated environments support stable, large-scale RL training—do the physics remain stable over thousands of interaction steps, or do accumulated simulation errors cause objects to drift or scenes to degrade? A negative result (RL fails to improve beyond imitation or causes simulator instabilities) would identify residual physical modeling gaps that are invisible in the static stability metrics but matter for dynamic interaction.

Comparative evaluation of visual quality with human judgments. The paper's visual quality metrics are scored entirely by GPT-4.1, with no validation against human perception. A rigorous follow-up would conduct a human evaluation study: recruit participants (ideally with some relevant expertise—interior designers, 3D artists, or roboticists who work with simulation), show them rendered images of scenes generated by SAGE, Holodeck, and SceneWeaver (randomized and anonymized), and ask them to rate each scene on the same dimensions (Realism, Functionality, Layout, Completeness) using the same scale. Key analyses: (1) what is the correlation between GPT-4.1 scores and mean human ratings? (2) are there systematic biases (e.g., GPT-4.1 consistently rates SAGE scenes higher than humans do, or vice versa)? (3) is the relative ordering of methods preserved between GPT-4.1 and humans? A finding that GPT-4.1 and human judgments are poorly correlated would undermine confidence in the paper's visual quality claims and motivate development of better automated metrics. A finding of strong correlation would validate LLM-as-judge for this domain and provide a practical evaluation methodology for future work.

Practical Applications and Downstream Use Cases

Automated benchmark generation for embodied AI. The embodied AI community relies heavily on benchmark suites—LIBERO, CALVIN, RoboCasa, BEHAVIOR-1K—that are expensive to create and maintain because each scene and task must be manually designed. SAGE offers a direct path to automated benchmark generation: a researcher specifies a task family ("pick-and-place in kitchens with varying clutter levels") and SAGE generates hundreds of diverse, physics-validated scenes along with motion-planned demonstrations. The key advantage is that benchmarks could be regenerated on demand—preventing the overfitting that plagues static benchmarks—and could be customized to specific research questions (e.g., "test whether my policy generalizes to mugs it has never seen by generating 50 novel mug instances"). The SAGE-10k dataset (10,000 scenes across 50 room types, 565,000 unique objects) already demonstrates the scale that is achievable. A concrete deployment: a conference workshop or challenge that uses SAGE to generate a fresh evaluation set for each submission, ensuring that no team can overfit to a fixed test set.

Data augmentation for real-world robot fine-tuning. A practical challenge in deploying robots to new environments (e.g., a robot being installed in a specific hospital, warehouse, or home) is that the training data collected in one setting does not generalize to the new setting's specific geometry and objects. SAGE could be used to generate scenes that approximate the target deployment environment: a user provides a text description or reference images of the target space, SAGE generates diverse simulated variants that match its layout and object types, and a policy pre-trained on large-scale SAGE data is fine-tuned on these deployment-specific scenes before real-world deployment. The benefit is reducing the amount of expensive real-world data collection needed—the policy already has general manipulation skills from broad SAGE training, and only needs to adapt to the specific visual and spatial properties of the deployment environment. The paper's cross-evaluation results (Table 5: SAGE policies achieve 39.1% on SceneWeaver scenes and 24.7% on Holodeck scenes without any fine-tuning) suggest that SAGE-trained policies are already reasonably robust to distribution shift, which is a prerequisite for effective fine-tuning.

Curriculum generation for渐进式 (progressive) robot skill learning. Many robot learning approaches use curricula—sequences of tasks or environments of increasing difficulty—to train policies more efficiently than random sampling. SAGE's agentic scene generation could be adapted to produce explicit difficulty curricula: given a task specification, SAGE could generate a series of scenes with systematically varying difficulty, such as increasing clutter density, decreasing object size (harder to grasp), increasing distance between pick and place locations, or introducing distractor objects. Because SAGE maintains physics validation throughout, each curriculum level would remain physically valid. The paper's augmentation strategies (configuration-level, category-level, layout-level) already provide mechanisms for controlled variation; extending them with explicit difficulty parameters would make SAGE a curriculum engine. A practitioner could specify "generate 10 levels of increasing difficulty for mug-on-table grasping, from isolated mug on empty table to mug surrounded by 5 distractor objects on a cluttered table," and SAGE would produce the corresponding scenes and demonstrations.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Generating Simulation-Ready 3D Environments at Scale for Embodied AI

The Four Desiderata and Why Simultaneous Satisfaction Is Hard

Where Prior Approaches Fall Short

Real2Sim: High Realism, Poor Scalability

Rule-Based/Procedural Systems: Physical Plausibility, Limited Flexibility

Data-Driven Approaches: Realistic Layouts, Closed Taxonomies

LLM/Foundation Model Pipelines: Open Vocabulary, Weak 3D Grounding

Agentic Systems: A Step Forward, but Still Not Simulation-Ready

The Broader Importance: Simulation-Driven Scaling for Embodied AI

How This Paper Positions Itself Relative to Existing Work

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Model Context Protocol (MCP) as Integration Substrate

Generator Tools: The Four Scene Construction Operations

Scene Initializer

Asset Placer

Asset Mover

Asset Remover

The Visual Critic: Semantic and Spatial Coherence Feedback

The Physics Critic: Simulator-in-the-Loop Stability Validation

The Iterative Self-Improvement Loop

Scene Augmentation: Scaling One Scene into Many

Object Configuration-Level Augmentation

Object Category-Level Augmentation

Scene Layout-Level Augmentation

Simulation-Ready Validation During Augmentation

Action Generation: From Scenes to Robot Demonstrations

Pick-and-Place Action Generation

Mobile Manipulation Action Generation

Failure Filtering

Parallelization

Policy Learning: From Demonstrations to Executable Policies

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Simulation-Readiness as a First-Class Generation Objective, Not an Afterthought

Innovation 2: Critic-Guided Self-Improvement as a General Pattern for Generative Systems with Complex Constraints

Innovation 3: Open-Vocabulary Object Synthesis Enables Unbounded Scene Diversity Without a Pre-Existing Asset Library

Innovation 4: Empirical Demonstration That Simulation-Generated Data Can Drive Policy Scaling for Embodied AI

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Scene Generation Quality (Table 2, Figures 3–5)

Ablation Study: Critic Components (Table 3)

Embodied AI Policy Learning (Figures 10–11, Tables 4–5)

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: SAGE produces simulation-ready scenes with state-of-the-art realism and physical validity.

Claim 2: The agentic framework with visual and physics critics enables adaptive tool use and self-improvement.

Claim 3: Scene augmentation strategies produce diverse scenes that improve policy generalization.

Claim 4: Policies trained on SAGE data exhibit clear scaling trends and generalize to unseen objects and layouts.

Overall Assessment

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted for in the Headline Numbers

The Agentic Orchestration Claim Is Undersupported by Controlled Experiments

The Evaluation Is Narrow in Scope: Single Domain, Single Simulator, No Real-World Validation

Visual Quality Metrics Rely on Unvalidated LLM-as-Judge Scoring

The System Does Not Generalize to Hard Problems: Articulated Objects, Outdoor Scenes, and Complex Physics

Policy Training Does Not Leverage Online Interaction or Reinforcement Learning

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

Single-minus gluon tree amplitudes are nonzero