SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM

ArXiv: 2603.23386

🎯 Pitch

Existing 3D generation pipelines produce static meshes useless for robotics simulators, but SIMART shows a single MLLM can jointly segment a mesh into moving parts and predict their kinematic joints—allowing a static AIGC chair to be immediately dropped into a physics engine. By encoding only occupied voxels, it slashes token counts by 70% and handles AI-generated meshes where dense baselines crash from memory exhaustion.

1. Executive Summary

This paper introduces SIMART, a unified multimodal large language model (MLLM) framework that jointly performs part-level mesh decomposition and kinematic parameter prediction to transform static 3D meshes into simulation-ready articulated assets. Evaluated on PartNet-Mobility and a newly curated SIMART-Bench containing AI-generated objects, SIMART combines a Sparse 3D VQ-VAE — which reduces token counts by 70% compared to dense voxel representations by selectively encoding only occupied surface voxels — with the Qwen3-VL-8B MLLM backbone to generate per-part geometry tokens alongside structured URDF specifications (joint types, axes, limits, and physical properties). The framework achieves state-of-the-art performance across all metrics, reaching 0.928 joint type accuracy and 0.080 axis error on in-domain items while simultaneously achieving 0.690 IoU and 0.087 Chamfer Distance — roughly halving the axis error and Chamfer Distance of the next-best method. By incorporating difficulty-adaptive allocation through sparse tokenization that bypasses empty space, SIMART establishes that MLLM-based part grounding and kinematic reasoning can achieve high-fidelity decomposition and articulation prediction on diverse, previously unseen geometries, succeeding on AI-generated meshes where dense baselines encounter out-of-memory failures on objects averaging only four parts.

2. Context and Motivation

The Core Problem: Static Meshes Are Everywhere, but Simulation-Ready Assets Are Rare

The paper addresses a fundamental disconnect in the 3D asset ecosystem. Over the past several years, 3D generative models have advanced rapidly — systems like Hunyuan3D, DreamGaussian, and Seed3D can now synthesize high-quality static meshes from text or images. Yet these outputs are almost exclusively monolithic, non-decomposed meshes — a single fused surface with no notion of separate functional parts, no kinematic metadata (where do joints go? what rotates relative to what?), and no physical properties (mass, friction, joint limits). They look good in a renderer but are useless in a physics simulator or interactive environment.

On the other hand, the domains that most urgently need 3D content are precisely the ones that can't use static meshes: embodied AI, robotic simulation, and VR/AR. A robot learning to open drawers or doors needs to understand that the drawer front is a separate rigid body attached via a prismatic joint, with specific axis direction and travel limits. A VR experience where users interact with furniture needs those interactions to obey kinematic constraints — a cabinet door that swings on hinges, not one that clips through the frame. The paper frames this gap explicitly (Section 1):

"a vast majority of existing 3D assets remain unarticulated... the manual creation of such assets is prohibitively labor-intensive, underscoring the critical need for robust, automated methods for sim-ready asset generation."

This is not merely a convenience problem. It's a scaling bottleneck for entire research communities. Embodied AI needs diverse, interactive training environments — thousands of articulated objects spanning different categories, geometries, and kinematic configurations. Manually annotating URDF (Unified Robot Description Format) files for even a single complex object requires an expert modeler to segment the mesh, identify joint axes, set limits, and define the kinematic tree. Scaling this to the hundreds of thousands of objects needed for robust policy learning is infeasible.

Why Existing Approaches Fall Short

The paper identifies three families of prior work, each with specific limitations that motivate SIMART's unified design.

Direct URDF Regression from Images Compromises Geometry

The most straightforward approach is end-to-end prediction: feed an image of an object into a model and regress its URDF parameters directly. Urdformer (Chen et al., 2024) exemplifies this paradigm, constructing simulation environments from real-world images. However, the paper notes a critical limitation (Section 1):

"While directly regressing URDF models from images is intuitive... it often compromises geometric fidelity, yielding coarse outputs unsuitable for high-quality simulation."

The issue is that images are 2D projections of 3D geometry. Regressing accurate 3D part boundaries, joint axes in world coordinates, and mesh-level segmentation from a single or few views is fundamentally ill-posed — the model must hallucinate geometric detail it cannot observe. This works for approximate scene layout but fails when the output must be loaded into a physics simulator where millimeter-level joint alignment determines whether the simulation is stable or explodes.

Multi-Stage Pipelines Accumulate Errors Across Decoupled Modules

A more common strategy, represented by methods like UMP (Xu et al., 2022) and various part-decomposition-then-assembly pipelines, splits the problem into sequential stages:

1. Part decomposition: Segment the input mesh into functional components (drawer, handle, cabinet body).
2. Joint parameter inference: For each adjacent pair, predict joint type, axis, origin, and limits.
3. Post-hoc assembly: Combine the segmented parts and joint parameters into a URDF.

The paper explicitly calls out why this decoupling is fragile (Section 1):

"Part decomposition is frequently not articulation-aware: prompting 2D vision(-language) models transfers unreliably to 3D boundaries, while 3D-native segmentation methods... primarily optimize for surface-level consistency and can miss mechanically meaningful link boundaries, producing parts that look plausible yet violate kinematic affordances."

This is a subtle but devastating point. A part segmentation that looks "correct" visually might group the drawer handle with the drawer front because they're geometrically contiguous — but mechanically, the handle might be fixed to the front while the front slides relative to the cabinet body. A surface-level segmenter has no way to distinguish these cases because it doesn't reason about motion. The segmentation must be articulation-aware — informed by what moves relative to what.

The joint estimation stage compounds this fragility:

"Joint estimation further amplifies this fragility. Whether predicted from 2D cues or optimized from imperfect geometry is sensitive to mesh artifacts and restrictive priors, so the inferred joints often become incompatible with the recovered part geometry, yielding physically invalid articulation."

The two stages operate in isolation: the part segmenter produces parts without knowing what joints will connect them; the joint estimator operates on parts it didn't choose and can't modify. A small segmentation error on a drawer edge becomes a large joint axis error that renders the URDF physically invalid. There's no feedback loop.

Reconstruction-Based Methods Require Impractical Multi-State Supervision

A separate line of work (ArtGS, ArticulatedGS, PARIS, Ditto) learns articulated structure by observing objects in multiple articulation states — e.g., a cabinet with the door both open and closed, captured from multiple viewpoints. These methods can recover high-fidelity geometry and accurate kinematics but have a crippling data requirement (Section 2.1):

"these methods typically require multi-view supervision across different articulation stages... Such high-quality, multi-state visual inputs are often difficult to obtain in the wild, leading to poor generalization when faced with incomplete observations or sparse viewpoints."

For most real-world objects, you don't have captured footage of them in multiple articulation states. For AIGC-generated meshes, there are no "states" to observe — the mesh is static by construction. Reconstruction approaches are valuable for carefully curated lab settings but don't generalize to the wild.

Generative Methods Are Limited by Data Scarcity

Category-level generative methods (CAGE, SINGAPO, NAP) learn priors over how objects in a given category (e.g., "cabinets") tend to articulate, then apply those priors to new instances. The paper notes (Section 2.1):

"these frameworks are hindered by the acute scarcity and limited diversity of articulated 3D datasets compared to rigid objects. Consequently, these models are prone to overfitting, often failing to produce structurally sound or novel articulations for uncommon object categories."

The fundamental issue is that articulated 3D datasets (like PartNet-Mobility, with roughly 2,000 models) are tiny compared to static 3D datasets (Objaverse has 800K+ models). Learning robust priors from a few thousand examples is exceptionally difficult, and the resulting models often fail when presented with geometries outside their training distribution — precisely the scenario that matters for handling diverse AI-generated meshes.

MLLM-Based Methods Struggle with 3D Tokenization Efficiency

The most recent approaches (Articulate-Anything, PhysX-Anything, Articulate AnyMesh) leverage multimodal LLMs for their physical reasoning capabilities — understanding that "drawers slide" and "doors swing" from pretraining on massive image-text corpora. However, these methods either:

• Rely exclusively on 2D visual inputs (Section 2.2): "Articulate-Anything and Articulate AnyMesh leverage the visual reasoning capabilities of MLLMs to infer motion structures from rendered images. However, these methods lack an integrated 3D geometric understanding and generation pathway." The MLLM sees rendered views, not the actual 3D surface, creating a geometric grounding gap that leads to "physically inconsistent joint estimations."

• Use dense volumetric tokenization that exhausts memory (Section 2.2): PhysX-Anything "struggles with capturing fine-grained spatial information due to the heavy computational overhead of dense voxel tokens." The core issue is that dense $N^3$ voxel grids encode empty space alongside occupied space, wasting most tokens. The paper notes that this $O(N^3)$ complexity "not only triggers memory-exhaustion on complex meshes but also necessitates heavy downsampling that compromises the geometric fidelity required for precise axis localization."

This tokenization bottleneck is the most immediate technical motivation for SIMART. When a dense representation requires ~4,100 tokens per part for even moderate-resolution voxel grids, and a typical articulated object has 4+ parts, the resulting sequence length exceeds the context window or memory capacity of the MLLM backbone. The paper's ablation (Table 3) makes this concrete: the dense baseline hits Out-of-Memory (OOM) errors during fine-tuning on objects averaging just four parts. This means existing 3D-native MLLMs are fundamentally unscalable to complex articulated objects — the very objects that need articulation the most.

The Underlying Gap: No Unified 3D-Native MLLM for Joint Understanding and Generation

The paper positions itself at the intersection of two trends that haven't previously been unified (Section 1):

1. MLLMs have demonstrated emergent physical reasoning. Models like Qwen3-VL, trained on vast corpora of images and text, understand concepts like "revolute joint," "hinge," "prismatic motion," and can recognize that cabinet doors swing open around a vertical axis positioned at one edge. This capability comes "for free" from internet-scale pretraining — the model has seen millions of images of cabinets, doors, drawers, and understands what makes them move.

2. 3D generative models can produce detailed static geometry. The quality gap between generated and manually modeled static meshes has narrowed dramatically in the past year.

The missing link is a native 3D representation that is compact enough for MLLM processing yet expressive enough for precise geometric reasoning about joints and part boundaries. This is why SIMART's primary technical contribution — the Sparse 3D VQ-VAE — is not merely an efficiency optimization but a structural enabler. By reducing token counts by 70% and encoding only occupied voxels with explicit spatial coordinates, it makes per-part geometry generation feasible within the MLLM's context constraints.

The paper's framing in Section 1 captures this succinctly:

"This motivates a unified MLLM paradigm that understands such an initial 3D asset and directly generates per-part geometry (as tokens) together with a structured URDF specification... However, existing 3D-native MLLM attempts are constrained by inefficient 3D tokenization: dense volumetric encodings waste most tokens on empty space, leading to prohibitive context length and memory overhead even for understanding, let alone part-level generation."

The phrase "even for understanding, let alone part-level generation" is key. Previous 3D-native MLLMs could barely process a single dense voxel grid for classification or captioning tasks. SIMART's ambition — generating multiple per-part voxel grids simultaneously with structured kinematic metadata — would be impossible with dense representations.

How SIMART Positions Itself

SIMART presents itself not as an incremental improvement on any single prior approach, but as a unifying architecture that resolves the key bottlenecks of multiple predecessor paradigms:

• Against multi-stage pipelines: SIMART is end-to-end — part segmentation and joint parameter prediction happen jointly within the same MLLM forward pass, with cross-attention between the geometric tokens and the URDF text tokens enabling articulation-aware decomposition. The model doesn't first segment then estimate joints; it reasons about both simultaneously, informed by the physical world knowledge encoded in the pretrained MLLM backbone.

• Against 2D-only MLLMs: SIMART processes actual 3D geometry (as sparse voxel tokens) rather than rendered 2D views, providing direct geometric grounding. This eliminates the pixel-to-3D projection ambiguity that causes physically inconsistent joint predictions in Articulate-Anything and similar methods. The coordinate-aware tokenization — where each voxel explicitly encodes its $(x, y, z)$ position — means the MLLM can reason about absolute spatial relationships (e.g., "the hinge is at the left edge of the door, aligned with the $y$-axis").

• Against dense voxel MLLMs: The Sparse 3D VQ-VAE with its specialized zero-token mechanism is the enabling technology that makes part-level generation feasible. The paper's ablation (Table 3) demonstrates this empirically: the dense baseline fails entirely (OOM), the force-sparse variant succeeds but with degraded performance (0.661 type accuracy), and the zero-token sparse variant achieves strong performance (0.794 type accuracy) while using the fewest tokens (516 per object on average). The zero token isn't just about saving tokens — it prevents the MLLM's attention from being diluted by irrelevant empty-space tokens, letting it focus on the occupied voxels that convey actual geometry.

• Against fully generative methods that sacrifice input fidelity: SIMART doesn't generate the mesh from scratch — it takes the input mesh as given and decomposes it. The Sparse VQ-VAE encodes the input geometry into tokens; the MLLM predicts which tokens belong to which part; the decoder reconstructs the parts from those tokens; and a graph-based surface segmentation (Section 3.4) maps the reconstructed parts back onto the original high-fidelity mesh surface. This "grounding on input" approach means SIMART preserves the original mesh's geometric detail, unlike Articulate-Anything and PhysX-Anything which generate simplified proxy geometry that only loosely matches the input. Table 1 quantifies this advantage: SIMART achieves 0.690 IoU and 0.087 CD on in-domain items, while Articulate-Anything achieves only 0.202 IoU and 0.239 CD — the latter's generated geometry is substantially different from the input.

The Data Problem and SIMART-Bench

A subtle but important motivation addressed in the paper is evaluation validity. Most prior methods evaluate primarily on PartNet-Mobility, which the paper argues has significant limitations (Section 4):

"While PartNet-Mobility is a standard resource, its data distribution is relatively homogeneous, with minimal geometric variance within categories."

Training and testing on PartNet-Mobility alone conflates geometric memorization with genuine reasoning capability. A method might achieve high accuracy by learning to recognize the specific cabinet handles and drawer configurations present in PartNet-Mobility's training set, then fail on an AI-generated cabinet with a novel handle geometry or unusual proportions. The introduction of SIMART-Bench — a benchmark that includes AIGC-generated objects with diverse topologies not present in PartNet-Mobility — is the paper's response to this evaluation gap. The performance gap between ID and AI-generated items in Table 1 (e.g., 0.928 vs. 0.831 accuracy for SIMART, or 0.822 vs. 0.817 for Particulate) validates that AI-generated objects represent a genuinely harder generalization challenge.

How Unsupervised Prior Work Creates a Unique Combination

This problem sits at the intersection of three fields that rarely meet: 3D geometric understanding (from computer vision / graphics), kinematic reasoning (from robotics / physics simulation), and MLLM architectures (from NLP / multimodal AI). Prior work typically addresses at most two of these:

• Part segmentation methods (PartField, P3SAM) handle geometry but not kinematics.
• Articulation estimation methods (CAGE, SINGAPO) handle kinematics but operate on rendered images, not raw 3D geometry.
• MLLM methods (Articulate-Anything, PhysX-Anything) handle physical reasoning but are constrained to 2D inputs or fail on complex 3D due to tokenization.

SIMART is the first system to integrate all three — true 3D geometry processing, physics-grounded kinematic reasoning via an MLLM, and efficient tokenization that makes the integration computationally feasible — into a single unified forward pass. This is what distinguishes it as a framework-level contribution rather than a point improvement on any single axis.

3. Technical Approach

3.1 Reader Orientation

SIMART is a unified neural architecture that takes a static 3D mesh as input and outputs a fully functional simulation-ready articulated asset — meaning it simultaneously segments the mesh into separate moving parts and predicts the kinematic structure (joint types, axes, origins, limits) that governs how those parts move relative to each other. The system solves the problem of transforming monolithic, non-interactive 3D models into physically meaningful assemblies by combining three capabilities in a single forward pass: a sparse 3D tokenizer that makes complex geometry processable by a language model, a multimodal LLM backbone that reasons about physical structure using pretrained world knowledge, and a structured output format that maps directly to the URDF specification used by physics simulators.

3.2 Big-Picture Architecture

The SIMART pipeline consists of five major components connected in sequence:

1. Sparse 3D VQ-VAE (tokenizer and decoder): Encodes the input voxelized mesh into a compact set of discrete geometric tokens, discarding empty space. After the MLLM processes these tokens, the decoder reconstructs part-specific voxel grids from the generated token sequences.

2. Vision Encoder (ViT): Processes a rendered 2D image of the object (252×252 pixels, 45-degree isometric view) to extract semantic context features that complement the geometric tokens.

3. Qwen3-VL-8B MLLM Backbone: The central reasoning engine that consumes the concatenated sequence of [geometric tokens + vision tokens + text instruction tokens] and generates two parallel outputs: per-part voxel token sequences and a structured JSON description of the kinematic hierarchy and physical properties.

4. Graph-Based Surface Segmentation: Maps the decoded part-specific voxel seeds back onto the high-fidelity input mesh surface using Gaussian kernel proximity and graph-smoothing to produce clean part boundaries along the original geometry.

5. URDF Assembly Module: Combines the segmented sub-meshes with the MLLM-generated kinematic metadata (joint types, axes, origins, limits, material properties) into a complete URDF specification loadable by physics simulators.

Information flows as follows: an input mesh enters the system → the Sparse VQ-VAE encoder converts it to sparse geometric tokens representing only occupied voxels → these tokens are concatenated with vision features from the rendered image and a task-specific text instruction → the MLLM processes the full multimodal sequence and autoregressively generates [per-part voxel tokens + structured URDF text] → the Sparse VQ-VAE decoder reconstructs each part's voxel grid → the graph-based segmentation maps these coarse part assignments onto the original high-resolution mesh → the URDF specification and segmented meshes are combined into a simulation-ready asset.

3.3 Roadmap for the Deep Dive

• First, the Sparse 3D VQ-VAE — the tokenizer and decoder that is the primary technical enabler. Understanding its architecture, the zero-token mechanism, and why it achieves 70% token reduction is prerequisite to everything else, since all geometric processing flows through this component.
• Second, the unified MLLM architecture — how the Qwen3-VL backbone ingests the multimodal sequence (geometry + vision + text), what each modality contributes, and how the model is trained to generate both geometric tokens and structured URDF text jointly.
• Third, the output format and training data generation — what SIMART produces (the exact structure of per-part voxel tokens and URDF JSON), how training data is constructed from PartNet-Mobility and PhysXNet (including the 20-state augmentation strategy), and the specific prompt templates used during instruction tuning.
• Fourth, the graph-based surface segmentation — how the coarse decoded voxel grids are mapped onto the high-fidelity input mesh to produce clean, watertight part boundaries while preserving the original surface detail.
• Fifth, the URDF assembly process — how the structured JSON output is converted into a physically valid kinematic specification with proper parent-child hierarchies, joint limits, and material properties for simulation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems paper whose core idea is that a unified MLLM can jointly perform articulation-aware part decomposition and kinematic parameter prediction by processing 3D geometry through an efficient sparse tokenization that makes per-part generation computationally feasible. The key insight is that the tokenization bottleneck — not the MLLM's reasoning capability — was the limiting factor preventing prior 3D-native MLLMs from handling articulated objects, and that explicitly encoding spatial coordinates alongside geometric tokens enables fine-grained joint axis and origin prediction.

Sparse 3D VQ-VAE Architecture and Training

The Sparse 3D VQ-VAE is the fundamental enabling component of SIMART. Without it, the entire pipeline would be impossible: dense voxel representations of multi-part objects exceed the MLLM's memory and context constraints at any useful resolution. This component serves dual roles — as an encoder that converts input geometry into tokens for the MLLM to process, and as a decoder that converts the MLLM's generated token sequences back into 3D geometry.

Input preprocessing and voxelization. The raw input mesh $G_{geo}$ is first discretized into a fixed-resolution voxel grid of dimensions $64 \times 64 \times 64$. Each voxel is labeled as either occupied (contains some portion of the mesh surface or interior) or unoccupied (empty space). This discretization is necessary because the 3D-UNet encoder expects a regular grid input; the raw mesh's continuous vertex positions and triangular faces cannot be directly consumed.

3D-UNet encoder architecture. The encoder is a 3D convolutional U-Net that progressively downsamples the $64 \times 64 \times 64$ binary occupancy grid through multiple spatial reduction stages. The paper describes this architecture by analogy to ShapeLLM-Omni (Ye et al., 2025) and TRELLIS (Xiang et al., 2025), from which the VAE weights are initialized. The encoder first maps the $64^3$ grid into an intermediate latent grid of dimensions $16 \times 16 \times 16 \times C$ (where $C$ is the feature dimension after this initial encoding). To further compress the representation for MLLM consumption, the model aggregates every eight neighboring tokens along the channel dimension, resulting in a final latent grid of $8 \times 8 \times 8 \times 64$. This means each spatial position in the $8 \times 8 \times 8$ grid holds a 64-dimensional feature vector representing the local geometry within an $8 \times 8 \times 8$ block of the original $64^3$ space.

The paper explicitly notes this resolution choice as a deliberate tradeoff (Appendix B):

"While expanding the latent resolution to $16 \times 8 \times 8$ provides further gains in geometric fidelity, such a configuration doubles the resulting token sequence length, which introduces a substantial memory overhead for the multimodal backbone. As a critical trade-off to ensure efficient fine-tuning without memory exhaustion, the $8 \times 8 \times 8$ latent grid is implemented."

The reconstruction quality at this resolution is empirically validated: the $8 \times 8 \times 8$ configuration achieves a Mean Squared Error of $1.84 \times 10^{-5}$ and a Chamfer Distance of $4.19 \times 10^{-5}$ when comparing decoded voxel grids against ground-truth voxelized inputs (Table 4, Appendix B).

Vector quantization and the codebook. After the encoder produces the $8 \times 8 \times 8 \times 64$ latent grid, each 64-dimensional feature vector at each spatial position must be mapped to a discrete token for the MLLM to process. This is done through vector quantization (VQ): the model maintains a learned codebook $\mathcal{C}$ containing 4,096 entries, each being a 64-dimensional embedding vector. For any latent feature $z_i$ at spatial index $i$, the standard VQ operation finds the nearest codebook entry:

$\hat{z}_i = \arg \min_{\mathbf{e}_j \in \mathcal{C}} \|z_i - \mathbf{e}_j\|_2$

where $\hat{z}_i$ is the quantized vector (the codebook entry that best approximates the encoder output), $z_i$ is the raw encoder output at position $i$, and $\mathbf{e}_j$ is the $j$-th entry in codebook $\mathcal{C}$.

What it computes: for each spatial position in the $8 \times 8 \times 8$ latent grid, this equation selects the single codebook vector (out of 4,096 candidates) that is closest in Euclidean distance to the encoder's output. The index $j$ of that selected codebook entry becomes the discrete geometric token that represents that voxel in the MLLM's input sequence. The quantized vectors $\hat{z}_i$ are what the decoder later uses to reconstruct geometry.

Why this form: vector quantization with a discrete codebook transforms a continuous latent space into a discrete token space, making it compatible with autoregressive token-based models like Qwen3-VL. The Euclidean distance ($\ell_2$ norm) is the standard metric for VQ because it corresponds to maximum-likelihood decoding under a Gaussian assumption on the encoder's output distribution around each codebook vector. Using a discrete codebook rather than continuous latents forces the model to learn a compressed, reusable vocabulary of local geometric patterns, analogous to how text tokenizers learn a vocabulary of subword units.

The codebook size of 4,096 was chosen empirically. The paper reports (Appendix B) that doubling to 8,192 entries "does not yield significant improvements in the reconstruction metrics, indicating that a 4,096-entry codebook is sufficient when utilizing the zero token to summarize unoccupied regions."

The zero-token mechanism — the critical innovation. Standard VQ-VAEs process every spatial position identically, including positions corresponding to empty space. For a $64^3$ input grid, even the $8 \times 8 \times 8 = 512$ latent positions mostly represent empty space for typical objects (a cabinet might occupy only 20-30% of its bounding volume). SIMART's key insight is that these empty-space tokens are not just redundant — they actively harm performance by diluting the MLLM's attention over irrelevant positions.

The zero-token mechanism works as follows. During encoding, the system identifies which voxels in the original $64^3$ grid are unoccupied. Any latent position in the $8 \times 8 \times 8$ grid whose corresponding $8 \times 8 \times 8$ block in the input space contains no occupied voxels is classified as unoccupied. For these positions, the encoder output is discarded and the 0-th codebook entry (designated as $\mathbf{e}_{zero}$) is assigned directly, bypassing the nearest-neighbor search:

$\hat{z}_i = \begin{cases} \mathbf{e}_{zero}, & \text{if Voxel } i \text{ is unoccupied} \\ \arg \min_{\mathbf{e}_j \in \mathcal{C} \setminus \{\mathbf{e}_{zero}\}} \|z_i - \mathbf{e}_j\|_2, & \text{otherwise} \end{cases}$

where $\mathbf{e}_{zero}$ is the reserved zero-token embedding, and $\mathcal{C} \setminus \{\mathbf{e}_{zero}\}$ means the codebook minus the zero token (i.e., only codebook entries 1 through 4,095 are candidates for occupied voxels).

What it computes: a conditional branching at each spatial position. If the position corresponds to empty space in the original input, it receives a fixed, learned "empty" token without any computation. If the position contains geometry, it undergoes the standard nearest-neighbor lookup against the remaining 4,095 geometric codebook entries. This produces a variable-length sequence containing only the tokens for occupied positions — typically 70% fewer tokens than the full $8 \times 8 \times 8 = 512$ positions would yield.

Why this form: the conditional bypass of empty voxels is what achieves the 70% token reduction. But the deeper justification involves an observation about VQ-VAE training dynamics described in Appendix B: during training of a dense VQ-VAE baseline, the authors observed that even without explicit zero-token supervision, "approximately two to four codebook entries consistently converge to represent the null distribution of empty voxels. These entries, when processed by the decoder, effectively reconstruct empty volumetric regions with high stability." The zero-token mechanism formalizes this emergent behavior by explicitly reserving one codebook entry for empty space, which provides two benefits: (1) it guarantees that all geometric codebook entries (1–4,095) are exclusively used for actual geometry, making better use of the limited codebook capacity, and (2) it prevents the MLLM from "wasting attention on irrelevant background tokens, thereby focusing its reasoning capacity on the functional and articulated parts of the 3D asset" (Appendix B).

Coordinate-aware tokenization. Because the sparse tokens no longer form a fixed grid with implicit spatial positions (unlike dense representations where the $i$-th token always corresponds to a known $(x, y, z)$ coordinate), each occupied voxel must explicitly encode its spatial location. The paper introduces a serialization format where each occupied voxel becomes exactly three atomic tokens:

⟨voxel⟩ [xyz] [K]

The components are:

• ⟨voxel⟩: a special start-of-voxel identifier token that signals the beginning of a voxel triplet and helps the MLLM parse the geometric sequence structure.

• [xyz]: a single integer token encoding the spatial coordinates using a linearized index mapping: $xyz = 64x + 8y + z$, where $x, y, z \in [0, 7]$. This produces a value in the range $[0, 1023]$ that uniquely identifies the $(x, y, z)$ position within the $8 \times 8 \times 8$ latent grid. The paper's system prompt (Table 5) specifies the coordinate system convention: "x: left → right (0–15); y: front → back (0–7); z: ground → top (0–7)." However, the actual latent grid is $8 \times 8 \times 8$, not $16 \times 8 \times 8$, which creates an apparent inconsistency. This is clarified by the system prompt's statement that the space is represented as a $16 \times 8 \times 8$ voxel grid but the Sparse VQ-VAE uses an $8 \times 8 \times 8$ latent grid — the final model employs the $8 \times 8 \times 8$ configuration as confirmed by the ablation in Table 4. The mapping formula $64x + 8y + z$ with $x, y, z \in [0, 7]$ produces indices in $[0, 511]$ for the $8 \times 8 \times 8$ grid; the range $[0, 1023]$ mentioned in the system prompt corresponds to the $16 \times 8 \times 8$ grid that was evaluated but not used in the final model due to memory constraints.

• [K]: a single integer token representing the codebook index $K \in [0, 4095]$ (or $[0, 8191]$ as stated in the system prompt for the larger grid configuration). This is the discrete token that encodes the actual geometric content — the local surface shape — at that spatial position.

This coordinate-aware format means the MLLM sees geometry as an explicit sequence of (position, shape-token) pairs, enabling it to reason about spatial relationships directly from the token stream. For example, it can learn that tokens with similar $y$ coordinates and high $x$ indices might form a vertical edge — the type of reasoning needed to localize hinge axes.

Sparse 3D VQ-VAE decoder. The decoder is a symmetric 3D-UNet architecture that performs the inverse operation. Given a set of discrete codebook indices and their spatial coordinates, the decoder:

1. Reconstructs a complete $8 \times 8 \times 8 \times 64$ latent grid by placing each quantized vector $\hat{z}_i$ at its specified $(x, y, z)$ position, and filling all unoccupied positions with the $\mathbf{e}_{zero}$ embedding decoded through the zero-token pathway.

2. Upsamples through a series of 3D transposed convolutions and skip connections (mirroring the encoder's downsampling path) to produce a full $64 \times 64 \times 64$ occupancy grid.

3. The output is a binary or continuous occupancy prediction for each of the $64^3$ voxels, which can be converted to a mesh surface via standard marching cubes or similar isosurface extraction.

The decoder is trained jointly with the encoder using the standard VQ-VAE loss formulation (Appendix B, Equation 3):

$\mathcal{L}_{total} = \mathcal{L}_{rec}(G_{geo}, \hat{G}_{geo}) + \|sg[E(G_{geo})] - \hat{z}\|_2^2 + \beta \|E(G_{geo}) - sg[\hat{z}]\|_2^2$

where $G_{geo}$ is the original voxelized mesh, $\hat{G}_{geo}$ is the reconstructed occupancy grid, $E(G_{geo})$ is the encoder output before quantization, $\hat{z}$ is the quantized vector after nearest-neighbor lookup, $sg[\cdot]$ is the stop-gradient operator (which treats its argument as a constant during backpropagation), and $\beta$ is a weighting hyperparameter.

What it computes: the sum of three loss terms. The first term $\mathcal{L}_{rec}$ is a binary cross-entropy reconstruction loss comparing the original and reconstructed occupancy grids voxel-by-voxel — this ensures the VQ-VAE faithfully encodes and decodes geometry. The second term $\|sg[E(G_{geo})] - \hat{z}\|_2^2$ is the codebook loss (also called the VQ commitment loss) — it pulls the codebook vectors toward the encoder outputs that quantize to them, but stops gradient on the encoder side so the encoder doesn't move toward the codebook. The third term $\beta \|E(G_{geo}) - sg[\hat{z}]\|_2^2$ is the commitment loss — it pulls the encoder outputs toward their assigned codebook vectors, with the stop-gradient on the codebook side and scaled by $\beta$ (typically 0.25 in VQ-VAE implementations, though the paper does not specify the exact value).

Why this form: the three-term decomposition is the standard VQ-VAE objective introduced by van den Oord et al. (2017). The reconstruction loss $\mathcal{L}_{rec}$ is the primary task objective. The codebook loss ensures the codebook entries are actually used and updated to represent the data distribution. The commitment loss prevents the encoder outputs from growing unboundedly (since the quantization step is a discrete bottleneck that could otherwise be circumvented by making the encoder outputs very large). The $\beta$ weight controls the tradeoff between codebook utilization and reconstruction fidelity — too large and the encoder over-commits to codebook vectors, limiting expressivity; too small and the encoder outputs drift away from the codebook, making the quantization meaningless.

Pretraining and implementation details. The Sparse 3D VQ-VAE is pretrained on a 500,000-object subset following the TRELLIS data distribution (Appendix B). Model weights are initialized from the TRELLIS VAE checkpoint, providing a strong starting point since TRELLIS already learned high-quality 3D shape features. The pretraining is conducted using 8 NVIDIA A100 GPUs for 60,000 steps in each of two stages (for a total of 120,000 steps), though the paper does not specify what distinguishes the two stages (likely an initial training with a higher reconstruction weight followed by fine-tuning with a higher commitment loss weight, typical for VQ-VAE training).

The reconstruction quality achieved by the final configuration is reported in Table 4 of Appendix B: MSE of $1.84 \times 10^{-5}$ and Chamfer Distance of $4.19 \times 10^{-5}$ (both scaled by $10^5$ for readability in the table). For context, the force-sparse variant (which discards empty-space tokens without using the specialized zero token) achieves substantially worse reconstruction: CD of $56.10 \times 10^{-5}$ compared to $4.19 \times 10^{-5}$ — over 13× worse — demonstrating that the zero-token is not merely a compression technique but essential for reconstruction quality. The decoder needs the explicit empty-space token to know what empty regions should look like; without it, the unoccupied positions are simply missing from the latent representation, and the decoder must hallucinate them from incomplete information.

Unified MLLM Architecture

The MLLM component is where all modalities converge and where the actual reasoning about articulation happens. It is built on Qwen3-VL-8B (Bai et al., 2025), chosen for its "powerful large-scale image-text pre-training and its emergent capability for physical world understanding" (Section 3.2). The key architectural decision is that SIMART does not build a custom 3D-specific model from scratch; instead, it extends a general-purpose multimodal LLM to process 3D geometry as an additional modality alongside text and images.

Why Qwen3-VL specifically. The paper's choice of backbone is strategic. Qwen3-VL was pretrained on a massive corpus of interleaved image-text data, giving it "emergent capability for physical world understanding" — it has seen enough images of cabinets, drawers, doors, and articulated mechanisms to learn conceptual priors about how objects move. A model pretrained only on text or on unrelated visual domains would lack this physical intuition. The 8B parameter scale balances capability with training efficiency: larger models (e.g., Qwen3-VL-235B used for the P3SAM baseline in Table 2) might provide better reasoning but would be prohibitively expensive to fine-tune on 960K instruction-following examples.

Multimodal input sequence construction. The input to the MLLM is a concatenated sequence of three token types, mapped into a unified latent space before entering the Transformer layers:

1. Geometric tokens ($F_{geo}$): The sparse voxel tokens from the Sparse VQ-VAE, serialized in the ⟨voxel⟩ [xyz] [K] format described above. Each triplet of atomic tokens represents one occupied voxel. The total number of geometric tokens depends on the object's complexity — a simple box might have ~100 occupied voxels (300 atomic tokens), while a complex articulated assembly might have ~170 occupied voxels (510 atomic tokens). The paper reports an average of 516 geometric tokens per object across the training dataset (Table 3, "Zero Sparse + Vision" configuration). These tokens are embedded through a learned embedding layer that maps each discrete codebook index $K$ to a dense vector of dimension $D$ (matching the MLLM's hidden dimension).

2. Vision tokens ($F_{vis}$): A single RGB image $I_{vis}$ rendered from a 45-degree isometric perspective at 252×252 pixel resolution. The paper explains this choice: "To resolve scale ambiguities and provide global context, each object is rendered as a 252x252 pixel image from a 45-degree isometric perspective" (Appendix C). The image is processed through a Vision Transformer (ViT) encoder — specifically the ViT integrated into Qwen3-VL — which produces a sequence of visual feature vectors. The isometric viewpoint is chosen because it simultaneously shows three faces of the object, providing maximum geometric information from a single view without the ambiguity of a canonical front-facing render. The 252×252 resolution balances information content with sequence length (processing at native ViT resolutions like 336×336 or 384×384 would produce more tokens without proportional benefit for this task).

   The vision tokens serve a crucial complementary role to the geometric tokens. The geometric tokens provide precise spatial occupancy but are low-resolution (8×8×8 latent grid) and abstract (discrete codebook indices). The image provides high-resolution texture, fine geometric detail (edges, corners, surface curvature), and semantic context (material appearance, object category cues) that the sparse voxels cannot capture. The ablation in Table 3 quantifies this contribution: adding vision improves joint type accuracy from 0.794 to 0.937 and halves the Chamfer Distance from 0.074 to 0.055.

3. Text instruction tokens ($F_{txt}$): A natural language prompt that specifies the task. The paper uses two primary templates:

   • For URDF generation: "Describe the object with real scale and separate the object to different functional parts with each physical properties."
   • For part grounding: "Generate the part of this object with description: [text]" where [text] is replaced with a functional description like "the drawer that slides open" or "the hinged lid."

   These instruction tokens are embedded using Qwen3-VL's standard text tokenizer and embedding layer. Importantly, the instructions are not just task specifiers — they modulate the generation by priming the MLLM to produce the appropriate output format (voxel sequences for part geometry vs. JSON for kinematic parameters).

The final input sequence has length $L = N_v + N_g + N_t$, where $N_v$ is the number of vision tokens (depends on ViT patch size; for a 252×252 image with standard 14×14 patches, this would be approximately $(252/14)^2 = 324$ tokens plus a CLS token), $N_g$ is the number of geometric tokens (approximately 516 on average per Table 3), and $N_t$ is the number of text instruction tokens (typically 20–50 tokens for the prompts described above).

What happens inside the MLLM. The concatenated multimodal token sequence is processed by Qwen3-VL's standard Transformer decoder architecture with causal self-attention. All token types — geometric, visual, textual — attend to each other across modality boundaries. This cross-modal attention is what enables the integrated reasoning that distinguishes SIMART from multi-stage pipelines:

• The geometric tokens for a particular spatial region can attend to visual features from the rendered image that show what that region looks like — for instance, a cluster of voxels near the edge of a box can attend to the image patch showing a visible seam, confirming that this is a part boundary rather than an arbitrary geometric feature.

• The text instruction tokens can attend to both geometric and visual features to ground functional language: the phrase "the drawer that slides open" can trigger attention patterns that focus on voxels with specific geometric signatures (planar region, orthogonal to a sliding axis, near the bottom of the object) and visual cues (handle appearance, gap lines).

• The URDF generation tokens can attend to the per-part voxel tokens to ensure consistency: the joint axis predicted in the JSON output should align with the geometric relationship between the parts whose voxels were generated in the same forward pass.

The autoregressive generation proceeds left-to-right: the model first generates the structured JSON text describing the kinematic hierarchy and physical properties, then generates the per-part voxel token sequences. This ordering is important because the JSON provides a scaffold — it declares how many parts there are, which parts connect to which, and what joint types connect them — before the voxel sequences fill in the geometric content of each part.

Training procedure and data. The MLLM is fine-tuned (not trained from scratch) for 30,000 steps on a cluster of 32 NVIDIA A100 GPUs. The training data comprises two large instruction-following datasets synthesized from PhysXNet and PartNet-Mobility:

• Source dataset: 39,600 3D objects total — 5,600 articulated models from PartNet-Mobility and PhysXNet, plus 34,000 static objects "intended to improve general shape comprehension" (Section 4). The inclusion of static objects is a deliberate curriculum design choice: seeing non-articulated objects teaches the model the general relationship between geometry tokens and object structure, preventing it from hallucinating articulations on every object it sees.

• Data augmentation via kinematic states: Each articulated model is rendered in 20 diverse kinematic states — e.g., a cabinet with the door closed, partially open, fully open; a drawer pulled out 0%, 25%, 50%, 75%, 100% — "effectively treating each state as an individual training instance" (Section 4). This multiplies the effective articulated training examples to $5,600 \times 20 = 112,000$ unique geometric configurations. The augmentation is critical because it teaches the model to recognize articulation irrespective of the specific state shown: a door is still a door whether it is open or closed, and the model must learn that the same voxel cluster can represent the same part in different positions.

• Two instruction-following datasets: From these augmented assets, the authors synthesize:

  • A URDF generation set with 960K QA pairs — each pair consists of the multimodal input (geometry + vision + task prompt) and the target output (JSON kinematic specification + per-part voxel tokens).
  • A part grounding set with 960K QA pairs — each pair asks the model to identify and output the voxel tokens corresponding to a specific functional part given a text description.

  The total of 1.92M QA pairs is substantial for 3D tasks, which typically suffer from data scarcity compared to text or 2D vision domains.

• Prompt template: A structured system prompt (Table 5, Appendix C) defines the task format, coordinate system, output constraints, and strict formatting rules. The prompt specifies the coordinate system conventions ("x: left → right, y: front → back, z: ground → top"), the output JSON structure, and critical constraints like "Output voxels must be a subset of the input" and "No extra text, headers, or formatting beyond the JSON."

The fine-tuning uses standard autoregressive language modeling loss (cross-entropy) on the target token sequences. The model learns to predict both the textual JSON tokens and the discrete voxel codebook indices — the geometric tokens are treated identically to text tokens for the purpose of the loss function. The key hyperparameter (the learning rate) is not explicitly stated for the MLLM training in the main text; the 3D VQ-VAE uses a two-stage procedure but the MLLM learning rate is not quoted. The paper notes that training is conducted on 32 NVIDIA A100 GPUs for 30,000 steps, which at typical batch sizes for 8B-parameter models (global batch size of 128–256 sequences) would process approximately 3.8–7.7 million examples — several epochs over the 1.92M-pair dataset.

Output Format and Structured Generation

SIMART generates a hybrid output that combines geometric content (per-part voxel tokens) with symbolic structure (JSON metadata). The output format is carefully designed to be directly consumable by downstream physics simulators without post-processing beyond the VQ-VAE decoding and surface segmentation steps.

The two-task output structure. The MLLM handles two distinct but related output modes depending on the task:

1. URDF generation task: The model outputs a complete JSON object with three top-level fields, illustrated in Table 6:

   • "object_captions": Object-level metadata including the object's name (e.g., "Storage Box with Frame") and a global scale factor. The scale is a single float representing the object's real-world size in centimeters — the MLLM estimates this from its world knowledge (a typical storage box is ~40 cm wide, not 4 cm or 400 cm). This is crucial for simulation because physical properties like mass, inertia, and joint stiffness scale with absolute size.
   • "parts_captions": A dictionary mapping part indices to kinematic and physical specifications. Each entry contains:
     • "type": The joint type — one of "fixed" (rigidly attached, no motion), "revolute" (rotational hinge), "prismatic" (linear sliding), "free" (unconstrained 6-DOF), or "rigid". The paper's system prompt lists these options and the MLLM selects the appropriate one based on its physical understanding of the object.
     • "parent": The index of the parent part in the kinematic chain (omitted for the root/base part, which is typically "fixed").
     • "center": The joint origin as integer coordinates $[x, y, z]$ in the range $[0, 200]$, representing grid points at 0.005 resolution (i.e., each integer unit corresponds to 0.005 × scale in real-world units). The range $[0, 200]$ was chosen to span the $64^3$ voxel grid mapped onto a normalized coordinate frame.
     • "axis": The joint axis as a direction vector $[dx, dy, dz]$ with integer components in $[0, 100]$. For a revolute joint, this is the rotation axis; for a prismatic joint, this is the sliding direction.
     • "limits": The joint limits as a range $[-val, val]$ where "100 represents 180° for revolute or max distance for prismatic" (Table 5). This normalized encoding means the MLLM outputs values on a consistent scale regardless of joint type.
     • "material": A semantic material string (e.g., "Plastic", "Metal", "Wood") — the MLLM infers this from visual appearance in the rendered image.
     • "density": Material density in g/cm³ — a real-valued physical property needed for mass/inertia computation.
     • "Young's Modulus (GPa)": Optional elastic modulus for deformable body simulation.
   • "parts_voxels": For each part index, a string containing the serialized sparse voxel tokens in the format "<voxel> 0 1785 <voxel> 1 649 ...". The tokens here are triplets of ⟨voxel⟩ [xyz] [K] as described in the tokenization scheme. Critically, the system prompt enforces: "Output voxels must be a subset of the input" — the model cannot hallucinate geometry that wasn't in the original mesh; it can only assign existing occupied voxels to parts.

2. Part grounding task: The model outputs only the ⟨voxel⟩ [xyz] [K] token sequence corresponding to the specific part described in the query. This is a simpler output — no JSON, no physical properties — just the geometric tokens that constitute the requested functional component.

Why JSON + tokens rather than a single unified format. The hybrid output design reflects a deliberate separation of concerns. The JSON captures symbolic, structured knowledge — joint types, spatial directions as integer vectors, physical properties as named fields — that benefits from the MLLM's text generation capabilities and can be parsed deterministically. The voxel tokens capture dense geometric information — 3D occupancy patterns — that would be impossibly verbose if described in natural language (imagine describing a 3D shape as text coordinates) but is naturally expressed in the discrete token vocabulary learned by the VQ-VAE. Generating both in the same autoregressive sequence allows the model to condition the geometric assignment on the kinematic reasoning: the model decides "this is a revolute joint between parts 0 and 1 with axis along x" before or while generating the voxel tokens, so it can ensure the voxels assigned to part 1 form a geometry consistent with rotation around the x-axis.

Training data construction for the output format. The 960K QA pairs for each task were synthesized automatically from the PartNet-Mobility and PhysXNet datasets. The paper does not provide exhaustive detail on the synthesis procedure, but the approach is inferrable: for each articulated model in the dataset, the ground-truth part segmentation and URDF parameters are already annotated (these datasets provide per-part meshes and joint specifications). The synthesis involves (1) rendering the model in each of the 20 kinematic states, (2) encoding each state through the Sparse VQ-VAE to obtain geometric tokens, (3) constructing the JSON output by extracting the known joint types, axes, and limits from the dataset annotations, and (4) pairing the multimodal input with the formatted output to create supervised training examples.

Scale estimation. A notable capability is the MLLM's estimation of real-world scale. The system prompt asks the model to "predict real-world scale" (Table 5), and the output JSON includes a "scale" field. This is not learned from the geometry alone — a cabinet mesh at 100× scale looks identical to a cabinet mesh at 1× scale in the normalized voxel grid — but rather from the MLLM's pretraining knowledge that "cabinets are roughly human-scale, around 1–2 meters tall." The isometric rendered image provides additional scale cues (texture details, relative proportions) that the MLLM can leverage from its vision-language training.

Graph-Based Surface Segmentation

The MLLM generates per-part voxel tokens at the coarse $8 \times 8 \times 8$ latent resolution. The Sparse VQ-VAE decoder reconstructs these into $64 \times 64 \times 64$ occupancy grids for each part. However, the decoded grids are still at a relatively low resolution and may contain quantization artifacts. The final step is mapping these coarse part assignments onto the original high-fidelity input mesh to produce clean, watertight part boundaries that preserve the input's vertex-level detail.

Why this step is necessary. The MLLM and VQ-VAE operate in a compressed latent space optimized for computational efficiency, not geometric precision. The $64^3$ decoded voxel grid has a resolution of approximately 1–2 cm per voxel for a typical household object — enough to capture the gross structure ("this region is the drawer") but insufficient for precise boundaries ("this exact vertex is the edge between the drawer face and the cabinet body"). The graph-based segmentation bridges this resolution gap by using the decoded voxels as soft guidance seeds rather than hard assignments.

Step 1: Seed point cloud extraction. For each part $p$, the decoded $64^3$ occupancy grid is thresholded and converted to a sparse point cloud $S_p$ — a set of 3D points representing the centers of occupied voxels assigned to that part. These point clouds are sparse and low-resolution but capture the rough spatial extent of each functional component.

Step 2: Vertex-wise probability initialization. For each vertex $v$ on the high-resolution input mesh surface, a probability distribution over parts is initialized using a Gaussian kernel centered on the seed points:

$P(v, p) \propto \exp \left( -\frac{d(v, S_p)^2}{2\sigma^2} \right)$

where $d(v, S_p)$ is the Euclidean distance from vertex $v$ to the nearest seed point in the point cloud $S_p$ of part $p$, and $\sigma$ is a scale hyperparameter set relative to the mesh bounding box diagonal.

What it computes: for each mesh vertex and each part, a soft assignment probability based on proximity. Vertices close to the seeds of part $p$ receive high probability for that part; vertices far from all seeds of part $p$ receive low probability. The Gaussian kernel decays smoothly with distance, so boundaries between parts are initially fuzzy rather than hard cuts. The $\sigma$ parameter controls how sharply the probability drops off — a smaller $\sigma$ means assignments are more localized around seed points, while a larger $\sigma$ means seeds influence a broader region.

Why this form: the Gaussian kernel is chosen for its smoothness and locality. The smoothness means there are no discontinuous jumps in assignment probability across the mesh surface, which would create jagged part boundaries. The locality (exponential decay with squared distance) means that each vertex is primarily influenced by the nearest seeds — a vertex in the middle of the cabinet door will be close to door seeds and far from drawer seeds, so it gets a clear assignment. An alternative like uniform assignment to the nearest seed would create Voronoi-like boundaries that are sensitive to individual seed placement noise. The Gaussian kernel with a finite $\sigma$ effectively "blurs" the seed influence, making the assignment more robust to the coarse resolution of the decoded voxel grids.

Step 3: Iterative graph smoothing. The initial vertex probabilities are refined by applying an iterative smoothing operator over the mesh's adjacency graph. At each iteration, each vertex's probability distribution is updated to be a weighted combination of its own probability and the probabilities of its immediate neighbors (vertices connected by mesh edges). The paper does not specify the exact smoothing formulation or number of iterations, but the standard approach (Laplacian smoothing on the mesh graph) would be:

$P^{(t+1)}(v, p) = \alpha P^{(t)}(v, p) + (1-\alpha) \frac{1}{|\mathcal{N}(v)|} \sum_{u \in \mathcal{N}(v)} P^{(t)}(u, p)$

where $\mathcal{N}(v)$ is the set of neighboring vertices of $v$, $\alpha \in [0, 1]$ is a smoothing weight, and $t$ indexes the iteration.

The purpose of this smoothing is to enforce spatial coherence: if a vertex is geometrically adjacent to vertices assigned to the cabinet body, it is likely also part of the cabinet body, even if its initial seed-based probability was ambiguous. The smoothing propagates label information along the mesh surface, filling in gaps and resolving boundary ambiguities.

Step 4: Face-level majority voting. After smoothing, the final part assignment is determined at the face (triangle) level by majority voting over each face's three vertices. For a face with vertices $v_1, v_2, v_3$, the assigned part is:

$\mathcal{M}_{seg}(\text{face}) = \arg\max_p \sum_{i=1}^{3} P(v_i, p)$

This produces a hard segmentation where each triangular face is assigned to exactly one part. Since adjacent faces share vertices, the boundary between parts naturally forms along mesh edges where the vertex probabilities transition from one dominant part to another — producing clean, connected boundaries without gaps or overlaps.

Why graph-based over volumetric or point-based segmentation. The graph-based approach operates directly on the input mesh's connectivity structure (vertices and edges), which has two important advantages: (1) it preserves the original mesh topology — the output segmented meshes are subsets of the input mesh, so they inherit the same vertex positions, face normals, and texture coordinates, and (2) it ensures watertight boundaries — since adjacent faces share edges and vertices, there are no gaps between parts after segmentation. A volumetric approach operating on the voxel grid directly would produce stair-step artifacts at boundaries and could not recover the input mesh's original surface detail. A point-cloud approach operating on sampled surface points would require expensive point-to-mesh registration and could leave holes.

Texture preservation. As a final step, the original mesh's texture (UV coordinates and texture maps) is preserved: "the original texture of the input mesh is preserved and adopted as the final texture output" (Section 3.4). This is possible because the segmentation operates on the mesh faces directly — each face retains its original material and texture assignment regardless of which part it is assigned to. This is a significant practical advantage over generative baselines (Articulate-Anything, PhysX-Anything) that produce simplified, untextured proxy geometry.

URDF Assembly and Simulation-Ready Output

The final stage integrates the segmented sub-meshes $\mathcal{M}_{seg} = \{m_1, m_2, \dots, m_n\}$ with the structured kinematic metadata $\mathcal{P}_{sim}$ from the MLLM's JSON output to produce a complete simulation-ready asset. This is not a learned component but a deterministic assembly process that constructs a URDF (Unified Robot Description Format) specification.

Kinematic chain construction. The JSON output includes a "parent" field for each non-root part, specifying which part it attaches to. This defines a tree-structured kinematic hierarchy (no kinematic loops — standard for URDF). The base part (typically "type": "fixed", representing the object's stationary frame, like the cabinet body) is the root. Each child part inherits its parent's coordinate frame and applies its joint transform (specified by "center", "axis", and "limits") to define its own frame.

For example, in the storage box example from Table 6:

• Part 0 is "fixed" — the box frame, which serves as the kinematic root.
• Part 1 has "parent": "0", "type": "revolute", "center": [100, 138, 101], "axis": [100, 0, 0], "limits": [-54, 45] — this defines a lid that rotates around an axis passing through the point (100, 138, 101) in the normalized coordinate frame, oriented purely along the x-axis, with a rotation range of approximately -97° to 81° (since 100 units = 180°, so 54 units ≈ 97° and 45 units ≈ 81°).

Physical property assignment. The JSON specifies material type, density, and optionally Young's modulus. For physics simulation, these are used to compute:

• Mass: volume (computed from mesh) × density.
• Inertia tensor: computed from the mesh geometry and density, typically using the moment of inertia integral over the volume.
• Joint dynamics: friction coefficients, damping, and stiffness can be assigned based on the material type and joint type (e.g., a revolute hinge might have higher friction if metal-on-metal than if lubricated plastic).

The paper does not specify whether these additional dynamic parameters are predicted by the MLLM or assigned by heuristics; the JSON output shown in Table 6 includes material and density but not friction or damping, suggesting these may use default values based on the material type.

Scale normalization. The MLLM-predicted "scale" value converts the normalized mesh coordinates (which were centered and scaled to fit in the $64^3$ voxel grid) to real-world dimensions. This is critical because physical simulation is scale-dependent: a 10 cm drawer has different mass, inertia, and required actuation force than a 1 m drawer with identical normalized geometry. The predicted scale also determines the absolute joint origin and axis positions in world coordinates.

Deployment to simulators. The assembled URDF can be directly imported into physics simulators like NVIDIA Isaac Sim, as demonstrated in Figure 6a and the supplementary video. The paper emphasizes that the entire pipeline — from raw mesh to simulating robotic manipulation — is automated:

"This automated pipeline provides three primary advantages for embodied AI: the scalable generation of diverse training scenarios, the facilitation of interactive learning via real-time dynamic feedback, and the provision of multi-modal observation data to benchmark advanced vision-language-action (VLA) models." (Section 5.1)

The VR/AR application workflow. Figure 6b shows an alternative deployment pathway where a user clicks on a region of an image captured in VR/AR, SAM3D generates a 3D mesh from that clicked region, and SIMART processes that mesh to produce an articulated URDF. This demonstrates that SIMART is not limited to processing pre-existing mesh files — it can serve as a component in interactive pipelines where geometry is generated on-the-fly from user interaction, then immediately functionalized for physical interaction.

Summary of Design Choices and Justifications

• Sparse over dense tokenization: Dense $64^3$ voxel grids produce ~4,100 tokens per part, causing OOM failures on objects averaging four parts (Table 3). Sparse tokenization with the zero-token mechanism reduces this to ~516 tokens per object (across all parts), a ~70% reduction that makes MLLM processing feasible while also improving performance by focusing attention on occupied geometry.

• Zero-token over force-sparse: Simply discarding empty-space tokens without the specialized $\mathbf{e}_{zero}$ embedding degrades reconstruction quality by 13× (CD of 56.10 vs. 4.19, Table 4) because the decoder needs explicit information about empty regions to reconstruct clean surfaces. The zero token formalizes an emergent VQ-VAE behavior where codebook entries naturally converge to represent empty space.

• Coordinate-aware ⟨voxel⟩ [xyz] [K] format: Without explicit coordinates, the MLLM would have no way to locate sparse tokens in 3D space — the sequence order is arbitrary and conveys no spatial information. The linearized index $64x + 8y + z$ provides a compact single-integer encoding of 3D position that the MLLM can learn to decode into spatial relationships.

• MLLM over task-specific architectures: Using Qwen3-VL rather than a custom 3D CNN or transformer leverages the model's pretrained physical world knowledge (what drawers and hinges look like, how they move) that would be impossible to learn from the limited articulated 3D datasets alone. The MLLM brings "free" generalization capability from its internet-scale pretraining.

• Single-stage joint generation over multi-stage pipelines: Generating part tokens and URDF JSON in the same autoregressive sequence enables cross-conditioning — the model can decide joint parameters informed by the geometric decomposition and vice versa, avoiding the error propagation that plagues decoupled pipelines.

• Graph-based surface segmentation over direct voxel-to-mesh conversion: Operating on the input mesh's connectivity structure preserves the original surface detail, texture, and watertightness, which direct voxel grid processing cannot achieve due to resolution limits and quantization artifacts.

• Isometric rendered image as auxiliary input: The 252×252 isometric render provides high-resolution visual context (texture, edges, semantic cues) that the coarse geometric tokens cannot capture. The ablation in Table 3 shows this adds ~15 percentage points to joint type accuracy and approximately halves Chamfer Distance compared to geometry-only input.

4. Key Insights and Innovations

Innovation 1: Single-Stage Articulation-Aware Decomposition via Joint Geometric and Kinematic Generation

The dominant paradigm in articulated asset creation prior to SIMART was the multi-stage pipeline: first segment the mesh into parts, then infer joints between those parts, then assemble. This decoupling seemed natural — part segmentation is a geometric problem, joint estimation is a kinematic problem — but the paper identifies a fundamental flaw that previous work had not systematically diagnosed: part decomposition without articulation awareness is underconstrained in a way that inevitably propagates errors downstream.

The paper's diagnostic contribution is in Section 1: "Part decomposition is frequently not articulation-aware: prompting 2D vision(-language) models transfers unreliably to 3D boundaries, while 3D-native segmentation methods... primarily optimize for surface-level consistency and can miss mechanically meaningful link boundaries, producing parts that look plausible yet violate kinematic affordances." This is not merely an observation that multi-stage pipelines are suboptimal — it identifies a specific causal mechanism for why they fail. A surface-level segmenter sees geometric continuity and groups faces together. A kinematic reasoner sees a revolute joint between two rigid bodies and needs the boundary precisely where relative motion occurs. When these two operations are decoupled, the segmenter has no way to know which geometric boundaries are mechanically meaningful, and the joint estimator cannot fix segmentation errors that make its predicted joints physically invalid.

Single-stage joint generation inside a unified MLLM forward pass resolves this precisely because the model generates per-part voxel tokens and structured URDF specifications in the same autoregressive sequence. The autoregressive nature means the model can condition its geometric assignments on its kinematic decisions: if the MLLM determines that two parts are connected by a revolute joint with axis along the x-direction, it can ensure the voxel tokens assigned to those parts form a boundary consistent with rotation around that axis. This is a conceptual reframing rather than merely an architectural improvement: articulation-aware decomposition is not decomposition followed by articulation, but a joint inference process where the kinematic structure constrains the geometric segmentation.

The significance extends beyond performance gains. If multi-stage pipelines are fundamentally limited by the articulation-unawareness of their decomposition step, then incremental improvements to the segmenter (better features, higher resolution, stronger priors) will asymptote at the point where geometric accuracy saturates but kinematic validity remains unaddressed. SIMART's unified approach sets a new performance ceiling by making articulation awareness a first-class property of the decomposition process itself. The quantitative evidence in Table 1 supports this: on AI-generated items where geometric novelty forces reasoning rather than memorization, SIMART achieves 0.831 joint type accuracy compared to 0.817 for Particulate (the strongest multi-stage baseline) and 0.765 for Articulate-Anything (a 2D-only MLLM pipeline), while simultaneously achieving dramatically better geometric fidelity — 0.777 IoU vs. 0.618 for Particulate, and 0.079 CD vs. 0.106. The IoU gap is particularly telling because Particulate uses a dedicated point segmentation module that should excel at geometric accuracy, yet SIMART's articulation-aware decomposition produces more accurate part boundaries precisely because the kinematic reasoning informs where those boundaries should be.

Innovation 2: The Zero-Token Mechanism as a Formalization of Emergent Empty-Space Representation in VQ-VAEs

At first glance, the Sparse 3D VQ-VAE with its specialized zero token appears to be an efficiency optimization — reduce token count to fit within MLLM memory constraints. This is true at the surface level, but the paper reveals a deeper conceptual insight through its ablation analysis that elevates this from engineering to a diagnostic finding about VQ-VAE training dynamics.

The critical observation, described in Appendix B, is that when training a standard dense VQ-VAE without any explicit empty-space supervision, "approximately two to four codebook entries consistently converge to represent the null distribution of empty voxels. These entries, when processed by the decoder, effectively reconstruct empty volumetric regions with high stability." In other words, VQ-VAEs spontaneously discover the concept of "emptiness" and allocate codebook capacity to represent it, even when the training objective treats all spatial positions identically.

This emergent behavior had not been previously documented or exploited in the 3D MLLM literature. Prior work like PhysX-Anything and ShapeLLM-Omni used dense tokenization that wastes codebook entries on empty space, and the field's solution was to simply accept the computational cost or downsample more aggressively. SIMART's innovation is recognizing that this emergent property can be deliberately harnessed: by explicitly reserving the 0-th codebook entry as $\mathbf{e}_{zero}$ and bypassing the quantization step for unoccupied positions, the model achieves three benefits simultaneously: (1) token reduction (~70%), (2) improved codebook utilization (all 4,095 remaining entries represent actual geometry rather than ~2-4 being wasted on empty space), and (3) cleaner reconstruction (the zero token provides stable, consistent empty-space decoding rather than relying on whichever codebook entries happened to converge to emptiness during training).

The ablation in Table 4 (Appendix B) quantifies why this matters beyond token efficiency. The "Force Sparse" configuration — which discards empty-space tokens without using the specialized zero token — achieves a Chamfer Distance of 56.10 (scaled by 10^5), compared to 4.19 for the zero-token variant. This is a ~13× degradation in reconstruction quality. What's happening is that when empty-space tokens are simply omitted, the decoder has no information about where empty space is — it must hallucinate the boundaries of the object from only the occupied voxel tokens. The zero token solves this by providing the decoder with an explicit "nothing here" signal for every latent position, enabling it to cleanly separate occupied from unoccupied regions.

This finding has implications beyond SIMART. It suggests a general principle for sparse 3D tokenization in multimodal models: compress by discarding repetitive tokens, but retain a learned token that explicitly signals absence. The naive approach (just drop empty positions) preserves the occupied tokens but destroys the spatial context that tells the decoder where the object boundary lies. The zero token is essentially a learned "background" embedding that makes the sparsity explicit rather than implicit.

The practical consequence is that SIMART can process objects with four parts at ~516 tokens (Table 3), where the dense baseline fails with OOM errors at ~4,100 tokens. But the intellectual contribution is the demonstration that sparsity in VQ-VAE tokenization requires maintaining an explicit empty-space representation to preserve reconstruction fidelity — and that this representation emerges naturally during training and can be formalized rather than engineered.

Innovation 3: Explicit Coordinate-Aware Tokenization Enables Fine-Grained Spatial Reasoning in MLLMs

When a standard dense voxel grid is tokenized, spatial information is implicit in the token's position in the sequence — the 47th token always corresponds to a known (x, y, z) coordinate because the grid is traversed in a fixed order. Sparse tokenization breaks this implicit mapping: if only occupied voxels are included, the sequence order becomes arbitrary and carries no spatial information. The field's prior sparse 3D tokenization approaches (e.g., point cloud serialization methods) typically relied on the MLLM to learn spatial relationships from the order in which tokens appear, which is brittle — the model must reconstruct 3D structure from 1D sequence position, and any permutation of the input changes the learned spatial mapping.

SIMART's coordinate-aware format — ⟨voxel⟩ [xyz] [K] — is a conceptual departure from this implicit approach. By encoding each voxel's position as an explicit integer token $xyz = 64x + 8y + z$ alongside its geometric content $K$, the model can attend to spatial relationships directly through the coordinate tokens rather than inferring them from sequence structure. This means the MLLM can learn operations like "find all voxels with similar y-coordinates and high x-coordinates" (locating a vertical edge) or "compare the z-distribution of voxels in part 0 vs. part 1" (determining which part is above which) — operations that are fundamental to joint axis and origin prediction but would be difficult to learn from implicit positional encoding alone.

The significance is not that explicit coordinates are a new idea (positional encodings are ancient in deep learning), but rather that discrete coordinate tokens in a shared vocabulary with geometric and text tokens enable cross-modal spatial attention. When the MLLM generates the joint axis prediction "axis": [100, 0, 0], its attention can directly connect the "100" token (representing a direction aligned with the x-axis) to the voxel tokens with high x-coordinate indices that form the edge of the rotating part. This cross-modal grounding between symbolic kinematic parameters and geometric tokens is what enables the precise axis and origin prediction that SIMART achieves (0.080 axis error on ID items, compared to 0.208 for the next-best method Particulate — a ~2.6× reduction).

The evidence for this claim comes from the part grounding results in Table 2. When asked to identify "the part of this object with description: [text]," SIMART achieves 0.807 IoU compared to 0.507 for a P3SAM + Qwen3-VL-235B baseline (where P3SAM segments the mesh and a much larger VLM verifies alignment with the description). The P3SAM baseline represents the state of the art in "segment first, understand later" — it produces geometrically accurate segments but then relies on a separate model to determine which segment matches the query. SIMART's 0.300 IoU advantage suggests that the coordinate-aware tokenization allows the MLLM to directly ground functional language in spatial coordinates — "the lid" maps to the voxel cluster with high z-values, "the drawer" maps to the voxel cluster with mid-range z-values and an extended x-range — without the intermediary of a pre-computed segmentation. The geometric tokens with explicit coordinates serve simultaneously as the representation for segmentation and the representation for language grounding.

This also explains a subtle result: the CD metric in Table 2 shows SIMART at 0.018 vs. P3SAM + Qwen3-VL at 0.234 — a 13× reduction. This is not just better segmentation accuracy; it reflects that SIMART outputs geometric tokens that are a subset of the input (as enforced by the system prompt), whereas P3SAM generates segments that may deviate from the original surface. The explicit coordinate tokens ensure that the MLLM's part assignment operates on the original geometry rather than generating new geometry, which is what enables the strict structural fidelity that the paper claims as a key advantage over generative baselines.

Innovation 4: SIMART-Bench as a Diagnostic Instrument for Articulation Generalization

The paper's introduction of SIMART-Bench appears in Section 4 as an evaluation contribution, but its intellectual significance runs deeper than providing a new test set. The key diagnostic insight is that evaluating on PartNet-Mobility alone systematically overestimates generalization capability because the dataset's intra-category homogeneity allows geometric memorization to masquerade as kinematic reasoning.

PartNet-Mobility contains approximately 2,000 articulated models, but within each category (e.g., "cabinet," "drawer," "door"), the geometric variation is limited — similar proportions, similar handle placements, similar joint configurations. A model that achieves high accuracy on PartNet-Mobility might be recognizing category-level templates ("this voxel configuration looks like the training set's cabinet #47") rather than performing generalizable kinematic reasoning ("this planar region bounded by a thin gap from the main body is likely a door with a vertical hinge at the gap edge"). SIMART-Bench introduces AI-generated objects (from Hunyuan3D-V3.1) with "diverse topologies that better challenge an algorithm's robustness beyond standard benchmarks" — objects with novel proportions, unusual part arrangements, and geometric features not present in any PartNet-Mobility category.

The diagnostic power of SIMART-Bench is visible in the pattern of performance drops between ID and AI-generated items in Table 1. SIMART drops from 0.928 to 0.831 type accuracy (~10% relative decline). Particulate drops from 0.822 to 0.817 (~0.6% relative decline). PhysX-Anything drops from 0.686 to 0.658 (~4% relative decline). The fact that SIMART shows the largest absolute drop (0.097) while still achieving the highest AI-generated accuracy (0.831) suggests that SIMART is learning generalizable reasoning — it achieves high ID performance through genuine understanding that partially transfers to novel geometries, rather than through perfect memorization that would show a catastrophic drop. If SIMART were merely memorizing PartNet-Mobility templates, its AI-generated performance would be far lower. The residual gap (0.097) indicates room for improvement, but the high absolute performance on AI-generated items validates the approach's generalization.

The benchmark also reveals a counterintuitive pattern: Particulate's nearly flat performance (0.822 vs. 0.817) could be interpreted either as excellent generalization (no drop) or as a ceiling effect where the model's standalone segmentation module, which was not trained on PartNet-Mobility articulation data specifically, achieves moderate accuracy on both distributions but cannot reach the higher accuracy that SIMART's MLLM-based reasoning enables on in-domain items. The benchmark thus serves not just to rank methods but to diagnose the nature of their generalization: SIMART shows strong transfer with room to grow, while methods relying on geometric heuristics without physical reasoning show consistent but lower performance across distributions.

The construction methodology of SIMART-Bench further distinguishes it as an evaluation contribution. Rather than using automated metrics that might reward plausible-looking but physically invalid segmentations, the benchmark's ground truth is produced through a human-in-the-loop pipeline: automated over-segmentation via P3SAM, manual merging into functional components, and expert annotation of joint axes and positions via a specialized Web UI. This ensures that the ground truth reflects actual kinematic affordances — parts that move relative to each other — rather than surface-level geometric clustering. The paper's claim that this "establishes a standardized metric for evaluating articulation accuracy on both in-domain and out-of-distribution assets" (Section 6) is supported by the systematic performance gaps it reveals: a method that looks competitive on PartNet-Mobility (e.g., PhysX-Anything at 0.686 type accuracy) drops to 0.658 on SIMART-Bench, exposing a brittleness invisible in prior evaluations.

Innovation 5: Demonstration That MLLM World Knowledge Substitutes for Articulated Training Data Scarcity

The paper's most provocative finding is implicit rather than explicitly claimed: the MLLM's pretrained physical world knowledge — derived from internet-scale images and text, not from 3D articulation data — is sufficient to drive high-accuracy joint parameter prediction and articulation-aware segmentation, even on novel geometries the model has never seen during fine-tuning. This is a fundamental insight about knowledge transfer across modalities that challenges the assumption that 3D articulation tasks require 3D articulation training data.

The field's dominant prior assumption, exemplified by methods like CAGE and SINGAPO, was that learning articulated structure requires large datasets of articulated 3D models with ground-truth joint annotations. The scarcity of such datasets (PartNet-Mobility has ~2K models, compared to ~800K static models in Objaverse) was seen as the primary bottleneck limiting articulated object generation. Section 2.1 articulates this explicitly: generative methods "are hindered by the acute scarcity and limited diversity of articulated 3D datasets compared to rigid objects. Consequently, these models are prone to overfitting, often failing to produce structurally sound or novel articulations for uncommon object categories."

SIMART's architecture effectively sidesteps this bottleneck by leveraging Qwen3-VL's pretraining. The MLLM was trained on a vast corpus of images and text that includes countless examples of articulated objects — cabinet doors, drawers, laptops, scissors, doors, boxes with hinged lids — depicted in natural contexts. From this data, the model has learned that certain visual patterns (a thin gap line on a surface, a handle positioned near an edge, a planar region offset from a frame) correlate with specific kinematic structures (revolute joints, prismatic joints, hinge axes). The fine-tuning on 960K QA pairs derived from PartNet-Mobility and PhysXNet then teaches the model to map this pre-existing understanding onto the specific output format (sparse voxel tokens + JSON URDF) and to ground it in the 3D geometric representation provided by the Sparse VQ-VAE.

The evidence for this transfer comes from the AI-generated results in Table 1, particularly the joint axis and origin metrics. On AI-generated items with geometries not present in PartNet-Mobility, SIMART achieves 0.136 axis error and 0.145 origin error — substantially better than any baseline, including Particulate (0.166 and 0.168). The MLLM was never fine-tuned on these novel geometries, yet it correctly identifies joint axes and origins because the underlying physical reasoning — "doors hinge on one edge, drawers slide perpendicular to their face" — transfers from its pretraining knowledge. If the model were relying solely on the 5,600 articulated training objects to learn kinematic patterns, it would struggle with geometries outside that distribution.

This finding has significant implications beyond SIMART's specific task. It suggests that multimodal pretraining on 2D data can substitute for scarce 3D task-specific data when the task requires physical reasoning that is well-represented in the 2D visual domain. The key requirement is a mechanism to ground the 2D-derived knowledge in 3D geometry — which SIMART provides through the Sparse VQ-VAE and coordinate-aware tokenization. This opens the door to applying MLLM-based approaches to other 3D tasks where labeled 3D data is scarce but 2D visual data is abundant: material property estimation, structural stability analysis, ergonomic assessment, and assembly sequence planning all require physical reasoning that MLLMs may possess from pretraining but that lacks large-scale 3D supervised datasets.

The limitation, however, is equally informative. On the hardest items (bin 5 difficulty-equivalent AI-generated objects with highly unusual geometries), Table 1 shows that all methods degrade. SIMART still outperforms baselines, but the absolute performance is lower than on in-domain items. This suggests that MLLM pretraining provides a strong prior for common physical structures (hinges, sliders, rotational joints in standard configurations) but may not cover exotic mechanisms (complex linkages, compound joints, non-standard axis orientations). The gap between common and exotic articulations defines the boundary of what MLLM world knowledge can substitute for — and where genuinely 3D-specific training data or physical simulation-based reasoning might be necessary.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The MLLM instruction tuning phase uses a dataset of 39,600 3D objects sourced from PhysXNet (Cao et al., 2025) and PartNet-Mobility (Mo et al., 2019), comprising 5,600 articulated models and 34,000 static objects. Each articulated model is rendered in 20 diverse kinematic states, yielding 112,000 effective articulated training configurations. From this collection, two large-scale instruction-following datasets are synthesized: a URDF generation set and a part grounding set, each containing 960K QA pairs. Evaluation is conducted on the 500-question MATH benchmark? No — evaluation is conducted on SIMART-Bench, a custom benchmark consolidating In-Domain (ID) assets from PartNet-Mobility with Out-of-Distribution (OOD) objects synthesized via AIGC pipelines (specifically Hunyuan3D-V3.1; Lai et al., 2025). The benchmark contains over 10 categories of articulated objects with 36 unified assets for comprehensive evaluation. The AIGC objects are annotated through a human-in-the-loop pipeline: automated over-segmentation via P3SAM, manual merging into 2–4 functional components, and expert annotation of motion axes and joint positions via a specialized Web UI to produce simulation-ready ground-truth metadata. The paper does not specify an explicit train/validation/test split for SIMART-Bench, suggesting the full 36 assets are used as a held-out test set for final evaluation, with MLLM training conducted on the separate 39,600-object dataset.

• Base model(s). The core reasoning engine uses Qwen3-VL-8B (Bai et al., 2025) as the MLLM backbone. This 8-billion-parameter multimodal model was chosen for its "powerful large-scale image-text pre-training and its emergent capability for physical world understanding" (Section 3.2). The Sparse 3D VQ-VAE component is initialized from the TRELLIS VAE (Xiang et al., 2025) checkpoint and pretrained on a 500,000-object subset following the TRELLIS data distribution using 8 NVIDIA A100 GPUs for two stages of 60,000 steps each (120,000 total). The full MLLM undergoes fine-tuning for 30,000 steps on a cluster of 32 NVIDIA A100 GPUs. The paper also uses Qwen3-VL-235B in one baseline configuration (P3SAM + Qwen3-VL-235B) for the part grounding task, representing a much larger model scale for comparison.

• Metrics. Five quantitative indicators evaluate articulated asset generation. Joint Type Accuracy (Type ↑): The fraction of predicted joint types (revolute, prismatic, fixed, etc.) that match the ground truth. Axis Error (Axis ↓): The angular deviation between the predicted joint axis direction vector and the ground-truth axis, measured in some normalized angular unit (the paper does not specify whether this is degrees, radians, or a normalized dot-product error, but the consistent scale across methods enables relative comparison). Origin Error (Origin ↓): The L2 Euclidean distance between the predicted joint origin position and the ground-truth origin, computed in the normalized coordinate frame. Intersection over Union (IoU ↑): The volumetric overlap between predicted and ground-truth part segments, measuring segmentation accuracy. Chamfer Distance (CD ↓): The symmetric nearest-neighbor distance between the predicted part mesh surface and the ground-truth part mesh surface, quantifying geometric reconstruction fidelity. For the part grounding task (Table 2), only IoU and CD are reported. For the Sparse 3D VQ-VAE reconstruction evaluation (Table 4), additional metrics include Mean Squared Error (MSE ↓) on the $64^3$ voxel occupancy grid, with both MSE and CD values scaled by $10^5$ for readability.

• Baselines. Four methods serve as comparisons in Table 1. Urdformer (Chen et al., 2024) is a pipeline for constructing articulated simulation environments from real-world images, representing direct URDF regression from 2D inputs. Articulate-Anything (Le et al., 2024) leverages vision-language foundation models to automatically generate articulated object models from visual observations — a 2D-only MLLM approach without native 3D processing. PhysX-Anything (Cao et al., 2025) extends MLLM-based articulation to 3D voxel generation, but uses dense volumetric tokenization that the paper identifies as the primary bottleneck. Particulate (Li et al., 2025) is a feed-forward method for 3D object articulation that operates on point clouds and includes a standalone part segmentation module — it is the strongest baseline on geometric metrics and the only prior method that processes raw 3D inputs. For the part grounding task (Table 2), two baselines are used: PhysX-Anything (as a representative generative baseline) and a custom combination of P3SAM (Ma et al., 2025) with Qwen3-VL-235B, where P3SAM performs native 3D part segmentation and the much larger VLM verifies whether the segmented parts align with functional descriptions.

• Generation budget / compute accounting. The paper does not measure compute in a standardized FLOPs or generation-count unit as is common in LLM scaling analyses. Instead, the key efficiency metric is token count: the average number of discrete tokens produced by the 3D tokenizer per object, which directly determines the MLLM's context length and memory requirements during both training and inference. The dense baseline requires approximately 4,138 tokens per object; the zero-token sparse configuration requires approximately 516 tokens per object — a ~87% reduction (the paper claims 70% reduction in the abstract, but Table 3 shows 4,138 → 516, which is an ~87.5% reduction; the 70% figure may refer to a different comparison point or an average across varying part counts). The paper emphasizes that the dense baseline encounters Out-of-Memory (OOM) errors during fine-tuning on objects averaging four parts, making token count the binding constraint rather than a soft efficiency preference. Training is conducted on 32 A100 GPUs for 30,000 MLLM steps, which at typical batch sizes for 8B models (global batch 128–256) would process approximately 3.8–7.7 million examples — several epochs over the 1.92M-pair dataset.

• Cross-validation / statistical protocol. The paper does not describe explicit cross-validation or statistical significance testing for the main results. The SIMART-Bench evaluation appears to use a single held-out set of 36 assets with fixed ground-truth annotations. For the Sparse VQ-VAE training, validation is performed on held-out data, but no details are provided about the split ratio or whether multiple random seeds were evaluated. The two-fold cross-validation protocol described in the reference example (for compute-optimal strategy selection) does not apply to this paper. Error bars, confidence intervals, and standard deviations are not reported for any metric in Tables 1–4, which is a limitation for assessing the statistical reliability of the ~0.01–0.02 differences observed between some methods.

Main Quantitative Results

Articulated Object Generation and Kinematic Awareness (Table 1, Figure 4)

The headline result is that SIMART achieves state-of-the-art performance across all five metrics on both In-Domain (ID) and AI-generated items. On ID items, SIMART attains 0.928 joint type accuracy, 0.080 axis error, 0.111 origin error, 0.690 IoU, and 0.087 Chamfer Distance. The next-best method, Particulate, achieves 0.822 type accuracy, 0.208 axis error, 0.204 origin error, 0.643 IoU, and 0.140 CD. This means SIMART reduces axis error by approximately 2.6× (0.080 vs. 0.208) and Chamfer Distance by approximately 1.6× (0.087 vs. 0.140) compared to the strongest baseline.

The performance pattern across baselines reveals a diagnostic structure. Methods that process only 2D inputs (Urdformer, Articulate-Anything) or dense 3D voxels (PhysX-Anything) achieve poor geometric fidelity: Urdformer at 0.002 IoU and 0.624 CD, Articulate-Anything at 0.202 IoU and 0.239 CD, and PhysX-Anything at 0.128 IoU and 0.278 CD. These methods fundamentally cannot match the input geometry because they generate new geometry rather than segmenting the existing mesh. Particulate, which operates on point clouds and segments the input, achieves substantially better geometric metrics (0.643 IoU, 0.140 CD), confirming that geometric fidelity requires processing the native 3D representation directly. SIMART extends this advantage further (0.690 IoU, 0.087 CD) while simultaneously achieving better kinematic accuracy, suggesting the MLLM-based joint reasoning provides complementary information that improves both axes simultaneously.

On AI-generated items, all methods degrade — confirming that SIMART-Bench's novel geometries represent a genuinely harder generalization challenge. SIMART achieves 0.831 type accuracy, 0.136 axis error, 0.145 origin error, 0.777 IoU, and 0.079 CD. The IoU on AI-generated items (0.777) is actually higher than on ID items (0.690), which is counterintuitive and warrants scrutiny. One possible explanation is that AI-generated objects have simpler part topologies that are easier to segment, even though their kinematic structures are harder to classify. Alternatively, the ID and AI-generated test sets may have different difficulty distributions — the ID set might contain more complex multi-part assemblies while AI-generated items might be individually simpler. The paper does not discuss this inversion.

The qualitative comparison in Figure 4 reinforces the quantitative findings. Generative baselines (Articulate-Anything, PhysX-Anything) produce "overly simplified or misaligned geometries that fail to match the input observations" while SIMART "achieves superior structural fidelity by directly processing the input mesh through the Sparse VQ-VAE and leveraging MLLM-driven part segmentation." The extended qualitative comparison in Figure 7 (Appendix) shows additional AI-generated examples where SIMART preserves fine geometric detail that Particulate's point-cloud-based segmentation loses.

3D Part Understanding (Table 2, Figure 5)

The part understanding task evaluates a narrower capability: given a functional text description of a specific component (e.g., "the hinged lid"), can the model identify and reconstruct precisely that component's geometry? SIMART achieves 0.807 IoU and 0.018 CD on AI-generated items. PhysX-Anything, the generative baseline, achieves only 0.067 IoU and 0.347 CD — essentially failing at this task, as expected since it generates simplified proxy geometry rather than segmenting the input. The more informative comparison is against the P3SAM + Qwen3-VL-235B baseline, which pairs a state-of-the-art 3D part segmenter (P3SAM) with a much larger VLM (235B vs. SIMART's 8B) that verifies whether the segments match the functional description. This baseline achieves 0.507 IoU and 0.234 CD.

SIMART's advantage over this strong baseline (0.300 IoU improvement, ~13× CD reduction) is the clearest evidence for the paper's central claim about the value of unified, articulation-aware processing. The P3SAM + Qwen3-VL setup represents the best possible "segment first, understand later" approach: P3SAM produces high-quality geometric segments, and a massive 235B-parameter VLM filters them. Yet SIMART's 8B MLLM outperforms this by a wide margin. Why? Because SIMART's MLLM directly grounds the functional language in the geometric tokens during segmentation, rather than segmenting blindly and then retroactively checking which segment matches the query. The coordinate-aware tokenization enables the model to associate "lid" with high-z-coordinate voxel clusters and "hinge" with edge-adjacent voxels during the forward pass, producing an articulation-aware segmentation from the start rather than filtering geometrically-motivated segments post-hoc.

The qualitative results in Figure 5 illustrate this with specific examples: SIMART precisely identifies and isolates functional components like lids and doors while maintaining "superior geometric consistency with the ground truth." The key advantage is that the output voxels are explicitly constrained to be a "subset of the input" (as enforced by the system prompt in Table 5), so SIMART cannot hallucinate geometry — it can only assign existing surface points to parts. The P3SAM baseline, while producing clean segments, may split the surface differently than the functional query requires.

Sparse 3D VQ-VAE Reconstruction Quality (Table 4, Appendix B)

This ablation, presented in the Appendix, evaluates the geometric fidelity of the VQ-VAE itself, independent of the MLLM's reasoning capabilities. Input meshes are voxelized into a $64^3$ grid, encoded through the Sparse VQ-VAE, decoded back to a $64^3$ grid, and compared against the original voxelization. The final $8 \times 8 \times 8$ sparse configuration achieves MSE of 1.84 and CD of 4.19 (both scaled by $10^5$). Expanding the latent resolution to $16 \times 8 \times 8$ improves reconstruction (MSE 1.15, CD 2.27) but doubles token sequence length, causing memory issues for the MLLM backbone — the $8 \times 8 \times 8$ configuration represents an explicit tradeoff between geometric fidelity and computational feasibility. Doubling the codebook size to 8,192 entries yields no improvement (MSE 1.84, CD 4.56 — slightly worse CD), indicating the 4,096-entry codebook is sufficient.

The most diagnostic result is the Force Sparse (No Zero Token) configuration, which achieves MSE 2.66 and CD 56.10. The CD degrades by a factor of ~13.4× compared to the zero-token variant (56.10 vs. 4.19), demonstrating that simply discarding empty-space tokens without the specialized $\mathbf{e}_{zero}$ embedding catastrophically harms reconstruction. The decoder, lacking information about where empty space is located, cannot cleanly separate object surface from background, resulting in highly noisy reconstructions.

Ablation Studies and Robustness Checks

Dense token vs. sparse token (Table 3, row "Dense token"): The dense voxel tokenization baseline requires approximately 4,138 tokens per object and encounters Out-of-Memory (OOM) errors during fine-tuning on objects averaging four parts. The MLLM's context length and GPU memory cannot accommodate the sequence length required for multi-part generation, confirming that dense tokenization is structurally incompatible with per-part articulated object processing at any reasonable resolution. This is the binding constraint that motivates the entire sparse tokenization approach.

Force sparse vs. zero-token sparse (Table 3, rows "Force Sparse" vs. "Zero Sparse"): The force-sparse configuration, which discards empty-space tokens without using the specialized $\mathbf{e}_{zero}$ embedding, achieves 0.661 type accuracy, 0.157 center error (equivalent to origin error in Table 1), 0.678 IoU, and 0.100 CD using 862 tokens on average. The zero-token sparse variant improves to 0.794 type accuracy, 0.108 center error, 0.745 IoU, and 0.074 CD while using only 516 tokens (a 40% further reduction from the already-sparse 862). This demonstrates that the zero token provides a double benefit: fewer tokens and better performance. The performance improvement is attributable to two factors working in concert: (1) the MLLM's attention is not diluted by tokens representing empty space that the force-sparse variant still encodes (albeit fewer), and (2) the decoder receives explicit empty-space information via the $\mathbf{e}_{zero}$ embedding, enabling cleaner part reconstructions that feed back into more accurate segmentation. The token count reduction from 862 to 516 for the zero-token variant is particularly interesting because it suggests the force-sparse approach is still encoding some empty-space tokens — possibly at the boundaries of the object where occupancy is ambiguous, or in interior cavities — that the zero token correctly identifies and suppresses.

Vision modality contribution (Table 3, row "Vision (ours)"): Adding vision features from the 252×252 isometric rendered image to the zero-token sparse geometry tokens (forming the complete SIMART model) improves performance across all metrics: type accuracy from 0.794 to 0.937 (+14.3 percentage points), center error from 0.108 to 0.074 (-31%), IoU from 0.745 to 0.832 (+8.7 percentage points), and CD from 0.074 to 0.055 (-26%). The token count remains at 516 (vision tokens are counted separately; the geometric token count is unchanged). This is the largest single-component improvement in the ablation, highlighting that geometric tokens alone — even with efficient sparse encoding and explicit coordinates — are insufficient for precise kinematic reasoning. The visual modality provides three orthogonal information channels that the geometric tokens cannot capture: (1) high-resolution surface detail (edges, corners, seams visible as pixel-level gradients rather than $8 \times 8 \times 8$ voxel occupancies), (2) semantic appearance cues (material type, object category, handle/grip affordances visible in the image), and (3) global context (the isometric view shows how parts relate at the object scale, which the local $8 \times 8 \times 8$ voxel patches may not fully capture). The fact that vision contributes more to type accuracy (+14.3 pp) than to IoU (+8.7 pp) suggests that appearance cues are particularly informative for determining how parts move (joint type classification) rather than where part boundaries lie (segmentation), which aligns with the intuition that functional articulation type is often inferrable from visual appearance (a thin gap line suggests a hinge; a handle on a flat front suggests a drawer).

Codebook size (Table 4, row "Codebook (8192)"): Doubling the codebook from 4,096 to 8,192 entries produces no improvement in reconstruction quality (MSE remains 1.84; CD increases slightly from 4.19 to 4.56, suggesting marginally worse surface fidelity with the larger codebook). This validates that the 4,096-entry codebook is sufficient when the zero token absorbs the empty-space representation burden — the remaining 4,095 geometric entries provide enough capacity to represent the diversity of local surface shapes encountered across the 500K-object pretraining set.

Latent resolution (Table 4, row "Sparse 16×8×8"): Expanding the latent grid from $8 \times 8 \times 8$ to $16 \times 8 \times 8$ improves reconstruction (MSE 1.84 → 1.15, CD 4.19 → 2.27) but doubles the token count, imposing "substantial memory overhead for the multimodal backbone." The paper chooses the $8 \times 8 \times 8$ configuration as the operating point, accepting moderately degraded reconstruction in exchange for feasible MLLM training. This tradeoff is not further optimized — the paper does not explore whether a $12 \times 8 \times 8$ grid or an asymmetric $8 \times 8 \times 4$ grid might provide intermediate cost-quality points.

Revision-specific verifier? Not applicable. This paper does not use sequential revision models or separate verifiers; the MLLM directly generates the final output in a single autoregressive pass. The equivalent robustness check would be evaluating whether the MLLM's output is consistent across multiple generations (i.e., pass@k reliability), which the paper does not test.

MLLM scale comparison (implicit in Table 2): SIMART's Qwen3-VL-8B backbone outperforms the P3SAM + Qwen3-VL-235B baseline (0.807 vs. 0.507 IoU), despite using a model approximately 29× smaller. This is not a controlled ablation (the architectures differ substantially beyond scale), but it demonstrates that SIMART's unified processing is more important than raw model capacity for this task — a 235B model applied post-hoc to generated segments cannot recover from the fact that those segments were produced without articulation awareness.

Critical Assessment

Does the Paper Demonstrate That SIMART "Jointly Performs Part-Level Decomposition and Kinematic Prediction"?

The paper's central architectural claim is that SIMART performs part segmentation and joint parameter prediction simultaneously in a single forward pass, with each informing the other. The experimental evidence for this is primarily indirect, and several gaps exist.

What the experiments demonstrate: The ablation in Table 3 shows that combining geometric + visual tokens in a unified MLLM achieves better performance on both geometric metrics (IoU, CD) and kinematic metrics (type accuracy, center error) than any ablated configuration. This is consistent with joint processing being beneficial — the model can use kinematic reasoning to improve segmentation, and vice versa.

What the experiments do not directly demonstrate: The paper does not include an experiment that would isolate the specific benefit of joint generation. The natural comparison would be a two-stage SIMART variant: first generate part voxels, then feed those part voxels as a separate input to a second MLLM forward pass that predicts joints. If the one-stage joint version outperforms this two-stage version (at the same total compute), that would directly evidence the value of simultaneous processing. Without this experiment, an alternative explanation is that any MLLM-based processing of 3D geometry outperforms non-MLLM methods, regardless of whether the stages are joint or sequential — the benefit might come from the MLLM's pretraining and the sparse tokenization, not from the simultaneity of the predictions.

A related concern is that the autoregressive generation order (JSON first, then voxel tokens) might limit genuine bidirectionality. The JSON is generated before the voxel tokens, so the kinematic reasoning (joint types, axes) can influence the subsequent geometric assignment, but the geometric assignment cannot retroactively influence the kinematic reasoning. This is a forward-only conditioning, not true joint inference. An experiment with multiple generation passes or iterative refinement would clarify whether the claimed "simultaneous" processing is realized in practice or is limited to sequential conditioning.

Does the Paper Demonstrate That the Sparse 3D VQ-VAE "Reduces Token Counts by 70%"?

The abstract claims 70% reduction; Table 3 shows 4,138 → 516 tokens, which is an 87.5% reduction. The 70% figure may derive from a different calculation (e.g., comparing against a theoretical $8 \times 8 \times 8 = 512$ grid without any empty space, where 516 occupied tokens would represent near-100% density, not a reduction — suggesting the denominator might be something else). Alternatively, the 70% may refer to the reduction from the force-sparse configuration (862 → 516 = 40% reduction) or from a dense grid at a different resolution. The paper does not clarify which comparison yields "70%," which is a minor but notable imprecision.

More importantly, the "70%" figure captures only the token compression from the encoder side. The decoder must reconstruct a full $64^3$ grid — so the compression is asymmetric (encoding is sparse, decoding is dense). The paper does not measure the wall-clock time or GPU memory savings from this token reduction. The OOM result for dense tokens (Table 3) provides a qualitative lower bound (dense is infeasible, sparse is feasible), but the practical efficiency gain (e.g., training throughput improvement, inference latency) is not quantified.

An additional unmeasured cost is the difficulty estimation overhead — in this paper's context, the cost of encoding the input mesh through the Sparse VQ-VAE before the MLLM can process it. For a $64^3$ grid, the 3D-UNet encoder forward pass is not trivially cheap, and this cost is incurred for every object regardless of complexity. The paper does not count this encoding cost in its token-efficiency calculation.

Does the Paper Demonstrate Generalization via SIMART-Bench?

The introduction of SIMART-Bench is a genuine contribution, and the systematic performance gaps between ID and AI-generated items in Table 1 provide evidence that prior metrics on PartNet-Mobility alone overestimate generalization. However, several limitations temper this conclusion:

Small test set size: SIMART-Bench contains 36 assets. With 10+ categories, this averages 3–4 objects per category. A single mislabeled ground-truth joint axis or poorly merged part boundary could swing metrics by several percentage points. The paper does not report inter-annotator agreement for the human-in-the-loop annotation pipeline, nor does it characterize the difficulty distribution of the 36 test items. The test set may not be representative of the broader distribution of AI-generated objects — it represents what was curated, and the curation criteria are not specified beyond "diverse topologies."

Single AIGC source: All AI-generated items come from Hunyuan3D-V3.1. Different 3D generation models produce different artifact patterns — some produce watertight meshes with clean topology, others produce noisy surfaces with holes and non-manifold edges. SIMART's performance on Hunyuan3D objects may not transfer to objects from other generators (DreamGaussian, Seed3D, etc.), which limits the "generalization to diverse AI-generated meshes" claim.

No analysis of failure modes on AI-generated items: The paper reports aggregate metrics but does not break down which types of AI-generated objects cause the largest performance drop. Do objects with thin structures, high-frequency surface detail, unusual aspect ratios, or multiple interacting joints cause disproportionate degradation? A per-category or per-characteristic breakdown would strengthen the diagnostic value of SIMART-Bench and guide future improvements.

Does the Paper Demonstrate That MLLM World Knowledge Enables Generalization Beyond Training Data?

The AI-generated results in Table 1 (0.831 type accuracy, 0.136 axis error) provide evidence consistent with this claim, but several important control experiments are missing:

No comparison against a non-pretrained or randomly-initialized MLLM: The paper attributes the MLLM's performance to pretrained world knowledge, but it does not show how a Qwen3-VL trained only on the 960K instruction-following examples (without internet-scale pretraining) would perform. Such an experiment is admittedly impractical (training an 8B model from scratch on 3D data alone would be both expensive and likely fail for unrelated optimization reasons), but it means the causal link between "internet-scale pretraining" and "generalization to novel geometries" is inferred rather than demonstrated.

No comparison against a non-MLLM approach with equivalent 3D supervision: The baselines (Particulate, PhysX-Anything) were trained on different data or with different objectives. A controlled experiment would be: take the same 39,600-object training set (including the 34,000 static objects), train a 3D-native segmentation + joint estimation model (not an MLLM) to convergence, and compare its AI-generated performance to SIMART's. If the non-MLLM model underperforms by a similar margin on AI-generated items as Particulate does on ID items, the case for MLLM-specific generalization is strengthened. The current experimental design leaves open the possibility that any model with sufficient capacity and the same training data would show similar generalization, and the MLLM's pretraining is not the causal factor.

The vision modality's role in generalization is underexplored: The ablation shows that vision features provide the largest single-component improvement (type accuracy +14.3 pp in Table 3). This is consistent with the hypothesis that visual appearance cues from the rendered image — which the MLLM can interpret using its pretrained visual understanding — are the primary vehicle for generalization to novel geometries, rather than the geometric tokens alone. But this means the generalization evidence is partially about 2D visual generalization (the MLLM recognizes novel cabinet geometries because they look like cabinets in the rendered image) rather than 3D geometric generalization (the MLLM reasons about novel voxel arrangements). The paper does not disentangle these two pathways. An experiment that ablates the image viewpoint (e.g., canonical front view vs. isometric view, or multiple views vs. single view) would clarify how much the 2D visual pathway contributes to generalization versus the 3D geometric pathway.

Weaknesses in Experimental Design

No error bars or statistical testing: All metrics in Tables 1–4 are reported as point estimates without standard deviations, confidence intervals, or significance tests. With a 36-item test set, differences of 0.01–0.02 in metrics like type accuracy correspond to less than one item being classified correctly vs. incorrectly — these differences could arise from a single annotation ambiguity. Without variance estimates, it is impossible to assess whether SIMART's 0.928 type accuracy is statistically distinguishable from Particulate's 0.822 or whether the gap reflects genuine improvement versus sampling noise.

The part grounding experiment uses an uncontrolled VLM baseline: The P3SAM + Qwen3-VL-235B baseline in Table 2 pairs a 3D segmenter with a 235B VLM, but the paper does not specify how the VLM is prompted, whether it was fine-tuned for the task, or how the verification step works. This makes the comparison difficult to interpret. If the 235B model was used zero-shot while SIMART's 8B model was fine-tuned on 960K QA pairs, the comparison is not about architecture but about task-specific training. Conversely, if the 235B model was also fine-tuned (which seems unlikely given the computational cost), the comparison is fairer but the paper doesn't clarify.

No evaluation of physical simulation validity: The paper claims SIMART produces "simulation-ready" assets and demonstrates deployment in Isaac Sim (Figure 6a), but it does not evaluate how well the generated URDFs perform in simulation. A physically valid joint axis with a 2° angular error might produce a simulation that looks correct in a static render but causes the simulated robot's gripper to miss the handle by 2 cm when executing a trajectory. Metrics like "simulation success rate" for standard manipulation tasks (e.g., open the drawer, rotate the lid to a target angle) would provide a more direct evaluation of simulation readiness than geometric error metrics. The paper defers to the supplementary video for this evidence, but videos are qualitative and cannot be systematically compared across methods.

No ablation on the 20-state augmentation strategy: The paper states that rendering each articulated model in 20 kinematic states multiplies the effective training data, but it does not ablate this choice. Would 5 states suffice? Would 50 states help? Is the benefit from seeing more kinematic diversity, or simply from having more training examples (any augmentation would help)? An experiment varying the number of kinematic states would clarify whether the augmentation strategy is critical or incidental.

Missing comparison against Articulate AnyMesh: The paper cites Articulate AnyMesh (Qiu et al., 2025) as a related MLLM-based method but does not include it as a baseline in Tables 1 or 2. This omission is notable because Articulate AnyMesh is one of the most recent and relevant comparisons — it also uses MLLMs for open-vocabulary articulated object modeling. Its absence from the quantitative comparison leaves unclear how SIMART compares to the most directly competitive approach.

No analysis of long-tail or failure cases: The paper shows qualitative successes in Figures 4, 5, 7, and 8, but does not show or analyze failure cases. On which types of objects does SIMART fail? What are the characteristic errors (e.g., confusing revolute and prismatic joints on visually ambiguous objects, missing small parts like handles entirely, misaligning joint axes by 90° on symmetric parts)? Understanding failure modes is essential for assessing practical deployability and for guiding future research, but the paper provides no such analysis.

Computational cost of the full pipeline is not reported: The paper states training configurations (32 A100 GPUs, 30,000 steps) but does not report inference latency or throughput. For a practitioner deciding whether to deploy SIMART versus a lighter-weight baseline like Particulate, knowing that SIMART requires approximately 516 geometric tokens plus vision tokens plus text tokens — totaling perhaps 900–1,000 tokens processed autoregressively by an 8B-parameter model — is essential. The paper provides no inference cost comparison.

Summary of Experimental Support for Central Claims

• "Unified MLLM framework for joint part decomposition and kinematic prediction": Supported by holistic performance improvements over multi-stage baselines (Table 1) and ablation showing that geometric + visual + text modalities outperform subset configurations (Table 3), but the specific value of joint (vs. sequential) generation is not isolated experimentally.

• "Sparse 3D VQ-VAE reduces token counts by 70%": Qualitatively supported — the dense baseline fails with OOM while the sparse variant trains successfully — but the exact reduction percentage is imprecise and the practical efficiency gain (training throughput, inference latency) is not quantified.

• "State-of-the-art performance on PartNet-Mobility and SIMART-Bench": Supported across all five metrics for both ID and AI-generated items in Table 1, though the small test set (36 AI-generated items) and lack of error bars limit the strength of this conclusion.

• "Downstream utility in physics-based simulation": Demonstrated qualitatively in Figure 6 and the supplementary video, but not evaluated quantitatively (no simulation success rate or physical validity metric).

• "Generalization to diverse static meshes": Partially supported by the AI-generated results in Table 1, but the single-source AIGC data, small test set, and absence of controlled experiments isolating MLLM pretraining as the causal generalization mechanism leave this claim as plausible but not rigorously established.

6. Limitations and Trade-offs

6.1 Sparse Tokenization Resolves the Memory Bottleneck but Introduces an Asymmetric Encoding-Decoding Cost That Is Not Quantified

The assumption or constraint. The Sparse 3D VQ-VAE achieves token reduction by encoding only occupied voxels on the MLLM input side, but the decoder must still reconstruct a full $64 \times 64 \times 64$ occupancy grid from the sparse token set. This encoding is asymmetric: the encoder produces a variable-length sequence proportional to object surface area (~516 tokens on average; Table 3), but the decoder operates on the full $64^3$ grid regardless of how sparse the encoding was. The paper acknowledges the tradeoff implicitly by selecting the $8 \times 8 \times 8$ latent resolution as a compromise — the $16 \times 8 \times 8$ configuration offers better reconstruction (MSE 1.15 vs. 1.84, CD 2.27 vs. 4.19 in Table 4) but is rejected because it "doubles the resulting token sequence length, which introduces a substantial memory overhead for the multimodal backbone" (Appendix B). The paper does not report the decoder's computational cost, memory footprint, or inference latency.

The consequence. The headline claim of "70% reduction" in tokens describes only the encoder-to-MLLM pathway. A practitioner deploying SIMART must run the 3D-UNet encoder and decoder in addition to the 8B-parameter MLLM forward pass. For a typical articulated object with 4 parts, the system must: (1) run the encoder once to produce geometric tokens for the MLLM, (2) run the MLLM autoregressively to generate per-part token sequences, and (3) run the decoder once per part to reconstruct each part's $64^3$ grid, plus additional decoder passes if the graph-based segmentation seeds require re-decoding. A 3D-UNet operating on $64^3$ grids is not computationally trivial — depending on the channel dimensions and depth, it may represent a non-negligible fraction of total inference cost. Without quantifying this cost, the practical efficiency advantage over dense baselines is only partially characterized: the dense baseline fails entirely (OOM in Table 3), but for objects small enough to fit in dense representations, the sparse variant's wall-clock advantage may be modest or even negative once decoder costs are included. The paper does not benchmark latency or throughput for end-to-end inference.

What evidence exists in the paper. The ablation in Table 3 provides token counts (4,138 dense vs. 516 sparse) but no timing or memory measurements. The reconstruction quality of the VQ-VAE is evaluated independently in Table 4 (Appendix B), but these measurements are performed on the VQ-VAE in isolation, not as part of the full pipeline. The paper does not report: encoder forward-pass time, decoder forward-pass time (per part), MLLM autoregressive generation time, or total end-to-end latency on representative hardware.

Mitigation status. Not addressed. The paper provides no analysis of inference cost beyond the token-count metric. Future work could profile the full pipeline's latency and memory footprint on standard hardware (e.g., a single A100 GPU), compare end-to-end cost against baselines that use lighter-weight geometric processing (e.g., Particulate's point cloud segmentation), and explore whether the decoder cost could be amortized by decoding all parts in a single batched pass rather than sequentially.

6.2 The SIMART-Bench Test Set Is Too Small and Single-Source to Support Strong Claims About Generalization

The assumption or constraint. SIMART-Bench is introduced to evaluate generalization beyond PartNet-Mobility's homogeneous category distributions. The benchmark contains 36 AI-generated objects from a single source (Hunyuan3D-V3.1; Lai et al., 2025) across "over 10 categories" (Section 4), averaging 3–4 objects per category. The paper acknowledges that PartNet-Mobility's "data distribution is relatively homogeneous, with minimal geometric variance within categories" (Section 4) and positions SIMART-Bench as the remedy, but does not acknowledge the limited size or single-source provenance of the test set as limitations.

The consequence. With only 36 test items — and with individual metrics like type accuracy effectively binned into per-category samples of 3–4 objects — a single annotation error or misclassified joint can shift type accuracy by ~3 percentage points (1/36). The observed gap between SIMART and Particulate on AI-generated type accuracy is 0.831 vs. 0.817 — a difference of 0.014, which corresponds to less than one item being classified differently (0.014 × 36 ≈ 0.5 items). Without error bars or confidence intervals (which the paper does not report), there is no statistical basis to claim this difference reflects genuine superiority rather than sampling noise.

More subtly, all AI-generated objects come from a single generator (Hunyuan3D-V3.1). Different 3D generation pipelines produce systematically different mesh characteristics: some produce watertight, manifold meshes with uniform triangulation; others produce non-manifold edges, internal geometry, inconsistent normals, or thin structures with degenerate triangles. A model that performs well on Hunyuan3D outputs may not transfer to objects from DreamGaussian (Tang et al., 2023), Seed3D (ByteDance Seed, 2025), or other generators — yet the paper's claim of generalization to "diverse static meshes" and "in-the-wild AIGC datasets" (abstract) implicitly encompasses all such sources. The single-source evaluation provides no evidence about this broader generalization.

What evidence exists in the paper. Table 1 reports aggregate metrics on the 36-item AI-generated set. The performance gap between ID and AI-generated items is documented (e.g., SIMART type accuracy drops from 0.928 to 0.831), which demonstrates that AI-generated items are harder, but the small absolute differences between methods on AI-generated items (0.831 vs. 0.817 vs. 0.765 for SIMART, Particulate, and Articulate-Anything respectively, on type accuracy) are presented without variance estimates. The paper does not report per-category breakdowns, inter-annotator agreement for the human-in-the-loop ground-truth pipeline, or results on objects from alternative AIGC sources.

Mitigation status. Partially acknowledged in Section 6: "the scarcity and inconsistent quality of existing articulated datasets remain a primary limitation for open-world generalization. Future work will focus on utilizing SIMART as a foundational tool to generate pre-verified articulation predictions, thereby accelerating the data-annotation loop." This frames the data scarcity as a field-level problem rather than an evaluation limitation specific to this paper, but the small test set size is within the authors' control to address by expanding SIMART-Bench. No concrete plan for benchmark expansion (target size, additional AIGC sources) is specified.

6.3 The Value of Joint (vs. Sequential) Part Decomposition and Kinematic Prediction Is Not Isolated Experimentally

The assumption or constraint. The paper's central architectural claim is that SIMART performs part segmentation and joint parameter prediction jointly in a single autoregressive forward pass, and that this joint processing resolves the error accumulation problem of multi-stage pipelines (Section 1). The autoregressive generation order — structured JSON first, then per-part voxel tokens — means the kinematic reasoning can condition the geometric assignment, but geometric assignment cannot retroactively influence kinematic reasoning. The paper does not provide an experiment that isolates the specific benefit of generating both outputs in one pass versus in two sequential passes using the same MLLM backbone.

The consequence. The observed performance advantage over multi-stage baselines (Particulate, Articulate-Anything) could be explained by factors other than joint generation: (1) the MLLM's pretrained physical world knowledge, which none of the baselines possess; (2) the sparse 3D tokenization enabling higher effective geometric resolution than dense baselines; (3) the vision modality providing appearance cues that non-MLLM baselines cannot leverage. The paper's ablation (Table 3) shows that each of these factors individually contributes substantial performance improvements, but it does not test whether the simultaneity of part and joint prediction provides an independent benefit beyond the sum of these components. A practitioner considering whether to adopt SIMART's unified architecture versus a simpler two-stage design using the same MLLM backbone (first pass: segment parts; second pass: predict joints from the segmentation) needs evidence that joint generation is worth the architectural complexity, but this evidence is absent.

Furthermore, the JSON-first generation order creates a subtle asymmetry: the kinematic structure is decided before the geometric tokens are generated, so if the MLLM makes an early kinematic error (e.g., predicting a revolute joint where a prismatic joint is needed), that error propagates into the subsequent voxel token generation. The part decomposition is conditioned on potentially incorrect kinematic assumptions — the same error propagation problem that the paper criticizes in multi-stage pipelines, just operating within a single model rather than across separate modules. No experiment evaluates whether this forward-only conditioning is a limitation in practice.

What evidence exists in the paper. Table 1 shows SIMART outperforming multi-stage baselines holistically. Table 3 shows ablation over input modalities (dense, force-sparse, zero-sparse, +vision) but does not include a "sequential generation" variant. The paper does not report: performance of a two-stage SIMART variant (segment then predict joints in separate passes), analysis of whether kinematic errors correlate with segmentation errors in the MLLM's output, or an experiment varying the generation order (voxels first, then JSON) to test whether the asymmetry matters.

Mitigation status. Not addressed. The paper does not acknowledge the absence of a sequential baseline or discuss the JSON-first generation order as a potential limitation. Section 8 (Conclusion) does not identify this as an area for future investigation. The claim of "joint" processing remains an architectural description rather than an empirically validated mechanism.

6.4 Physical Simulation Validity Is Claimed but Not Evaluated Quantitatively

The assumption or constraint. SIMART's stated purpose is producing "simulation-ready articulated assets" for "physics-based animation and robotic interactive simulation" (Section 1). The paper demonstrates qualitative deployment in NVIDIA Isaac Sim (Figure 6a) and references a supplementary video showing "a diverse range of kinematic motion sequences and articulated behaviors" (Appendix A). However, the quantitative evaluation (Tables 1–3) uses purely geometric and kinematic metrics — joint type accuracy, axis/origin error, IoU, Chamfer Distance — that measure how well the predicted URDF parameters match ground-truth annotations, not how well the generated assets actually perform in simulation.

The consequence. Geometric metrics can be misleading proxies for simulation readiness. A joint axis with 2° angular error (well within SIMART's reported 0.080 axis error on ID items) may produce a URDF that loads without errors and looks correct in a static visualization, but causes the simulated robot to miss a grasp point by centimeters when executing a trajectory over the full range of motion. An origin error of 0.111 units (Table 1) in the normalized coordinate frame could translate to a joint pivot point offset of 5–10 mm at real-world scale — enough to cause inter-part penetration or physically impossible configurations at joint limits. Joint limits predicted as [-54, 45] instead of the true [-50, 50] (Table 6 example) may cause the simulation to permit physically impossible configurations that a real object cannot achieve.

None of these failure modes are captured by the geometric metrics. A URDF could score perfectly on type accuracy, axis error, and origin error while producing a simulation where parts interpenetrate, joints bind at limits, or the kinematic chain is physically inconsistent. The paper's claim that SIMART produces "simulation-ready" assets is therefore supported qualitatively (videos, Figure 6) but not validated with a quantitative metric that directly measures simulation success.

What evidence exists in the paper. Figure 6a shows a visualization of a SIMART-generated cabinet in Isaac Sim with a robotic arm, and the supplementary video (referenced in Appendix A) demonstrates kinematic motion across various objects. No quantitative simulation metric is reported — no "simulation success rate," no "grasp success rate," no "inter-penetration rate," no comparison of simulation rollout trajectories against ground truth. The metrics in Table 1 evaluate parameter accuracy against annotated URDFs, not functional validity in simulation.

Mitigation status. The paper does not acknowledge this gap. Section 5.1 describes the Isaac Sim deployment as a demonstration of capability, not as an evaluation. The supplementary video is offered as supporting evidence, but video demonstrations are non-quantitative, cannot be systematically compared across methods, and typically show cherry-picked successes. The paper could have defined simulation-based metrics (e.g., "fraction of generated URDFs that successfully complete a standardized open/close trajectory without physics errors") and evaluated all methods under this criterion, but does not.

6.5 The Revision-Like Correct-to-Incorrect Failure Mode Is Present but Neither Measured nor Mitigated in the Generation Setting

The assumption or constraint. The paper does not use sequential revisions in the sense of iterative model refinement, but the autoregressive generation process introduces an analogous failure mode: the MLLM generates the JSON kinematic specification first, then generates per-part voxel tokens conditioned on that JSON. An error in the JSON — a misclassified joint type, a wrong parent-child relationship, an incorrect axis direction — becomes context for the subsequent voxel token generation, potentially corrupting the part segmentation. This is structurally analogous to the "correct-to-incorrect reversion" problem in chain-of-thought or revision models, where an early error propagates forward and degrades later outputs, but the paper does not measure how frequently this occurs or attempt to mitigate it.

The consequence. The joint generation architecture lacks any mechanism for the geometric tokens to "correct" an earlier kinematic error. If the MLLM predicts a prismatic joint for what is actually a revolute hinge, the voxel tokens for the affected parts are generated conditioned on the (incorrect) assumption of linear sliding motion. This could systematically bias the part boundaries — for instance, the model might produce a segmentation where the moving part extends in the sliding direction rather than being compact around the rotation axis, because the earlier JSON context primes it to expect prismatic motion. The resulting asset would not only have an incorrect joint specification but also a geometrically distorted part segmentation that compounds the error. The paper's performance metrics (Table 1) evaluate joint parameters and part geometry separately, so this compounding effect — where joint errors cause segmentation errors, or vice versa — is not captured by any single metric. A model could score reasonably on both type accuracy and IoU while producing outputs where the kinematic and geometric predictions are jointly inconsistent in ways that neither metric individually detects.

What evidence exists in the paper. The autoregressive generation order (JSON first, then voxels) is described in Section 3.2 and Appendix C. The paper does not report: the correlation between joint type errors and segmentation IoU on the same objects, an analysis of error propagation (do objects with incorrect joint predictions have systematically worse segmentation?), or an experiment with reversed generation order (voxels first, then JSON) to test whether the conditioning direction matters.

Mitigation status. Not addressed. This is a structural property of the autoregressive generation design, not a parameter that can be tuned. Mitigation would require architectural changes — e.g., generating multiple candidate JSON specifications and selecting the one that best fits the subsequently generated voxel tokens (a rejection sampling approach), or adding a separate verification pass that checks kinematic-geometric consistency and flags or corrects inconsistent outputs. The paper does not discuss these possibilities.

6.6 The Method Is Validated on a Single MLLM Backbone and Single 3D VQ-VAE Initialization, Leaving Model-Specific Behavior Uncharacterized

The assumption or constraint. All experiments use Qwen3-VL-8B as the MLLM backbone and TRELLIS VAE initialization for the Sparse 3D VQ-VAE (Section 4, Appendix B). The paper states that Qwen3-VL was chosen for its "emergent capability for physical world understanding" and argues that "this model is representative of the capabilities of many contemporary LLMs" (this phrasing parallels the PaLM 2-S* justification in Section 4, though the exact wording differs). The Sparse 3D VQ-VAE is initialized from TRELLIS and pretrained on 500K objects following the TRELLIS data distribution. No experiments are conducted with alternative backbones (e.g., LLaMA-based VLMs, other 3D VAE initializations).

The consequence. Two aspects of the results are potentially backbone-specific in ways the paper does not characterize. First, Qwen3-VL's physical world understanding derives from its specific pretraining data mixture — a different VLM pretrained on a different image-text distribution might have weaker (or stronger) physical reasoning capabilities, changing the absolute performance and potentially the relative benefit of vision features vs. geometric tokens. The vision ablation in Table 3 shows that adding the rendered image contributes +14.3 percentage points to type accuracy — this contribution could be substantially smaller with a VLM that has less physical-world visual pretraining, or larger with a VLM that has more. A practitioner choosing between Qwen3-VL and an alternative VLM has no guidance on whether SIMART's performance is robust to this choice.

Second, the Sparse 3D VQ-VAE's reconstruction quality and the emergent zero-token behavior depend on the TRELLIS initialization and pretraining distribution. The paper documents that even without explicit zero-token supervision, "approximately two to four codebook entries consistently converge to represent the null distribution of empty voxels" (Appendix B) — but this observation was made on the specific VAE architecture and training data used. A different VAE architecture, codebook size, or pretraining dataset might not exhibit this emergent behavior, requiring explicit engineering of the zero-token mechanism rather than formalization of a naturally occurring phenomenon. The paper does not test whether the zero-token approach transfers to other VAE initializations.

What evidence exists in the paper. All main results (Tables 1–3) use Qwen3-VL-8B + TRELLIS-initialized Sparse VQ-VAE. The only model variation explored is the codebook size ablation (Table 4: 4,096 vs. 8,192 entries), which tests a VQ-VAE hyperparameter but not the backbone architecture or initialization. The MLLM scale comparison in Table 2 (SIMART's 8B vs. P3SAM + Qwen3-VL-235B) compares against a different architecture and pipeline, not against SIMART with a different backbone. No experiment tests SIMART with an alternative VLM (even a smaller/larger Qwen variant) or with a randomly-initialized (non-TRELLIS) VQ-VAE.

Mitigation status. Not addressed. The paper treats the backbone choice as a design decision justified by Qwen3-VL's capabilities, not as a variable to be validated. No future work is proposed on backbone robustness or transferability. The limitation is common in systems papers (where demonstrating one working configuration is the primary goal), but it means the paper's performance claims are contingent on the specific backbone and cannot be assumed to generalize to other MLLM architectures or VAE initializations without additional experiments.

7. Implications and Future Directions

How This Work Changes the Landscape

SIMART shifts the articulated asset creation problem from a multi-stage engineering challenge to a unified perception-and-reasoning task solvable by a single MLLM forward pass. This is more than an architectural improvement — it redefines what the field considers the core bottleneck. Prior to SIMART, the dominant assumption (implicit in Urdformer, CAGE, SINGAPO, and the multi-stage pipelines the paper critiques) was that part segmentation and kinematic inference are fundamentally different problems requiring different architectures: geometric segmentation calls for 3D convolutions or point-cloud processing, while kinematic reasoning calls for physical priors or learned category-level templates. SIMART demonstrates that both can be collapsed into a single autoregressive token-generation process, provided the tokenization makes 3D geometry processable by a language model. The consequence is that progress on articulated asset creation is now coupled to progress on MLLM pretraining and efficient 3D tokenization, rather than to the design of bespoke segmentation or joint-estimation modules.

This reframing has two immediate effects on what research directions become attractive. First, improving the MLLM's physical world understanding becomes a direct lever on articulation accuracy. The paper's vision ablation in Table 3 — where adding a single rendered image improves joint type accuracy from 0.794 to 0.937 — demonstrates that the MLLM's pretrained visual knowledge is currently the dominant performance driver, contributing more than any other single component. This implies that a stronger VLM backbone (with more physical-world pretraining data, better spatial reasoning capabilities, or explicit physics simulation objectives during pretraining) would directly improve articulation prediction without any changes to the 3D tokenizer or training pipeline. Second, 3D tokenization research now has a clear downstream metric — token count per object directly determines whether multi-part articulated objects can be processed at all, and reconstruction fidelity determines the geometric precision of joint axis and origin prediction. The paper's finding that the dense baseline fails with OOM errors on objects averaging just four parts (Table 3) establishes a concrete threshold: any viable 3D tokenization for articulated objects must produce well under ~1,000 tokens per object to fit within current MLLM memory budgets.

The paper also resolves the tension between geometric fidelity and kinematic accuracy that plagued prior work. Multi-stage pipelines forced a choice: methods that processed native 3D geometry (like Particulate) achieved good IoU and CD but could not match MLLM-based kinematic reasoning, while methods that leveraged MLLMs (like Articulate-Anything) achieved reasonable joint predictions but produced simplified geometry that failed to match the input (IoU of 0.202 vs. SIMART's 0.690). SIMART demonstrates that this tradeoff is not fundamental — it is an artifact of architectures that cannot simultaneously process raw 3D geometry and reason about physical structure. By embedding geometry as tokens in the same sequence as text and vision features, SIMART achieves state-of-the-art results on both geometric metrics (0.690 IoU, 0.087 CD) and kinematic metrics (0.928 type accuracy, 0.080 axis error), establishing that the two objectives are complementary rather than competing when processed jointly.

Perhaps most importantly, SIMART establishes that MLLM world knowledge can partially substitute for the extreme scarcity of articulated 3D training data. The 5,600 articulated models used for fine-tuning represent a tiny fraction of what would be needed to train a 3D-native model from scratch to recognize joint types across diverse geometries — yet SIMART achieves 0.831 type accuracy on AI-generated objects with geometries never seen during training (Table 1). This has a broader implication for 3D vision: tasks that require physical or functional reasoning but lack large-scale 3D supervision may be newly tractable by grounding MLLM knowledge in sparse 3D representations, without needing to collect 3D annotations at the scale that 2D pretraining provides.

Follow-Up Research This Work Enables

Direct comparison of joint vs. sequential generation to isolate the claimed benefit of unified processing. The paper's central architectural claim — that generating part voxels and URDF parameters in the same autoregressive sequence prevents error accumulation — is never isolated experimentally. A controlled follow-up would train two variants of SIMART with identical backbones, identical training data, and identical total compute: (1) the joint-generation version (as in the paper), and (2) a two-stage version where the first pass generates only per-part voxel tokens, and a second MLLM pass (or a separate head on the same frozen backbone) predicts joint parameters from those tokens. If joint generation provides a statistically significant improvement on SIMART-Bench metrics beyond what sequential generation achieves, the paper's central claim is validated. If the two variants perform similarly, the benefit comes from MLLM pretraining and tokenization rather than from simultaneity — an important clarification for practitioners choosing between simpler sequential and more complex joint architectures.

Cross-generator stress-test of SIMART-Bench generalization. The current SIMART-Bench uses 36 AI-generated objects from a single source (Hunyuan3D-V3.1). A rigorous generalization evaluation would construct an expanded benchmark with 100–200 objects drawn from 4–6 different 3D generation pipelines (e.g., Hunyuan3D, DreamGaussian, Seed3D, TRELLIS, Meshy, Zero-1-to-3 derivatives), annotated with the same human-in-the-loop pipeline, and would report per-generator performance breakdowns. This would answer two questions the current paper leaves open: (1) is SIMART's performance on Hunyuan3D objects representative of its performance on AIGC objects generally, or does it exploit Hunyuan3D-specific mesh characteristics? (2) which types of generator artifacts (non-manifold edges, inconsistent normals, thin structures, internal geometry) cause the largest performance degradation? The per-generator breakdown would guide practitioners on which generation pipelines SIMART is compatible with and would inform generator developers about which mesh properties matter for downstream articulation.

Quantitative simulation-validity metrics to replace geometric proxies. The paper evaluates "simulation-ready" quality using geometric and kinematic metrics (IoU, CD, axis error) that do not measure whether the generated URDF actually functions in a physics simulator. A follow-up would define a suite of simulation-based metrics and evaluate SIMART against baselines under these metrics. Concrete candidates: (1) load success rate — the fraction of generated URDFs that load into Isaac Sim or MuJoCo without parser errors; (2) kinematic trajectory error — the mean deviation between the ground-truth part trajectory (e.g., door swing arc, drawer slide path) and the simulated trajectory over the full joint range; (3) inter-penetration rate — the fraction of simulation timesteps where parts interpenetrate at joint limits; (4) manipulation success rate — the fraction of standardized robotic manipulation tasks (grasp handle, open to 50%, close) that succeed with the generated URDF. This would test whether SIMART's 0.080 axis error (Table 1) translates to sub-degree simulation accuracy or whether error amplification through the kinematic chain makes even small parameter errors practically significant. The paper's Isaac Sim deployment (Figure 6a) demonstrates feasibility; a systematic evaluation would quantify reliability.

Fine-tuning the Sparse VQ-VAE jointly with the MLLM to optimize tokenization for kinematic reasoning rather than reconstruction alone. The current Sparse VQ-VAE is pretrained independently on a 500K-object set with a pure reconstruction objective (binary cross-entropy on occupancy). The codebook entries (4,095 geometric tokens) thus optimize for surface reconstruction fidelity, not for discriminability of features relevant to articulation — the tokens might waste capacity on geometric detail that is irrelevant for joint prediction while merging features that distinguish revolute from prismatic boundaries. A follow-up could add a contrastive or task-aware loss during VQ-VAE fine-tuning that encourages codebook entries to form clusters corresponding to kinematic function (e.g., "planar surface with adjacent gap" tokens should be distinguishable from "continuous curved surface" tokens). Alternatively, the VQ-VAE could be fine-tuned end-to-end with the MLLM by backpropagating the language modeling loss through the codebook embeddings (using straight-through gradient estimation), allowing the token vocabulary to adapt to the specific geometric distinctions that the MLLM needs for articulation reasoning. The evaluation would compare: (1) reconstruction quality (MSE, CD) to ensure task-aware training doesn't degrade geometric fidelity, and (2) downstream articulation metrics (type accuracy, axis error) on SIMART-Bench to measure whether articulation-optimized tokens improve kinematic prediction.

Scaling analysis: how does articulation accuracy improve with MLLM backbone scale? The paper uses Qwen3-VL-8B and shows one comparison against Qwen3-VL-235B (in the P3SAM baseline, not in SIMART). A systematic scaling study would fine-tune SIMART with Qwen3-VL variants at multiple scales (e.g., 1B, 3B, 8B, 32B, 72B) while keeping the VQ-VAE and training data fixed, then plot articulation accuracy against model scale on both ID and AI-generated items. This would reveal: (1) whether the 0.831 vs. 0.817 type accuracy gap between SIMART and Particulate on AI-generated items widens or narrows with scale (if the gap grows with scale, MLLM world knowledge is the causal driver; if it plateaus, other factors dominate); (2) the scaling exponent for articulation accuracy — is it log-linear like many NLP benchmarks, or does it saturate early due to data scarcity?; (3) the minimum viable model scale for practical deployment (e.g., does the 1B variant retain acceptable accuracy for easy categories?). The P3SAM + Qwen3-VL-235B result in Table 2 (0.507 IoU vs. SIMART's 0.807) already hints that architecture matters more than scale for this task, but a controlled within-SIMART scaling study would be more informative.

Part-conditioned generation of novel articulation types not present in training data. The paper demonstrates that SIMART generalizes to novel geometries within known articulation categories (e.g., novel cabinet shapes with hinges and drawers). An open question is whether it generalizes to novel articulation types — mechanisms that don't appear in the 5,600-object training set. For example, would SIMART correctly identify a four-bar linkage, a scissor mechanism, or a cam-follower assembly if presented with a mesh of one? A diagnostic experiment would curate a small test set of objects with non-standard articulations (sourced from mechanical engineering datasets or manually modeled), evaluate SIMART zero-shot (without fine-tuning on these types), and characterize failure modes. Two outcomes are possible: (1) the MLLM's pretraining includes exposure to these mechanisms in diagrams or videos, enabling some generalization, or (2) the MLLM defaults to the nearest known articulation type (e.g., classifying a four-bar linkage as a series of revolute joints), revealing the boundary of what internet-scale pretraining covers. Either result defines the scope of MLLM-based articulation and identifies categories where 3D-specific training data remains essential.

Practical Applications and Downstream Use Cases

Scalable training environment generation for robotic manipulation. The paper's Isaac Sim deployment (Figure 6a) demonstrates automated conversion of static meshes into interactive URDFs, but the practical significance is throughput: SIMART can process the hundreds of thousands of static 3D assets now available from generative models (Objaverse alone contains 800K+ static objects) and convert them into diverse, articulation-aware training environments for robot policies. A robotics lab training a general-purpose cabinet-opening policy currently relies on perhaps dozens of manually annotated cabinet URDFs with limited geometric diversity. With SIMART, the same lab could generate thousands of procedurally varied cabinets — different sizes, handle positions, door configurations, hinge placements — all with automatically inferred kinematic structures. The paper's 0.690 IoU and 0.087 CD on in-domain items means the generated URDF geometry closely matches the input mesh, so the training diversity comes at minimal cost to physical realism. The 0.928 joint type accuracy means fewer than 1 in 13 generated cabinets would have an incorrect joint specification requiring manual correction. At 32 A100 GPUs for training (the paper's configuration), a deployed SIMART could amortize its training cost over millions of generated training assets, dramatically lowering the per-asset cost of simulation-ready content.

Click-to-functionalize for VR/AR content creation. The VR/AR application demonstrated in Figure 6b — where a user clicks on an image region, SAM3D generates a mesh, and SIMART functionalizes it — enables a workflow where non-expert users populate immersive environments with interactive objects. A VR architect designing a virtual office could capture a static 3D scan of a real filing cabinet, click "functionalize," and receive a cabinet with an opening drawer and swinging door within seconds, complete with realistic joint limits and material properties. The paper's 0.081 type accuracy (implied: 1 − 0.919 ≈ 0.081 error rate, or 0.928 accuracy as directly reported) and 0.080 axis error on in-domain items suggest that for common furniture categories (cabinets, drawers, boxes), the generated articulation would be correct and physically plausible more than 90% of the time — sufficient for consumer VR/AR where exact millimeter-level precision is not required but physically impossible configurations (doors swinging through frames) would break immersion. The key enabler is that SIMART requires no per-category training or template fitting; the same model handles any object the user might capture or generate.

Automated annotation for articulated 3D datasets. The paper acknowledges (Section 6) that "the scarcity and inconsistent quality of existing articulated datasets remain a primary limitation for open-world generalization" and proposes using SIMART "as a foundational tool to generate pre-verified articulation predictions, thereby accelerating the data-annotation loop." Concretely, SIMART could process a large collection of unannotated static meshes (e.g., the full Objaverse or a crawl of 3D marketplaces), produce initial URDF predictions, and flag low-confidence outputs (e.g., where the MLLM's token probabilities for joint type are uncertain, or where multiple plausible kinematic interpretations exist) for human review. A human annotator using the specialized Web UI described in SIMART-Bench's annotation pipeline (Section 4, Appendix D) would then correct only the flagged predictions, reducing annotation time from ~30 minutes per object (full manual annotation) to perhaps ~5 minutes per object (verification and correction of automated predictions). At SIMART's 0.928 type accuracy, approximately 7% of objects would require joint type corrections; the remaining 93% would need only verification. This human-in-the-loop pipeline could scale articulated dataset creation from the current ~2,000 PartNet-Mobility models to tens or hundreds of thousands of models, directly addressing the data scarcity that the paper identifies as the primary field-level bottleneck. The resulting expanded dataset would in turn enable training stronger articulation models (including future SIMART variants) and category-level generative methods that currently overfit due to limited training diversity.