Part-X-MLLM: Part-aware 3D Multimodal Large Language Model

🎯 Pitch

Part-X-MLLM unifies 3D generation, editing, and Q&A by having a language model output structured, executable programs of part bounding boxes and edit commands—rather than just text or geometry. This symbolic planning frontend decouples semantic reasoning from geometric synthesis, enabling state-of-the-art part-level grounding. The dual-encoder architecture, by separating structure from appearance, boosts bounding box accuracy by over 7 IoU points and lifts Q&A metrics by 18+ SBERT over the best prior model.

1. Executive Summary

Part-X-MLLM introduces a native 3D part-aware multimodal large language model that unifies diverse 3D tasks—generation, editing, and question answering—by formulating them as a single, geometry-aware program in a structured grammar of part-level bounding boxes, semantic descriptions, and edit commands (e.g., emitting <adds>/<adde> tokens with quantized box coordinates to drive a synthesis module). The system uses a dual-encoder architecture that decouples structure (XYZ+normals) from appearance (RGB) to avoid representational conflicts, instruction-tuned on a large-scale part-centric dataset of ~86k objects and evaluated on the introduced UniPart-Bench spanning 11 task families. For bounding box generation, the dual-encoder design improves BBox IoU by a substantial margin over a single-encoder variant (+7.06 absolute points), while on part-level Q&A the model achieves gains of +18.7 SBERT, +25.5 SimCSE, and +21.3 BLEU-1 over the strongest baseline. The programmatic interface serves as a language-native, model-agnostic control surface that delegates geometric synthesis to off-the-shelf engines (OmniPart for generation, Nano3D/VoxHammer for editing), establishing that structured, executable token sequences can unify part-level understanding and manipulation only when the planning stage is decoupled from the geometric execution backend.

2. Context and Motivation

The Core Problem: Structural Opaqueness in 3D Understanding and Generation

The fundamental gap this paper addresses is what the authors call "structural opaqueness" in modern 3D AI systems. Most existing models treat 3D objects as monolithic, static forms — a single holistic entity that can be captioned, generated, or edited as an indivisible whole. But real-world objects don't work that way: a chair is an assembly of legs, a seat, and a backrest; a lamp has a base, a pole, and a shade; a mechanical device might contain dozens of functionally distinct subcomponents. These parts have persistent identity (this specific leg versus that specific leg), semantic meaning (a handle is for grasping), and spatial relationships (the seat connects to all four legs at specific positions).

Without a language model that natively understands and addresses these parts, a wide range of practical 3D workflows remain inaccessible or require brittle, task-specific workarounds. The paper gives concrete examples of what the gap means in practice:

An artist who wants to "swap the handles without touching the body" of a 3D asset cannot simply issue a natural-language command — they must manually select vertices, apply masks, or work through a specialized editing interface that knows nothing about parts.
A roboticist reasoning about "graspable subcomponents" needs to identify which parts of an object afford certain interactions, which requires a structured understanding of part semantics tied to spatial locations.
Downstream pipelines for animation, simulation, and procedural generation rely on consistent, addressable structure — if a character model doesn't know where its left arm begins and ends, you cannot rig it for animation; if a vehicle model doesn't label its wheels, you cannot simulate its motion.

The problem is therefore both practical (affecting real creative and engineering workflows) and representational (existing models lack the vocabulary and architecture to express part-level knowledge in an executable way). The paper argues that what's missing is not a better generator or a better segmenter, but a unifying linguistic interface that can express part-level reasoning, grounding, and manipulation commands — and that this interface should be natively 3D and executable, not simply descriptive.

Why This Problem Matters Now

The timing of this work is driven by several converging factors.

Generative 3D has matured, but part awareness has stalled. Recent years have seen dramatic advances in holistic 3D shape generation — models like TRELLIS (Zhang et al., 2024), Hunyuan3D (Zhao et al., 2025), and Clay (Lai et al., 2025) can produce high-fidelity 3D assets from text or images. Yet these systems output shapes as monolithic entities with no explicit part structure. As the paper notes, they "expose limited semantic addressability" — you can prompt them to generate "a wooden chair," but you cannot subsequently ask "make the left front leg thinner" without re-generating the entire asset or using a separate editing tool. The generation capability has outpaced the interface for interacting with generated content.

Editing tools exist but lack a language-native planning layer. The paper surveys a range of 3D editing methods: optimization-based approaches like Instruct-NeRF2NeRF (Haque et al., 2023) that edit multi-view images and optimize a NeRF; feed-forward latent editors like Shap-Editor (Chen et al., 2024); and training-free volumetric editors like Nano3D (Ye et al., 2025) and VoxHammer (Li et al., 2025). These tools are powerful on the geometry side, but the paper makes a crucial distinction: they are "tool-side" solutions rather than "language-native frontends." They require the user to provide masks, bounding boxes, or other spatial specifications alongside text, rather than understanding language instructions about parts and automatically computing the spatial bounds. The paper characterizes this gap sharply: "these methods are typically tool-side: they do not provide a language-native model that reasons about parts and emits executable edit programs with precise spatial grounding."

Scene-level 3D MLLMs handle dialogue but not parts. The paper positions itself against a wave of recent 3D multimodal LLMs — PointLLM (Xu et al., 2024), 3D-LLM (Hong et al., 2023), GPT4Point (Qi et al., 2024), ShapeLLM (Qi et al., 2024), and scene-level models like Chat-Scene (Huang et al., 2024) and LL3DA (Chen et al., 2024). These systems align point clouds with language to perform captioning, question answering, and dialogue. However, the paper identifies a specific limitation: they "largely treat objects as monolithic and lack persistent part identifiers, grounded references, and executable outputs." Even when these models describe a chair, they say "a wooden chair with four legs and a curved backrest" — descriptive text, not executable tokens that a geometry engine can use. The grounding is missing: there are no persistent references (like bounding box tokens) that connect the linguistic mention of "left front leg" to a specific spatial region in the point cloud that a downstream tool could act upon.

Part pipelines are fragmented across 2D and 3D. The paper's most extensive critique targets existing part-generation and part-decomposition approaches. These fall into two camps, both problematic:

2D-driven pipelines that lift multi-view segmentations to 3D. Methods like Part123 (Liu et al., 2024), PartGen (Chen et al., 2025), SAMPart3D (Yang et al., 2024), PartField (Liu et al., 2025), and HoloPart (Yang et al., 2025) use 2D segmentation models (like SAM) on rendered views, then project the results back to 3D. The paper identifies two failure modes: these approaches are "prone to view inconsistencies" (what appears segmented from one camera angle may not align with segmentations from another), and they have "weak 3D constraints" (2D masks don't naturally respect 3D geometry — a part that appears separated in a 2D view may actually be connected in 3D space).
Native 3D part generators that work directly with 3D representations but lack a unified language interface. AutoPartGen (Chen et al., 2025) generates parts autoregressively; BANG (Zhang et al., 2025) produces exploded-view decompositions; OmniPart (Yang et al., 2025) unifies part generation through autoregressive box planning followed by TRELLIS-based synthesis; X-Part (Yan et al., 2025) generates parts from semantics via vector set representations. While each of these makes contributions on the geometry side, the paper argues they share a critical weakness: none provides a "unified language interface" that can handle understanding, naming, grounding, and editing through a single consistent vocabulary and grammar.

The Specific Gap: Four Missing Capabilities

The paper crystallizes the gap into a clear specification. No existing model simultaneously achieves all four of the following:

Understands and names parts — can parse a point cloud and identify that "this region is the left armrest," producing both the semantic label and the spatial reference.
Grounds references to persistent bounding boxes — connects linguistic mentions to specific, coordinate-quantized spatial regions using a vocabulary of bounding box tokens (not just descriptive text that a human would read).
Compiles executable add/delete/modify programs — given a natural language instruction like "remove the handle," produces a structured token sequence (e.g., <dels> <box_token_sequence> <dele>) that a separate geometry engine can directly execute without parsing or interpreting.
Controls semantic granularity — supports transitioning from coarse labels (e.g., "leg") to fine-grained descriptions (e.g., "a cylindrical wooden leg with a tapered bottom and decorative groove") under a single interface, without requiring a different model architecture or training procedure for each granularity level.

The paper argues that these four capabilities must coexist in a single model because real-world 3D workflows demand them in combination: an artist editing a chair needs the model to name parts (capability 1), ground them to spatial locations (capability 2), produce machine-executable edit commands (capability 3), and optionally surface either coarse labels for quick selection or fine descriptions for precise specification (capability 4). The absence of any one capability breaks the workflow.

Why Prior Approaches Fall Short Individually

The paper's critique of existing work is organized around this capabilities checklist, showing how each prior approach misses at least one requirement:

Scene-level 3D MLLMs (PointLLM, 3D-LLM, GPT4Point, ShapeLLM): These achieve understanding and naming (capability 1) but fail on grounding (capability 2) and executability (capability 3). Their outputs are natural language captions or answers — a human can read them, but no machine can use them to locate and manipulate specific parts in 3D space.

Geometry-oriented generators (TRELLIS, Hunyuan3D, 3DShape2VecSet): These handle executable generation but lack the linguistic intelligence for understanding and naming. They produce 3D shapes from text prompts but do not output a parseable part structure or support subsequent language-driven manipulation.

2D-lifting part pipelines (Part123, SAMPart3D, PartField): These can produce part segmentations and even naming, but their grounding is unreliable due to view inconsistency ("weak 3D constraints"), and they don't produce executable programs — the segmentation is the output, not a structured token command.

Native 3D part generators (AutoPartGen, BANG, OmniPart, X-Part): These come closest to the complete picture but still fail the unified interface test. OmniPart, for example, does autoregressive box planning but its planning module is tightly coupled to the TRELLIS synthesis backend — you cannot use a different geometry engine without retraining. The paper's key design insight is that this coupling is unnecessary and limiting: by separating the planning stage (producing token programs) from the synthesis stage (executing those programs with any compatible geometry engine), the system becomes modular, extensible, and language-native.

The Disentanglement Insight: Planning vs. Execution

The paper's core intellectual move is to reframe the problem as a separation of concerns. Instead of building an end-to-end model that both plans the part decomposition AND synthesizes the final geometry, the paper argues that the planning problem (given a point cloud and text, what are the parts and their spatial extents?) is fundamentally a language modeling problem that can be solved by an LLM with appropriate 3D encoders and a specialized output grammar. The execution problem (given bounding boxes and part descriptions, generate high-fidelity 3D geometry) is a geometry synthesis problem best left to specialized engines.

This decomposition has several consequences that the paper explicitly positions as advantages:

Model-agnostic backend: Because the output is a sequence of tokens in a well-defined grammar, any geometry engine that can consume bounding boxes and part descriptions can serve as the backend. The paper uses OmniPart for generation and Nano3D/VoxHammer for editing, but these are pluggable — a better geometry engine in the future can be swapped in without retraining the planning model.
Auditable intermediate representations: The structured token programs are human-readable and machine-verifiable. If the system produces an incorrect part decomposition, you can inspect the bounding box tokens to see exactly which spatial region was misidentified, rather than debugging an opaque learned representation.
Stable part identity: Because parts are referenced by quantized coordinate tokens (six tokens per axis-aligned bounding box), the same part can be referenced consistently across multiple steps of a complex instruction. This is what the paper calls "persistent references" — the planning language gives parts stable identifiers that survive across different tasks (the same leg bounding box token serves in captioning, Q&A, and editing).
Single interface for heterogeneous tasks: The same model architecture, same training procedure, and same output grammar handle understanding (Q&A about parts), generation (producing bounding boxes for a synthesis engine), and editing (emitting delete/modify/add programs). This is the "unification" claim: 11 different task families are all reformulated as instruction-following problems where the target output is always a program in the same grammar.

Positioning Relative to the Broader Landscape

The paper situates itself at the intersection of several research threads but explicitly distinguishes its contribution from each:

Relative to 3D MLLMs: The paper positions Part-X-MLLM not as a replacement for scene-level 3D MLLMs but as a complementary capability at a different level of abstraction. Scene-level models handle holistic understanding across large environments; Part-X-MLLM handles fine-grained, part-level understanding within individual objects. The novel contribution is the part-centric framing and the executable output grammar — existing 3D MLLMs produce descriptive text, while Part-X-MLLM produces programs that can be executed by downstream tools.

Relative to part generation methods: The paper positions itself as providing the missing planning layer for existing part generation pipelines. OmniPart, for example, already has a box planning component, but it's internally coupled to the TRELLIS synthesis module. Part-X-MLLM generalizes this concept: a single LLM-based planner can serve as the language frontend for multiple different geometry backends, and the planning is learned in a task-agnostic way through instruction tuning rather than being built into a specific generation architecture.

Relative to 3D editing tools: The paper positions Part-X-MLLM as providing automatic localization for editing operations. Existing editing tools require the user to specify the spatial region to edit (through masks, bounding boxes, or view selection). Part-X-MLLM takes only the language instruction ("remove the handle") and the point cloud, and automatically generates the bounding box program that localizes the edit. This is a "language-native frontend" — the model does the work of understanding which spatial region corresponds to "the handle" and produces the machine-readable specification.

Relative to programmatic 3D representations: The paper acknowledges related work on programmatic scene abstractions, such as Scene-LLM (Fu et al., 2024) and the Scene Language (Zhang et al., 2025), which also use structured representations to describe scenes. The distinction is in granularity (parts within objects versus objects within scenes) and execution (Part-X-MLLM's programs are explicitly designed to drive geometry engines, not just to represent scenes).

The Architectural Hypothesis: Dual Encoders Avoid Representational Conflict

Beyond the systems-level contribution, the paper advances a specific architectural hypothesis that motivates the dual-encoder design. The claim, tested in the ablation study (Table 2, discussed in detail in the experimental sections), is that forcing a single encoder to handle both geometric structure (XYZ coordinates, surface normals) and visual appearance (RGB color) creates a representational conflict. The intuition is that geometry and color carry fundamentally different kinds of information with different invariance properties:

Structural information (XYZ + normals) encodes the shape, topology, and spatial extent of objects. It is invariant to texture and color — a wooden chair and a metal chair with identical shape should produce similar structural features. The relevant computation is about spatial relationships: which points form a contiguous surface, where does one part end and another begin, what is the local curvature and orientation.
Appearance information (RGB) encodes material properties, color patterns, and texture. It is invariant to shape in many cases — a red surface and a green surface with identical geometry should produce different appearance features but could share structural features.

The paper hypothesizes that when these two modalities are fused in a single encoder (as in the single-encoder baseline that consumes a unified XYZ+RGB point cloud), the encoder must learn representations that simultaneously encode geometric structure and visual appearance. This creates tension: two chair legs of different colors should have different appearance features but nearly identical structural features. A single encoder may struggle to disentangle these signals, leading to worse performance on tasks that require precise geometric reasoning (like bounding box prediction) or tasks that require appearance-based disambiguation (like distinguishing between visually identical structural parts that differ only in color).

The dual-encoder design avoids this by processing structure and semantics in separate pathways, each specialized for its modality. The structure encoder, pretrained extensively on geometry-only data (RGB-less point clouds) in Stage 1, develops robust part decomposition capabilities purely from shape. The semantic encoder, introduced in Stage 2, learns to associate color patterns with textual descriptions. The LLM decoder then fuses these two information streams, using structural features for spatial localization and semantic features for part naming and description.

The paper positions this as a design choice motivated by a representational hypothesis, not simply an architecture that happens to work. The ablation study (Table 2) is framed as testing this hypothesis: if the dual-encoder consistently outperforms the single-encoder variant across diverse tasks (pure geometry tasks like box listing AND language-intensive tasks like part Q&A), that supports the claim that representational conflict is real and damaging. The paper reports the results as confirming this: +7.06 BBox IoU improvement on box listing, and uniform gains across all metrics for part Q&A and multi-part grounding.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

Part-X-MLLM is a 3D multimodal large language model that acts as a universal planning frontend: it takes an RGB point cloud and a natural language instruction, and outputs a single, structured token sequence — part bounding boxes with coordinates, part names, descriptions, and edit commands — that a separate geometry engine can execute directly. The problem it solves is structural opaqueness in existing 3D systems: current models either describe objects holistically without part-level grounding, or generate/edit geometry without a language-native interface for reasoning about parts. The shape of the solution is a decoupling of symbolic planning from geometric synthesis — the LLM handles the language understanding, part reasoning, and spatial grounding (producing an executable program), while off-the-shelf geometry modules handle the actual mesh/voxel/NeRF generation (executing that program).

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a planning→execution pipeline:

Dual 3D Encoders — a Structure Encoder processes raw point cloud geometry (XYZ coordinates + surface normals) to extract structural tokens; a parallel Semantic Encoder processes RGB color information to extract appearance tokens. These two pathways are architecturally identical but process different input channels, avoiding representational conflict.
Text Tokenizer — the standard Qwen 2.5 VL tokenizer converts the natural language prompt (e.g., "remove the handle" or "describe all parts") into text tokens. The vocabulary is extended with task-specific special tokens (e.g., <boxs>, <boxe>, <adds>, <dels>, <mods>) during Stage 2 training.
Autoregressive Decoder (Qwen 2.5 VL) — a decoder-only transformer that takes the fused sequence of structural tokens, semantic tokens, and text tokens as input. It is trained to autoregressively generate a program-like output — a token sequence in the structured planning language that encodes bounding boxes (six quantized coordinate tokens wrapped in <boxs>...<boxe>), part descriptions, and edit operators (e.g., <dels> [box] <dele>).
Structured Planning Language — a formal grammar defining the vocabulary and syntax of the model's output. It specifies special token pairs for part representation (<boxs>/<boxe> enveloping six coordinate tokens) and edit operations (<adds>/<adde> for addition, <dels>/<dele> for deletion, <mods>/<mode> for modification), plus free text for part names and descriptions.
Downstream Geometry Interfaces — external, off-the-shelf modules that consume the structured program and produce geometry. For generation, the output bounding boxes and part text are passed to OmniPart's synthesis module. For editing, the program and derived cuboid masks are passed to Nano3D or VoxHammer.

Information flow: A point cloud enters the dual encoders → structural and semantic tokens are extracted in parallel → these tokens are fused with the text prompt's tokens into a single sequence → the autoregressive decoder generates a program token sequence → the program is parsed by a downstream geometry module → geometry module produces the final 3D asset (mesh, edited shape, or answer with grounded BBox references).

3.3 Roadmap for the Deep Dive

First, the dual-encoder architecture and why it separates structure from semantics — this is the foundation that enables all subsequent capabilities and the key design choice tested in ablations.
Second, the structured planning language (its grammar, special tokens, coordinate quantization scheme, and edit operators) — this is the "executable interface" that unifies heterogeneous tasks.
Third, the multi-stage training curriculum (Stage 1 geometry-only BBox pretraining, Stage 2 full instruction tuning) — this explains how the model acquires part decomposition ability and then aligns it with language.
Fourth, the dataset construction and training data pipeline (model-assisted labeling, 11 task templates, and how raw annotations become instruction-following samples) — this is the data engine that enables the model to learn the planning language.
Fifth, the downstream geometry interfaces and end-to-end task realization — this shows how the abstract token programs become concrete 3D outputs for generation, editing, and understanding.
Sixth, the semantic granularity control mechanism (CLIP-based part clustering with DBSCAN) — this is a post-hoc capability enabled by the box-and-text representation that requires no model retraining.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that 3D part-level understanding, generation, and editing can be unified under a single structured output grammar, and that decoupling the planning stage (LLM with dual encoders) from the execution stage (geometry engines) creates a language-native, model-agnostic control surface for 3D assets.

Dual-Encoder Architecture: Separating Structure from Semantics

The paper's central architectural innovation is the dual-pathway encoding of 3D point clouds — one encoder for geometric structure, one for visual appearance — feeding into a shared autoregressive decoder. This design is motivated by a representational hypothesis: forcing a single encoder to handle both geometry (XYZ coordinates, surface normals) and color (RGB) creates a conflict because these modalities carry fundamentally different types of information with different invariance properties.

Structure Encoder. This pathway processes only geometric information — the raw point cloud's XYZ coordinates and surface normals, with no color data. The input is a fixed-size point cloud of shape (40960, 6) containing (x, y, z) coordinates and surface normals for 40,960 points. The encoder is initialized from the Hunyuan 2.1 3D Shape VAE Encoder and downsamples features by a factor of 20× to produce a latent representation of length 2048 tokens. This encoder is domain-specialized through Stage 1 pretraining (discussed below) to reliably disentangle and localize part bounding boxes purely from shape information. The intuition: a chair leg and a table leg have similar structural signatures (elongated vertical cylinders) regardless of their color, and this encoder learns to recognize those structural primitives independent of appearance.

Semantic Encoder. This pathway processes appearance information — the same point cloud but with (x, y, z) coordinates and (r, g, b) color data. The input shape is (10240, 6) (fewer points than the structure encoder: 10,240 vs. 40,960). The semantic encoder has the same architecture as the structure encoder but is trained from scratch during Stage 2 instruction tuning (it is not pretrained). Its job is to capture visual appearance — material properties, color patterns, textures — that distinguish structurally similar parts. The paper gives the example of "two identical chair legs of different colors": the structure encoder sees them as nearly identical, but the semantic encoder produces distinct features for the red leg versus the blue leg.

Why 40,960 vs. 10,240 points? The paper does not explicitly justify the different point counts, but the implication is that structure requires denser sampling (more points per surface to capture curvature, edge boundaries, and connectivity), while appearance can be captured at lower resolution because color varies more slowly across surfaces. This is a practical engineering choice: the structure encoder is pretrained on a large geometry-only corpus at the higher resolution, and the semantic encoder is trained from scratch at a lower resolution to save computation during Stage 2.

Token Fusion. The outputs from both encoders — 2048 structural tokens and 1024 semantic tokens (inferred from the 10× fewer input points and 20× downsampling) — are concatenated with the text prompt tokens into a single sequence. This fused sequence feeds into the autoregressive decoder (the Qwen 2.5 VL transformer). The decoder's attention mechanism can then learn to attend to structural tokens when spatial reasoning is needed (e.g., "where is the leftmost leg?") and to semantic tokens when appearance-based disambiguation is needed (e.g., "which leg is red?").

Ablation justification. The single-encoder baseline (Table 2) that the dual design is compared against consumes a unified point cloud with fused geometry (XYZ) and color (RGB) — i.e., a standard (N, 6) point cloud with XYZRGB. The paper reports that the dual-encoder design improves BBox IoU by +7.06 absolute points on pure box listing (a geometry-heavy task) and provides uniform gains across all metrics on part Q&A and multi-part grounding (language-intensive tasks). This is presented as evidence for the representational conflict hypothesis: when a single encoder must represent both shape and color in the same feature space, the model performs worse on both geometry tasks and language tasks, because the features are entangled.

What the dual-encoder is NOT. It is important to clarify that the paper does not claim the structure encoder is "blind to color" or the semantic encoder is "blind to shape." Both encoders receive XYZ coordinates; the semantic encoder additionally receives RGB. The structure encoder processes XYZ + normals, which is purely geometric but still carries information about part boundaries and surface orientation. The semantic encoder processes XYZ + RGB, so it has access to some geometric information (the coordinates) alongside color. The separation is therefore about what each encoder specializes in, not about information being strictly partitioned. The training procedure (Stage 1 pretraining without RGB, Stage 2 fine-tuning with RGB) enforces this specialization.

Structured Planning Language: The Executable Grammar

The planning language is a formal token grammar that defines the model's output space. Every task — whether it's listing parts, answering questions, or generating edit programs — is reformulated as producing a token sequence in this grammar. This is the mechanism that unifies the 11 task families under a single interface.

Special tokens. The vocabulary is extended with task-specific tokens that serve as structural delimiters and operators:

Box tokens: <boxs> (box start) and <boxe> (box end) wrap a six-token coordinate sequence representing a single axis-aligned bounding box. The six tokens encode (x_min, y_min, z_min, x_max, y_max, z_max) in that order.
Edit operator tokens: <adds>/<adde> (add operation), <dels>/<dele> (delete operation), and <mods>/<mode> (modify operation). These function as structured delimiters: the opening token marks the start of an edit command, the content specifies the target bounding box(es) and optional new text, and the closing token marks the end.

Coordinate quantization. Continuous 3D coordinates are discretized into a fixed vocabulary of 128 special tokens. This is necessary because the LLM operates over a discrete token vocabulary — it cannot output continuous floating-point numbers directly. The quantization scheme maps each continuous coordinate $x \in [-1, 1]$ (the canonical 3D volume) to a discrete bin index:

$q(x) = \text{round}\!\left(\frac{x + 1}{2}(K - 1)\right)$

where $K = 128$ is the number of quantization bins, $x$ is the continuous coordinate in the normalized range $[-1, 1]$ , and $q(x)$ is the integer bin index in $\{0, 1, \ldots, 127\}$ .

What it computes: the affine transformation $\frac{x+1}{2}$ maps from $[-1, 1]$ to $[0, 1]$ , then multiplication by $(K-1) = 127$ maps to $[0, 127]$ , and rounding produces a discrete bin index. The reverse mapping (dequantization) reconstructs an approximate continuous coordinate:

$\tilde{x} = 2 \cdot \frac{q(x)}{K - 1} - 1$

This is simply the inverse: integer bin index $q(x)$ scaled back to $[-1, 1]$ .

Why this form: Uniform quantization over the normalized volume is the simplest scheme that maps continuous coordinates to a discrete, learnable token vocabulary. The choice of $K = 128$ balances spatial resolution (each bin spans $\frac{2}{127} \approx 0.0157$ units in normalized coordinates) against vocabulary size (128 tokens per coordinate × 6 coordinates = 768 potential box token combinations in the vocabulary, though in practice the model learns a structured embedding manifold for these tokens, as shown in the t-SNE visualization in Figure 10). Fewer bins would reduce spatial precision for bounding box prediction; more bins would increase vocabulary size and make the autoregressive prediction task harder.

Serialization order. Parts in a list are deterministically ordered by $(q(z_{\min}), q(y_{\min}), q(x_{\min}))$ — i.e., sorted by their minimum Z, then Y, then X coordinates. This deterministic ordering is crucial for reproducibility: given the same point cloud, the model should produce the same box ordering every time, which enables consistent evaluation and downstream processing. Without a canonical ordering, the model might learn arbitrary permutations that would confuse both training and evaluation.

Example program structures:

Pure box listing (Type 0): Input: "Detect all bounding boxes in this point cloud." Output: <boxs> t1 t2 t3 t4 t5 t6 <boxe> <boxs> t7 t8 t9 t10 t11 t12 <boxe> ... (one box per part, no text).
Multi-part grounding with coarse text (Type 1): Output: <boxs> tokens <boxe> leg <boxs> tokens <boxe> seat <boxs> tokens <boxe> backrest (box tokens followed by part name, per part).
Multi-part grounding with fine text (Type 2): Output: "A wooden dining chair with four turned legs and a cushioned seat. <boxs> tokens <boxe> The left front leg is a turned wooden cylinder with a tapered bottom. <boxs> tokens <boxe> The seat is a padded rectangular cushion upholstered in beige fabric." (overall description first, then per-part box tokens with detailed descriptions).
Deletion program (Type 8): Input: "Please remove the handle from this object." Output: <dels> <boxs> tokens <boxe> <dele> (delete operator wrapping the target part's box).
Addition program (Type 10): Input: "Add a spherical glass knob to this lamp." Output: <adds> <boxs> tokens <boxe> spherical glass knob <adde> (add operator with target box and part description).

Why a grammar rather than free-form text: The structured grammar provides three concrete benefits that the paper claims. (1) Stable part identity and grounding: because each part is referenced by persistent token sequences (the six quantized coordinates in <boxs>...<boxe>), the same part can be referenced consistently across different tasks — the same bounding box token sequence serves for captioning, Q&A, and editing commands. This would be impossible with free-form text descriptions like "the left front leg" because parsing such text into spatial coordinates is a separate unsolved problem. (2) Controllable semantic granularity: the box-and-text representation preserves both the spatial extent (the box tokens) and the semantic description (the text tokens) for each part, enabling post-hoc merging of parts based on text similarity without losing spatial information (see semantic granularity control below). (3) Model-agnostic execution: because the output is a well-defined token grammar, any downstream module that knows how to parse <boxs>...<boxe> and edit operators can consume the model's output. This decouples the planning interface from specific geometry backends.

Multi-Stage Training Curriculum

The model is trained in two stages with a carefully designed progression: first, learn to decompose shapes into bounding boxes from pure geometry; then, align this decomposition capability with language, appearance, and task-specific grammar.

Stage 1: Geometry-Only BBox Pretraining. The goal of Stage 1 is to domain-specialize the structure encoder to reliably decompose 3D shapes into part-level bounding boxes — before any language, color, or task-specific grammar is introduced. This stage operates entirely on geometry and establishes the spatial reasoning foundation that all subsequent capabilities build upon.

Architecture during Stage 1: The structure encoder (Hunyuan 2.1 3D Shape VAE Encoder) is paired with a lightweight autoregressive decoder. This decoder is NOT the Qwen 2.5 VL model — it is a small, purpose-built transformer whose sole task is to predict part-level bounding boxes from the structure encoder's latent features. The output is a sequence of bounding box tokens (the six-coordinate token format, but without any text, edit operators, or other grammar tokens).

Training data: 3.6 million objects, each providing a fixed-size RGB-less point cloud of shape (40960, 6) — 40,960 points with (x, y, z) coordinates and surface normals, no color information. Each object is annotated with axis-aligned part bounding boxes (AABBs) in the quantized coordinate format.

Training procedure: The lightweight decoder is trained for 10 epochs on this geometry-only corpus. The optimization uses AdamW with learning rate $2 \times 10^{-4}$ and batch size 256 (Section 4.1). Training runs on 64 A100 GPUs for 2 days.

What happens after Stage 1: The lightweight decoder is discarded — it served only as a training tool to force bounding box knowledge into the structure encoder. The structure encoder weights are retained and frozen for Stage 2. This means that during instruction tuning, the structure encoder's representations are fixed — they encode part decomposition knowledge distilled from 3.6M objects of pure geometry training.

Why this design: The paper does not explicitly justify discarding the lightweight decoder, but the reasoning is implied by the architecture: the lightweight decoder is a domain-specific tool for the pretraining task (bounding box prediction from geometry). For Stage 2's diverse instruction-following tasks (which require language generation, edit program construction, and question answering), a much more powerful language model (Qwen 2.5 VL) is needed. The pretrained structure encoder serves as a frozen feature extractor that provides high-quality structural tokens to this more powerful decoder, without requiring the language model to learn geometry decomposition from scratch.

Stage 2: Full Instruction Tuning with a Dual-Encoder LLM. The goal of Stage 2 is to align the pretrained structure encoder, a newly introduced semantic encoder, and a powerful language model with the task-specific grammar, enabling the model to follow diverse instructions and produce executable programs.

Architecture during Stage 2: The full Part-X-MLLM architecture is assembled:

Structure Encoder: the pretrained encoder from Stage 1, frozen (weights not updated).
Semantic Encoder: architecturally identical to the structure encoder but initialized from scratch, processes (10240, 6) point clouds with (x, y, z) and (r, g, b) data.
Qwen 2.5 VL Decoder: the autoregressive transformer that takes the fused token sequence as input and generates the program output. The original Qwen 2.5 VL token embeddings are frozen (preserving the pretrained language knowledge).
New Special Token Embeddings: embeddings for the newly added grammar tokens (<boxs>, <boxe>, <adds>, <adde>, etc.) are initialized randomly and trained from scratch.

What is trained vs. frozen: During Stage 2, the paper trains only the following components:

The Semantic Encoder — learns to produce appearance tokens that complement the frozen structural tokens.
The autoregressive transformer layers of the Qwen 2.5 VL decoder — adapts the pretrained language model to the 3D planning grammar and dual-encoder conditioning.
The embeddings for newly added special tokens — learns the representation of the grammar vocabulary.

The frozen components are:

The Structure Encoder (preserving geometry decomposition from Stage 1).
The original Qwen 2.5 VL token embeddings (preserving the pretrained language knowledge for standard text tokens).

Why this selective freezing: This training strategy has two purposes. First, it preserves the structure encoder's hard-won geometry decomposition capability — fine-tuning it on language-rich data could dilute its spatial reasoning, since the language supervision signal might dominate the gradient updates and cause the encoder to "forget" how to decompose shapes. Second, it preserves the Qwen 2.5 VL model's general language capabilities — freezing the original token embeddings prevents catastrophic forgetting of pretrained language knowledge. The paper's limitation section explicitly notes that "fine-tuning on 3D tasks may reduce the base LLM's general language capabilities," and this selective freezing is the mitigation strategy. Only the transformer layers (which learn attention patterns and cross-modal fusion) and the new token embeddings (which have no pretrained equivalent) are updated.

Training data: The 85,771-object part-centric dataset described in Appendix A.6, converted into instruction-following samples across 11 task templates (Types 0–10). The dataset provides point clouds (with both geometry and RGB), natural language prompts (from template pools), and target outputs in the structured grammar.

Training procedure: AdamW optimizer with learning rate $5 \times 10^{-8}$ and batch size 64 (Section 4.1). Training runs on 64 A100 GPUs for 5 days. The dramatically lower learning rate compared to Stage 1 ( $5 \times 10^{-8}$ vs. $2 \times 10^{-4}$ ) reflects the fact that Stage 2 is fine-tuning a large pretrained model (Qwen 2.5 VL) rather than training a small decoder from scratch — a high learning rate would destabilize the frozen token embeddings and cause catastrophic forgetting.

The learning rate design choice: The four-order-of-magnitude difference between Stage 1 and Stage 2 learning rates ( $2 \times 10^{-4}$ vs. $5 \times 10^{-8}$ ) is non-obvious but well-motivated. Stage 1 trains a small, randomly initialized decoder on a single narrow task (bounding box prediction), where a high learning rate enables fast convergence. Stage 2 fine-tunes a massive pretrained language model with frozen embeddings — the transformer layers need to learn cross-modal attention and grammar production without overwriting the language prior, which requires very small, careful gradient steps. If the learning rate were higher, the decoder layers might rapidly adapt to the new 3D grammar vocabulary at the cost of losing general language capability.

Dataset Construction and Task Templates

The training data pipeline is a critical component, since the model's ability to produce structured programs depends entirely on the quality and diversity of the instruction-following examples it is trained on. The paper constructs a dataset of 85,771 unique 3D objects with an average of 23 parts per object, then instantiates 11 task templates to create diverse training samples.

Model-assisted labeling pipeline. Instead of manual annotation (which would be prohibitively expensive at this scale), the paper uses a structured labeling pipeline driven by an external large multimodal model (LMM):

Input preparation: For each 3D object, the pipeline renders one full-asset image (front view) and $K$ part close-up crops (one per part, showing the part in isolation or highlighted context).
Structured tool schema: The LMM is called with a JSON response format and deterministic field ordering. For each part, it outputs:
- Q1: a short, coarse part name (e.g., "leg," "seat," "handle").
- Q2: a fine-grained natural description, constrained to $\leq 15$ words, avoiding irrelevant rendering terms (e.g., "a turned wooden leg with tapered bottom and decorative groove").
- Q3: a confidence flag (Yes/No), presumably to filter low-quality annotations.
JSON validation: The LMM's output is validated against the schema and stored. Objects with low-confidence or malformed annotations are filtered out.

Why model-assisted labeling: The paper needs part-level natural language descriptions at two granularities for ~86K objects × ~23 parts = ~2 million part annotations. Manual annotation at this scale is infeasible. Using an LMM with a structured schema ensures consistent, machine-parseable outputs while leveraging the LMM's visual understanding capabilities. The Q1/Q2 split is deliberate: Q1 provides coarse labels for tasks like part naming and coarse grounding (Types 1, 3, 5), while Q2 provides rich descriptions for tasks like fine grounding and detailed captioning (Types 2, 4, 6). The 15-word limit on Q2 prevents the LMM from producing excessively verbose or irrelevant descriptions.

Data building: converting raw annotations to instruction-following samples. The raw annotations (bounding boxes + Q1/Q2 text per part) are converted into training samples through a template instantiation process (Algorithm 1 in Appendix A.6.2):

Serialize parts: For each object, every part's AABB is serialized to the six-token quantized coordinate format. Parts are sorted deterministically by $(z_{\min}, y_{\min}, x_{\min})$ .
Template instantiation: For each of the 11 task types, a natural-language prompt is sampled from a template pool (e.g., for Type 1: "What distinct components does this contain? Please annotate with bounding boxes and provide short labels" is one of 37 variants). The corresponding target output is constructed from the serialized boxes and text annotations.
Train/test split: The file list is partitioned deterministically at 0.5% for test and 99.5% for train. The UniPart-Bench evaluation set (400 objects) is drawn from the test partition.

The 11 task templates (Types 0–10):

Type 0: Pure box listing. Input: prompt asking to detect all bounding boxes. Output: sequence of <boxs> tokens <boxe> blocks, no text. This teaches the model to decompose shapes into parts without any semantic labeling — pure geometric reasoning.
Type 1: Multi-part grounding (coarse text). Input: prompt asking "what components does this contain?" Output: per-part <boxs> tokens <boxe> Q1_name (box + short label). This teaches the model to associate geometric parts with coarse semantic names.
Type 2: Multi-part grounding (fine text). Input: prompt asking for an overall description followed by per-part details. Output: overall object description (free text), then per-part <boxs> tokens <boxe> Q2_description (box + detailed description). This teaches the model to generate rich, descriptive part captions alongside spatial grounding.
Type 3: Single-part grounding (from coarse text). Input: "Find the {part_name} in this model." Output: all bounding boxes of parts matching that name, with descriptions. This teaches the model to localize parts by name — a retrieval task.
Type 4: Single-part grounding (from fine text). Input: "Where is the part corresponding to this description: {part_description}?" Output: a single bounding box. This teaches the model to localize parts by detailed appearance description — a harder retrieval task requiring fine-grained matching.
Type 5: Box-to-text (coarse). Input: a bounding box is provided in the prompt; ask "What is this part?" Output: the Q1 name. This teaches the reverse mapping: given a spatial location, produce the semantic label. It complements Types 1 and 3 by providing bidirectional grounding supervision.
Type 6: Box-to-text (fine). Input: a bounding box is provided; ask "Describe this part in detail." Output: the Q2 description. This teaches detailed description generation conditioned on spatial location.
Type 7: Part-aware Q&A. Input: a question about parts from the raw annotations' QA field. Output: the answer, with textual part references replaced by <boxs> tokens <boxe> tokens. For example, "Which part connects the seat to the backrest?" → "The <boxs> side_bracket_tokens <boxe> connects the seat to the backrest." This teaches the model to ground its language answers with spatial references — the key capability missing from existing 3D MLLMs.
Type 8: Deletion program. Input: "Please remove the {part_name}" or "Delete the part that is {part_description}." Output: <dels> <boxs> tokens <boxe> <dele>. This teaches the model to produce executable delete commands.
Type 9: Modification program. Input: "Change the {part_name} to be {new_description}." Output: <mods> <boxs> tokens <boxe> new_description <mode>. This teaches the model to produce executable modify commands, where the output specifies both the target part (by box) and the desired new appearance (by text).
Type 10: Addition program. Input: "Add the {part_name} to this 3D asset." Output: <adds> <boxs> tokens <boxe> part_description <adde>. This teaches the model to produce executable add commands, where it must generate a target bounding box (where to add) and a part description (what to add).

Balancing and duplication. Template counts are adjusted after initial construction: Types 1–2 are "lightly duplicated to increase multi-part coverage" (the most complex output format, requiring bounding boxes + text for multiple parts); Types 3–7 are downsampled to a fixed budget (to prevent simpler task formats from dominating training); edit templates (Types 8–10) are capped per shard. The Full counts in Table 5 reflect the final balanced dataset.

Why 11 task types: This diversity is not arbitrary. The set covers the Cartesian product of three dimensions: (1) direction (box→text vs. text→box vs. both), (2) granularity (coarse Q1 vs. fine Q2), and (3) output type (description vs. program). Types 0–2 are description tasks (output contains boxes and text). Types 3–4 are localization tasks (input is text, output is boxes). Types 5–6 are reverse-localization tasks (input is boxes, output is text). Type 7 is grounded Q&A (input and output both contain text and boxes). Types 8–10 are edit programs (output is an executable command with boxes and operators). This diversity ensures the model learns all aspects of the part grammar: generating boxes from text, generating text from boxes, using boxes as references in answers, and producing executable commands.

Downstream Geometry Interfaces and End-to-End Task Realization

The final component of the system is the interface between the model's token programs and the geometry engines that produce actual 3D assets. The paper's key claim is that these interfaces are model-agnostic — the planning model (Part-X-MLLM) can drive any geometry engine that understands bounding boxes and part descriptions.

Part-aware synthesis interface. For generation tasks, the model's output program is parsed to extract:

A set of part bounding boxes (the six-coordinate tokens inside <boxs>...<boxe> blocks).
Optional part text descriptions (the text tokens following each box block in Type 2 outputs).

These are passed to OmniPart's synthesis module (Yang et al., 2025), which treats the bounding boxes as spatial guides. OmniPart uses an autoregressive box-planning approach: it takes the bounding boxes, places them in 3D space, and then uses the TRELLIS sparse-voxel generation pipeline to synthesize high-fidelity geometry within each box, respecting the box boundaries as constraints. Because Part-X-MLLM provides both the box extents and the part descriptions, OmniPart can generate semantically appropriate geometry for each part (e.g., a "cylindrical wooden leg with a tapered bottom" receives geometry consistent with that description).

Why OmniPart: The paper chooses OmniPart because it already has a box-conditioned synthesis interface — it was designed to take bounding boxes and part descriptions as input and produce per-part geometry. Part-X-MLLM simply replaces OmniPart's internal box planning module (which was tightly coupled to the TRELLIS backend) with a language-native planner (which can be swapped to any compatible synthesis engine). This is the "separation of planning and execution" in practice.

Localized editing interface. For editing tasks, the model's output program is parsed to extract:

The edit operator (<dels>, <mods>, or <adds>).
The target bounding box(es) (for delete and modify) or the target insertion box (for add).
Optional new text description (for modify and add).

The bounding box(es) are converted to cuboid masks — binary volumetric masks where voxels inside the box are 1 (to be edited) and voxels outside are 0 (to be preserved). These masks, along with the user's natural language instruction, are passed to the editing backend:

For deletion: the cuboid mask defines the region to erase.
For modification: the cuboid mask defines the local region where the edit applies; the geometry engine modifies only voxels inside the mask while preserving the rest.
For addition: the cuboid mask defines where the new part should be inserted.

The paper uses two training-free volumetric editors: Nano3D (Ye et al., 2025) and VoxHammer (Li et al., 2025). Both are "training-free" in the sense that they don't require fine-tuning for specific editing operations — they take a 3D volume, a mask, and a text instruction, and produce an edited volume. By coupling Part-X-MLLM's language-driven planning with these engines, the system achieves language-native 3D editing without requiring the editing tool itself to understand language about parts.

Why training-free editors: Training-free editors can be applied to any asset without per-object or per-edit-type fine-tuning, which aligns with the paper's goal of a universal, model-agnostic interface. However, the paper is not making a claim about which editor is best — any editor that accepts cuboid masks and text could serve as the backend. The contribution is the automatic localization (converting "remove the handle" into the correct bounding box and cuboid mask) rather than the editing algorithm itself.

End-to-end examples from Figure 3:

Part-aware mesh generation (top row, task type 1): The model receives an RGB point cloud and a prompt asking for decomposition. It outputs a sequence of <boxs> tokens <boxe> text blocks. OmniPart's synthesis module generates per-box geometry, producing a multi-part mesh where each part is a separate geometric entity (not just a visual segment on a monolithic mesh).
Grounded Q&A (middle row, task type 7): The model receives a question about parts ("Where is the seat cushion?") and outputs an answer embedding BBox tokens: "The seat cushion is located at <boxs> tokens <boxe>." The BBox tokens provide machine-readable spatial grounding — a downstream visualization system could highlight the referenced box, or a robotic system could compute grasp coordinates.
Auto-located 3D editing (bottom row, task type 8): The model receives "remove the handle" and the point cloud. It outputs <dels> <boxs> handle_box_tokens <boxe> <dele>. The downstream editor extracts the bounding box, creates a cuboid mask, and applies the deletion — removing the handle geometry while preserving the rest of the object.

Semantic Granularity Control via Part Clustering

This is a post-hoc capability enabled by the structured output representation, not a learned model behavior. Because each part is represented by both a bounding box (spatial extent) and a text description (semantic identity), parts can be merged based on semantic similarity without requiring model retraining or architectural changes.

The clustering algorithm (detailed in Appendix A.2.1):

Step 1: Feature extraction. For each predicted part $p_i$ , two feature vectors are computed:

Semantic feature: The text description $d_i$ is encoded using a pretrained CLIP model:

$\mathbf{f}_{\text{sem},i} = \text{CLIP-Encode}(d_i)$

This produces a high-dimensional embedding capturing the semantic meaning of the description (e.g., "leg" vs. "armrest" vs. "backrest").

Spatial feature: The bounding box's center $\mathbf{c}_i = (\mathbf{x}_{\min} + \mathbf{x}_{\max}) / 2$ (the midpoint) and size $\mathbf{s}_i = \mathbf{x}_{\max} - \mathbf{x}_{\min}$ (the extent along each axis) are concatenated into a 6-dimensional vector:

$\mathbf{f}_{\text{spat},i} = [\mathbf{c}_i, \mathbf{s}_i]$

This vector is then normalized across all $N$ parts in the object (presumably z-score normalization per dimension) to produce $\hat{\mathbf{f}}_{\text{spat},i}$ .

Step 2: Hybrid feature fusion. The semantic and spatial features are combined with a weighting factor $\alpha \in [0, 1]$ , and the resulting vector is L2-normalized:

$\mathbf{f}_i = \frac{(1 - \alpha) \mathbf{f}_{\text{sem},i} \oplus \alpha \hat{\mathbf{f}}_{\text{spat},i}}{\|(1 - \alpha) \mathbf{f}_{\text{sem},i} \oplus \alpha \hat{\mathbf{f}}_{\text{spat},i}\|_2}$

where $\oplus$ denotes vector concatenation, $\alpha$ controls the relative importance of spatial proximity vs. semantic similarity, and the denominator is the Euclidean norm ensuring the final feature vector has unit length.

What it computes: A single feature vector per part that balances semantic similarity (do these parts have similar descriptions?) and spatial proximity (are these parts close together in 3D?). When $\alpha = 0$ , only semantic similarity matters (two "leg" parts merge regardless of position); when $\alpha = 1$ , only spatial proximity matters (adjacent parts merge regardless of what they're called). The L2 normalization ensures that the clustering algorithm (DBSCAN) operates on cosine-like similarities in a normalized space.

Why concatenation rather than a weighted sum: Concatenation preserves the separate semantic and spatial information in the feature vector; the clustering algorithm can learn which dimensions to attend to based on the data distribution. A weighted sum would force a trade-off where semantic and spatial information compete for the same feature dimensions, potentially losing important distinctions.

Step 3: DBSCAN clustering. The set of feature vectors $\{\mathbf{f}_i\}_{i=1}^N$ is clustered using DBSCAN (Density-Based Spatial Clustering of Applications with Noise), parameterized by:

$\epsilon$ : the maximum distance between two points for them to be considered neighbors.
minPts: the minimum number of points required to form a dense region (a cluster).

A point is a core point if its $\epsilon$ -neighborhood contains at least minPts other points. A cluster is formed by starting from a core point and recursively adding all points that are density-connected (reachable through a chain of points where each step is within $\epsilon$ and each intermediate point is a core point). Points not reachable from any core point are labeled as noise.

DBSCAN outputs $K$ clusters $\mathcal{C} = \{C_1, \ldots, C_K\}$ (where $K$ is automatically determined, not specified in advance) and a set of noise points $\mathcal{N}$ .

Why DBSCAN: The key property that makes DBSCAN appropriate is that it does not require specifying the number of clusters in advance — the algorithm discovers the natural grouping based on the density of points in the feature space. This is essential for the granularity control use case because different objects have different natural decomposition levels (a simple chair might have 3–5 semantic groups; a complex mechanical assembly might have 20+). DBSCAN also handles outliers (noise points) gracefully — parts that don't fit any cluster remain as individual parts rather than being forcibly merged.

Step 4: Merging. For each cluster $C_k$ , a single merged bounding box $B_k$ is computed by taking the component-wise minimum and maximum over all bounding boxes in the cluster:

$\mathbf{X}_{\min,k} = \min_{i \;|\; b_i \in C_k} (\mathbf{x}_{\min,i}), \quad \mathbf{X}_{\max,k} = \max_{i \;|\; b_i \in C_k} (\mathbf{x}_{\max,i})$

What it computes: The tightest axis-aligned bounding box that encloses all parts in the cluster — simply the minimum lower corner and maximum upper corner across all member parts. This is geometrically conservative: it guarantees that the merged box contains all original part boxes (and therefore all original part geometry).

Why min/max rather than union or averaging: Taking the component-wise min and max of bounding boxes produces a tight enclosing box — the smallest axis-aligned box that contains all member boxes. A union of boxes would produce a non-convex volume (potentially with holes), which is harder for downstream synthesis modules to work with. Averaging box coordinates would lose the spatial extent information and produce a box that doesn't enclose its constituents.

The granularity control in practice (Figure 6): The algorithm is applied progressively, starting from the finest granularity (one box per predicted part) and merging based on semantic similarity. By adjusting $\epsilon$ and $\alpha$ , the user can control how aggressively parts are merged:

Low $\epsilon$ (strict): only nearly identical descriptions in nearly the same location merge → many small, fine-grained clusters.
High $\epsilon$ (permissive): broadly similar descriptions in similar locations merge → few large, coarse clusters.

Figure 6 shows the process reducing a 22-part decomposition to 2 coarse components (presumably by gradually increasing $\epsilon$ or adjusting $\alpha$ ).

Why this capability matters: Existing part decomposition systems either (1) require the user to pre-specify the number of parts (like PartPacker), (2) produce a fixed granularity determined by the training data, or (3) require manual mask merging (like OmniPart). The box-and-text representation enables automatic, data-driven granularity control without changing the model or requiring manual intervention. The same Part-X-MLLM output can serve both a user who wants fine-grained part manipulation ("adjust the left armrest's front screw") and a user who wants coarse semantic grouping ("modify all armrests together").

4. Key Insights and Innovations

Innovation 1: Executable Program Generation as a Unifying Abstraction for 3D Interaction

The paper's most fundamental conceptual move is reframing 3D understanding, generation, and editing not as separate task-specific architectures, but as a single program synthesis problem. The model emits a token sequence in a structured grammar — bounding boxes, part names, descriptions, edit operators — and the downstream interpretation of that program determines whether the output is an answer, a generated mesh, or an edited asset.

What makes this intellectually distinctive is not the use of a grammar per se (programmatic representations of 3D scenes exist in prior work, e.g., Scene-LLM and the Scene Language), but the claim that a single grammar can serve as both a descriptive and an imperative interface. In prior work, scene programs describe what exists; in Part-X-MLLM, the same grammar describes what exists (for Q&A and captioning), what to create (for generation), and what to change (for editing). The grammar tokens carry dual semantics: a bounding box token sequence <boxs> t1...t6 <boxe> is simultaneously a spatial reference (this is where the part is), a persistent identifier (the same token sequence can be reused across tasks), and an executable command (a downstream module will act on this spatial region). This is a category shift from programs-as-descriptions to programs-as-commands, enabled by the decoupling of planning from execution.

The significance of this reframing extends beyond the specific architecture. It establishes a design pattern for multimodal LLMs interacting with external tools: rather than training the LLM to produce geometry directly (which couples the model to a specific synthesis pipeline and representation), train it to produce a tool-agnostic intermediate representation that any compatible execution backend can consume. The paper's demonstration that the same planning model works with OmniPart for generation, Nano3D for editing, and VoxHammer for alternative editing validates this pattern. A better geometry engine in the future can be swapped in without retraining the LLM — the grammar serves as a stable interface contract between the language model and the geometry modules.

The evidence for this reframing is not captured in a single table but in the architecture's scope: the 11 task families in UniPart-Bench, spanning pure geometry tasks (Type 0 box listing), grounded language tasks (Types 3–7), and imperative programs (Types 8–10), are all handled by the same model weights through the same output grammar. The per-task results in Table 7 confirm that the model produces valid programs across all types, with language-intensive tasks (T7, T10) achieving the strongest text metrics (SBERT/SimCSE) and geometry-intensive tasks (T0, T2) achieving the strongest IoU metrics. No prior 3D MLLM or part pipeline demonstrates this breadth under a single interface.

Innovation 2: The Representational Conflict Hypothesis and Dual-Encoder Design as a Diagnostic Intervention

The dual-encoder architecture (separate structure and semantic pathways) is not just an engineering choice that improves metrics — it embodies a diagnostic hypothesis about why single-encoder 3D-language models underperform on part-level tasks. The paper's claim, tested in the ablation study (Table 2), is that forcing one encoder to represent both geometry (shape, topology, surface orientation) and appearance (color, texture, material) creates a representational conflict: the two modalities have fundamentally different invariance properties, and a single feature space must compromise between encoding shape distinctions (which should be color-invariant) and color distinctions (which should be shape-invariant).

Prior 3D MLLMs (PointLLM, 3D-LLM, GPT4Point, ShapeLLM) all use single-encoder designs that fuse XYZ and RGB into a unified point cloud representation before encoding. The dominant assumption has been that a sufficiently powerful encoder can learn to disentangle these modalities internally without architectural separation. The dual-encoder ablation challenges this assumption directly: if a single encoder could disentangle structure and appearance, adding a second encoder should provide minimal benefit. Instead, the dual-encoder design yields +7.06 BBox IoU on pure box listing (a geometry-heavy task where color should be irrelevant) and uniform gains across all metrics for part Q&A and multi-part grounding (language-heavy tasks where both modalities matter). This pattern — improvement on both geometry-only and language-heavy tasks — is the key diagnostic signal: a single encoder's fused representation degrades performance in both directions, not just one.

The significance of this finding is that it identifies the encoder design as the bottleneck, not the decoder capacity or the training data scale. The decoder (Qwen 2.5 VL) is the same in both conditions; only the encoding pathway changes. This suggests that for 3D-language tasks, modality-specific encoding followed by cross-modal fusion in the decoder may be a general design principle, not just a specific trick for part-level tasks. The Stage 1 geometry-only pretraining further supports this: by specializing the structure encoder on pure geometry (3.6M objects, no RGB, no language), the model forces shape decomposition knowledge into a pathway that cannot rely on color shortcuts (e.g., using green pixels to segment a green chair leg), producing representations that genuinely reflect 3D structure rather than correlated 2D appearance.

This is a fundamental contribution to how we think about 3D multimodal architectures, not just an incremental improvement. It reframes the problem from "more data, bigger model" to "architectural separation of modality-specific invariances," and the ablation provides causal evidence (modify the architecture, measure the effect) rather than just correlational scaling observations.

Innovation 3: Difficulty-Aware Semantic Granularity via Post-Hoc Part Clustering

The semantic granularity control mechanism (Appendix A.2.1, Figure 6) is not a learned behavior of the model — it is a post-processing algorithm applied to the model's output. What makes it innovative is the insight that the box-and-text representation carries enough information for automatic granularity control without model retraining. This resolves a long-standing tension in part-based 3D systems: different applications require different part decomposition granularities, but training separate models for each granularity level is expensive and the "correct" number of parts is often subjective.

Prior approaches to this problem impose constraints that Part-X-MLLM avoids. PartPacker requires the user to pre-specify the number of parts. OmniPart produces a fixed decomposition and requires manual mask merging to change granularity. Hierarchical part models embed a fixed tree structure in their architecture. The innovation here is that by separating spatial extent (bounding boxes) from semantic identity (text descriptions), the same model output can be re-clustered at query time without retraining. The CLIP-based feature extraction computes semantic similarity from the part descriptions; the DBSCAN clustering determines natural groupings without a preset cluster count; the bounding box merging (component-wise min/max) preserves geometric validity. Different users, or different applications, can apply different clustering parameters to the same model output.

The significance is architectural rather than metric-driven. It validates the design choice to keep spatial and semantic information separate in the model's output grammar rather than fusing them into a single learned representation. If the model output a monolithic "part embedding" that fused spatial and semantic features, granularity control would require retraining or architectural changes. By outputting explicit box tokens and text tokens, the system enables query-time adaptability — the model does one forward pass, and the decomposition granularity is decided afterward. This is a form of delayed binding that precomputes the spatial-semantic description but postpones the semantic aggregation.

The CLIP + DBSCAN pipeline (Equations 1–4 in Appendix A.2.1) is a specific instantiation of this idea, but the deeper contribution is the representational principle: when a model's output separates the continuous spatial signal from the discrete semantic label, downstream systems gain flexibility that fused representations preclude. This is transferable beyond 3D — any domain where entities have both spatial extent and semantic identity (video object tracking, robotic scene understanding, geographic information systems) could benefit from this separation.

Innovation 4: Automatic Edit Localization as a Language-Native Frontend for Geometry Tools

Existing 3D editing tools (Instruct-NeRF2NeRF, Shap-Editor, Nano3D, VoxHammer) are capable geometry engines, but they require the user to provide spatial specifications — masks, bounding boxes, view selections, or 3D regions — alongside the language instruction. The intellectual contribution here is not the editing itself (which is delegated to existing tools) but the recognition that the localization problem — converting "remove the handle" into a specific spatial region — is a language modeling problem that can be solved by an LLM with 3D encoders, independently of how the edit is subsequently executed.

This separates what was previously a monolithic editing pipeline into two stages: (1) a language-driven planning stage that reasons about parts, identifies which spatial region corresponds to the linguistic reference, and produces a machine-readable spatial specification; and (2) a geometry-execution stage that applies the edit within the specified region using any compatible editing algorithm. Prior work bundled these stages — the editing tool had to understand language AND manipulate geometry, which meant that improvements in language understanding required changes to the geometry pipeline and vice versa. The decoupling means that the language frontend can improve independently of the geometry backend, and the geometry backend can be swapped without affecting language understanding.

The evidence for this contribution is in the qualitative editing results (Figure 5) and the edit program task types in UniPart-Bench (Types 8–10, Table 7). The model successfully localizes edit targets from language alone ("remove the handle," "change the seat to be a cushioned leather seat") and produces structured delete/modify/add programs with the correct bounding boxes. The automatic localization is the novel capability — prior editing tools would require the user to manually draw a mask around the handle or select the handle's vertices. The contribution is making editing tools language-addressable rather than manually-addressable.

This reframes the editing tool landscape: rather than competing on which editor produces better geometry (a geometry-quality arms race), future work can focus on which language frontend provides better localization and part reasoning, with geometry quality determined by whichever backend is plugged in. It also suggests a path toward composable 3D tools: a user could issue a sequence of language commands ("remove the legs, then add modern tapered legs, then change the backrest material to walnut"), and the language frontend would produce a sequence of spatial programs that the geometry backends execute in order, without the user touching any geometry directly.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the authors' in-house part-centric 3D dataset, comprising 85,771 unique 3D objects with an average of 23 parts per object (Section 4.2, Appendix A.6). Each object is annotated with axis-aligned part bounding boxes (AABBs) and paired natural language annotations at two granularities: a coarse part label (Q1) and a fine-grained part description (Q2). Annotations are generated through a model-assisted labeling pipeline (Appendix A.6.1) using an external large multimodal model (LMM) with a structured JSON schema. The dataset is partitioned deterministically at ~99.5/0.5% for train/test. The evaluation benchmark, UniPart-Bench, is a held-out subset of 400 objects drawn from the test partition, spanning 11 task families (Types 0–10) covering pure box listing, multi-part grounding at coarse and fine granularity, single-part grounding, box-to-text captioning, part-aware Q&A, and edit programs (deletion, modification, addition).
Base model(s). The core model is built on Qwen 2.5 VL, initialized from its pretrained weights (Section 3.5, Stage 2). The structure encoder is initialized from the Hunyuan 2.1 3D Shape VAE Encoder and domain-specialized through Stage 1 pretraining on 3.6M geometry-only objects. The semantic encoder shares the same architecture as the structure encoder but is trained from scratch during Stage 2. All experiments use a single model scale — no model scaling experiments (e.g., varying the LLM size or encoder capacity) are conducted. The baseline comparisons in Tables 1, 3, and 4 are against prior published models (PointLLM, 3D-LLM, GPT4Point, ShapeLLM, LL3DA, 3D-LLAVA, PartField, OmniPart) run on the same UniPart-Bench evaluation set. The paper does not report the model sizes of these baselines, making FLOPs-matched comparisons impossible.
Metrics. The evaluation uses two families of metrics corresponding to the two output modalities in the structured grammar. Geometric metrics (for bounding box quality): BBox IoU — the Intersection over Union between predicted and ground-truth bounding boxes, computed by matching each ground-truth box with its nearest predicted box; Voxel Recall — the fraction of ground-truth voxels within the predicted bounding box; and Voxel IoU — the IoU computed over voxel occupancy within the predicted and ground-truth boxes (Table 1). Linguistic metrics (for text quality): SBERT and SimCSE — sentence-level embedding similarities measuring semantic alignment between predicted and ground-truth text; BLEU-1, ROUGE-L, and METEOR — token-level n-gram overlap metrics (Tables 3 and 4). For tasks where ground-truth contains both boxes and text (Types 1–7), both metric families are reported. For pure geometry tasks (Type 0), only box metrics apply. For pure text tasks, only linguistic metrics apply. Per-task breakdowns are provided in Table 7.
Baselines. The paper compares against two categories of baselines. For bounding box generation (Table 1): PartField (Liu et al., 2025) — a 2D-lifting approach where voxels derived from the asset are treated as a point cloud, segmented at the ground-truth part count, and bounding boxes are computed per segment; and the generation model from OmniPart (Yang et al., 2025) — a native 3D part generation system using SAM-based segmentation with small mask filtering (area ratio less than 1600/1024²). For 3D understanding tasks (Tables 3 and 4): a suite of prior 3D MLLMs including PointLLM (Xu et al., 2024), 3D-LLM (Hong et al., 2023), GPT4Point (Qi et al., 2024), ShapeLLM (Qi et al., 2024), LL3DA (Chen et al., 2024), and 3D-LLAVA (Deng et al., 2025). The paper reports that these baselines are evaluated on UniPart-Bench using their respective public implementations.
Generation budget / compute accounting. The paper does not define a standardized unit of inference compute (e.g., FLOPs, number of generated tokens, or number of forward passes). Training compute is reported descriptively rather than analytically: Stage 1 runs for 2 days on 64 A100 GPUs; Stage 2 runs for 5 days on 64 A100 GPUs (Section 4.1). No FLOP-matched comparison is conducted between Part-X-MLLM and any baseline — the paper does not claim or test whether the performance improvements are cost-effective relative to simply scaling the base model or using more test-time compute. Inference cost is not discussed at all, despite the model using two separate encoders (each processing different point cloud resolutions: 40,960 and 10,240 points) and an autoregressive decoder — the latency and memory implications of this design vs. a single-encoder alternative are not analyzed.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report statistical significance for any metric. The train/test split is a single deterministic partition of the file list at 0.5%, with no indication that results are averaged over multiple random splits. Error bars or confidence intervals are absent from all tables and figures. For the ablation study (Table 2), the paper reports raw metric values without indicating whether the observed differences are statistically reliable given the test set size (400 objects for UniPart-Bench, with per-task sample counts ranging from dozens to hundreds depending on the template type). The reported BBox IoU improvements (+7.06 on Type 0 box listing) and linguistic metric improvements (+18.7 SBERT on part Q&A) are presented as point estimates only.

Main Quantitative Results

Bounding Box Generation (Table 1)

The primary geometric evaluation compares Part-X-MLLM against PartField and OmniPart on bounding box generation quality across three metrics: BBox IoU, Voxel Recall, and Voxel IoU. The paper reports that Part-X-MLLM outperforms both baselines on all three metrics (Section 4.3, Table 1). However, the paper does not provide the actual numerical values in the main text — Table 1 is referenced but its content is not reproduced or summarized with specific numbers. The qualitative results in Figure 4 are claimed to show "superior performance in generating semantically coherent and geometrically accurate part segmentations" with Part-X-MLLM "successfully capturing fine-grained details and maintaining structural integrity, outperforming baselines that often produce fragmented or inaccurate decompositions."

For the PartField baseline, the evaluation protocol treats voxels derived from the asset as a point cloud, segments them at the ground-truth part count, then computes bounding boxes per segment. This means PartField is given the oracle part count — it knows exactly how many parts to segment into, while Part-X-MLLM must predict the number of parts as part of the autoregressive generation. For OmniPart, masks are obtained from SAM and small masks are filtered out with an area ratio threshold of 1600/1024². The paper does not discuss whether this filtering threshold was tuned on the test set or whether OmniPart's performance is sensitive to this parameter.

A critical detail: the paper does not specify whether the bounding box generation evaluation is conducted on all 11 task types or only on a subset. Types 0, 1, 2, 3, 4, 8, 9, and 10 all produce bounding box outputs, but each type has different input prompts and expected output formats. The aggregated results in Table 1 may mask significant per-task variation.

Qualitative Generation and Editing (Figures 4 and 5)

Figure 4 presents qualitative shape decomposition results, comparing Part-X-MLLM's part segmentations against unspecified baselines. The paper claims the model captures "fine-grained details and maintains structural integrity," but no quantitative metrics accompany these visualizations — the reader cannot assess whether the visible improvements are representative or cherry-picked.

Figure 5 presents qualitative results for part-aware editing, showing the model interpreting natural language instructions to add, remove, or modify specific parts. The editing is executed by off-the-shelf geometry engines (Nano3D for editing, VoxHammer as an alternative), with Part-X-MLLM providing the bounding box and edit command. The paper claims the model "successfully interprets user instructions to add, remove, or modify specific parts, executing the edits while preserving the rest of the object's structure." However, no quantitative editing metrics are reported — no IoU between the edited region and a ground-truth edit mask, no user study measuring edit quality or instruction adherence, and no comparison against editing baselines that do not use a language frontend (e.g., manually specifying masks vs. using Part-X-MLLM's auto-localization). The editing results therefore remain anecdotal — the paper demonstrates feasibility but does not quantify how well the localization works or how often edits fail.

Semantic Granularity Control (Figure 6)

Figure 6 demonstrates the post-hoc part clustering algorithm (detailed in Appendix A.2.1) progressively merging fine-grained parts into coarser components. The paper shows a reduction from 22 parts to 2 parts by adjusting clustering parameters. This is a qualitative demonstration of the algorithm's behavior — no quantitative evaluation of clustering quality is provided (e.g., comparing merged clusters against human-annotated coarse part groupings, measuring cluster purity, or reporting the sensitivity of the output to the choice of ε and minPts in DBSCAN). The paper does not report whether the clustering consistently produces semantically meaningful groupings across diverse object categories, or whether there are systematic failure modes (e.g., over-merging semantically distinct parts that happen to have similar descriptions, or under-merging near-identical parts).

Dual-Encoder Ablation Study (Table 2)

This is the only rigorously quantified ablation in the paper. The experiment compares the full dual-encoder model against a single-encoder variant that consumes a unified point cloud with fused geometry (XYZ) and color (RGB). The evaluation is on UniPart-Bench across multiple task types.

On pure box listing (Type 0): The dual-encoder design improves BBox IoU by +7.06 absolute percentage points over the single-encoder baseline. This is the largest single-metric gain reported and provides the strongest evidence for the representational conflict hypothesis: when geometry must share a feature space with color, spatial prediction suffers substantially.

On Part QA and Multi-Part Grounding: The paper reports "uniform gains across all metrics" but does not provide the specific per-metric differences in the main text. The claim is that both language-heavy tasks (Part QA, which requires semantic understanding) and geometry-heavy tasks (Multi-Part Grounding, which requires spatial localization with text) benefit from the dual-encoder design. Critically, this pattern of improvement supports the paper's architectural hypothesis: if the dual-encoder only improved geometry tasks, one could argue that the single encoder was simply underparameterized; the fact that language tasks also improve suggests that the fused representation in the single encoder is genuinely entangled in a way that harms both modalities.

Table 2 is cited as the source but its content is not reproduced in the main text, making it impossible to verify the magnitude of the language-task improvements or whether they are uniform (as claimed) or concentrated in specific metrics.

Part Understanding Q&A (Table 3)

The paper reports part-level understanding and reasoning results on UniPart-Bench, comparing against PointLLM, 3D-LLM, GPT4Point, ShapeLLM, LL3DA, and 3D-LLAVA. The metrics include sentence-level similarities (SBERT, SimCSE) and token-level metrics (BLEU-1, ROUGE-L, METEOR). The paper claims substantial gains over the strongest baseline across all metrics:

"compared to the best non-ours scores, Part-X-MLLM improves by +18.7 SBERT, +25.5 SimCSE, +21.3 BLEU-1, +13.0 ROUGE-L, and +14.2 METEOR"

These improvements are absolute percentage points in the respective metric scales. To interpret these: SBERT and SimCSE are cosine similarities in embedding space (typically in [0, 1] or [-1, 1]), so +18.7 SBERT is a very large jump if the metric maximum is 100 (suggesting the baselines score relatively low on part-level Q&A). BLEU-1 is a precision-based unigram overlap metric (0–100 scale), where +21.3 is also substantial. ROUGE-L is recall-oriented and METEOR incorporates synonym matching — improvements of +13.0 and +14.2 respectively indicate that baselines struggle significantly with part-level language generation compared to Part-X-MLLM.

However, several caveats apply. First, the baseline models (PointLLM, 3D-LLM, etc.) were not designed or trained for part-level Q&A — they are general 3D MLLMs trained for holistic object/scene understanding. Their poor performance on part-level tasks may reflect a training data mismatch rather than an architectural limitation. The paper does not control for this by fine-tuning the baselines on the same part-centric dataset, making the comparison partially a test of "model trained for parts vs. model not trained for parts" rather than "better architecture vs. worse architecture." Second, UniPart-Bench is a held-out subset of the same dataset Part-X-MLLM was trained on, with identical data construction pipeline and distribution characteristics. Baseline models, trained on different datasets, face a distribution shift that Part-X-MLLM does not. Third, the paper does not report whether the answer format expected by UniPart-Bench (with embedded bounding box tokens) is even supported by the baseline model architectures — if baselines cannot produce BBox tokens in their outputs, they are evaluated on a task they fundamentally cannot perform, making the comparison uninformative about relative architectural merit.

Overall 3D Object Captioning (Table 4)

The paper evaluates holistic object understanding on UniPart-Bench, comparing against the same baselines as Table 3. The metrics are identical: SBERT, SimCSE, BLEU-1, ROUGE-L, METEOR. The paper reports:

"absolute improvements of +10.4 SBERT, +9.4 SimCSE, +18.8 BLEU-1, +20.2 ROUGE-L, and +15.2 METEOR"

These gains are also substantial, particularly the token-level metrics (BLEU-1 +18.8, ROUGE-L +20.2). The paper notes that "the large gains on token-based metrics suggest stronger lexical coverage and structure in object-level descriptions." This is likely because Part-X-MLLM's training data includes object-level captions as part of Type 2 task templates (the overall description preceding per-part details), meaning the model has been explicitly trained to produce the format expected by UniPart-Bench's object captioning evaluation.

The same caveats apply as for part Q&A: distribution shift for baselines, training data mismatch, and the possibility that Part-X-MLLM's advantage reflects training on the evaluation distribution rather than superior architectural capability.

Per-Task Breakdown (Table 7)

Table 7 (Appendix A.8) provides a comprehensive per-task breakdown on UniPart-Bench for all 11 task templates (Types 0–10). The paper summarizes the results qualitatively rather than reporting specific numbers in the main text:

"Language-intensive tasks (T7 Part QA, T10 Edit—Add) obtain the highest SBERT/SimCSE and strong lexical metrics, indicating robust alignment between our planned box-conditioned answers/programs and textual GT."

"Among IoU-based tasks, T0/T2/T10 show the strongest geometric alignment, reflecting reliable planning for pure detection, fine grounding, and edit addition respectively."

"Blank text or IoU entries arise by design when a task's GT lacks the corresponding modality."

This per-task breakdown is important because aggregated metrics (Tables 1, 3, 4) can mask task-specific strengths and weaknesses. For instance, the claim that T10 (Edit—Add) achieves strong geometric alignment is notable because addition requires the model to generate a bounding box for where to insert a new part — a fundamentally different spatial reasoning problem from detecting existing parts. The paper does not report whether T8 (Deletion) and T9 (Modification) perform comparably to T10, or whether edit program tasks are systematically harder or easier than grounding tasks.

Qualitative Understanding Results (Figures 8 and 9)

Figure 8 provides qualitative examples for overall object captioning, comparing Part-X-MLLM against baseline models. The paper highlights a specific example: "our model correctly identifies an object as a 'pink teddy bear mascot costume with a purple bow tie,' while other models provide less specific or incorrect descriptions." Figure 9 (Appendix A.3) provides qualitative examples for part-aware Q&A, showing the model's grounding capabilities with box-annotated answers. As with the generation and editing qualitative results, no quantitative metrics or human evaluation accompanies these visualizations.

t-SNE Visualization of Special Token Embeddings (Figure 10)

Figure 10 (Appendix A.5) shows a t-SNE visualization of the learned embeddings for the 128 box tokens (representing quantized coordinates) and the grammar special tokens (<boxs>, <boxe>, edit operators). The paper reports three observations: (1) Point, Box, and Edit tokens form distinct functional clusters; (2) the 128 box tokens form a continuous, ordered manifold, indicating the model has learned the ordinal nature of spatial coordinates rather than treating them as independent categorical variables; (3) paired tokens like <adds>/<adde> are positioned closely together. This is offered as evidence that "the model has successfully learned a robust and interpretable representation of our executable grammar." This is a diagnostic visualization rather than a performance metric — it demonstrates that the learned token embeddings are structured, but does not quantify how this structure contributes to downstream task performance.

Confidence-Aware Face Segmentation (Figure 7)

Section 4.3 and Appendix A.4 present an additional downstream application: using the generated bounding boxes and their associated token probabilities (Equation 5) to perform face-level mesh segmentation. The algorithm assigns each mesh face to a bounding box based on centroid containment (Equation 6), with conflict resolution using a containment rule (smaller box wins) and a confidence rule (higher-confidence box wins). Figure 7 shows qualitative segmentation results. The paper presents this as an emergent capability — "requires no additional training and relies solely on the generated bounding boxes and token probabilities." However, no quantitative segmentation metrics (e.g., part mIoU against ground-truth face labels) are reported, making it impossible to assess whether the segmentation is practically useful or merely visually plausible.

Ablation Studies and Robustness Checks

Dual-encoder vs. single-encoder (Table 2): The dual-encoder architecture consistently outperforms a single-encoder variant across all evaluated tasks. On pure box listing (Type 0), dual-encoder improves BBox IoU by +7.06 absolute points. On Part QA and Multi-Part Grounding, the paper reports "uniform gains across all metrics" without specifying per-metric values. This is the only architectural ablation reported — there is no ablation on encoder capacity (varying the number of structural vs. semantic tokens), no ablation on the pretraining curriculum (Stage 1 only vs. Stage 1 + Stage 2 vs. Stage 2 only), no ablation on freezing strategy (what if the structure encoder is fine-tuned in Stage 2?), and no ablation on the choice of LLM backbone (Qwen 2.5 VL vs. alternatives).

Coordinate quantization granularity (no explicit ablation): The paper uses K = 128 bins for coordinate quantization (Section 3.5, Equation 7) but does not ablate this choice. The t-SNE visualization (Figure 10) shows that 128 tokens form a structured manifold, but the paper does not test whether 64 or 256 bins would change performance — this is a design parameter with direct implications for vocabulary size, spatial resolution (each bin spans ~0.0157 units in normalized coordinates), and the difficulty of the autoregressive prediction task.

Training data scale and diversity (no ablation): The model is trained on 85,771 objects with ~23 parts each. The paper does not ablate dataset size — no experiments show how performance scales with the number of training objects or the diversity of object categories. Category distribution is reported in Table 6 (top-19 categories by frequency), but there is no analysis of whether performance varies systematically across categories (e.g., better on furniture than on mechanical assemblies).

Stage 1 pretraining necessity (no ablation): Stage 1 pretrains the structure encoder on 3.6M geometry-only objects. The paper does not report what happens if Stage 1 is skipped — i.e., if the structure encoder is initialized from Hunyuan 2.1 3D Shape VAE and immediately used in Stage 2 without geometry-only pretraining. This makes it impossible to assess whether the Stage 1 pretraining is essential for the reported performance or whether the gains come primarily from the dual-encoder architecture and instruction tuning.

Freezing strategy (no ablation): In Stage 2, the structure encoder and original Qwen 2.5 VL token embeddings are frozen, while the semantic encoder, transformer layers, and new token embeddings are trained. The paper does not report ablations testing alternative freezing strategies — e.g., fine-tuning the structure encoder in Stage 2 (which might improve multimodal fusion at the cost of catastrophic interference), or freezing the transformer layers (which would test whether the gains come from the LLM's language prior or from the learned cross-modal attention patterns).

Choice of geometry backends (no comparison of alternative backends): Part-X-MLLM uses OmniPart for generation and Nano3D/VoxHammer for editing. The paper does not compare performance when the same Part-X-MLLM outputs are executed by different geometry backends — e.g., does OmniPart produce better generation from Part-X-MLLM's boxes than an alternative like TRELLIS or Hunyuan3D? This would test the claimed "model-agnostic" property of the output grammar. Without such a comparison, the model-agnostic claim remains a design principle rather than an empirically validated property.

Semantic granularity control robustness (no sensitivity analysis): The clustering algorithm (Appendix A.2.1) depends on hyperparameters ε and minPts in DBSCAN, and the mixing weight α in feature fusion. The paper shows one trajectory (22 parts → 2 parts) but does not analyze sensitivity to these hyperparameters — e.g., how does the clustering change with different α values? Are there categories where the clustering consistently fails? Does the algorithm produce the same clusters when run with different random seeds (DBSCAN is deterministic for a given dataset, so this is less of a concern)?

Confidence-aware face segmentation quality (no quantitative evaluation): Figure 7 shows qualitative segmentation results, but no metrics (IoU, boundary F1, part-level accuracy against ground-truth face labels) are reported. The paper claims "high-quality, fine-grained face-level segmentation," but without quantitative evaluation this is unverified.

ReSTᵉᵐ revision model (not applicable — this is from the reference example, not Part-X-MLLM): The paper does not include RL-based fine-tuning or iterative self-improvement experiments. The limitation section mentions that "fine-tuning on 3D tasks may reduce the base LLM's general language capabilities" but does not evaluate this empirically — no experiments measure the model's performance on general language benchmarks before and after Stage 2 training.

Critical Assessment

Central Claim 1: Part-X-MLLM unifies generation, editing, and reasoning under a single, executable grammar

Partially supported. The paper demonstrates that a single model architecture, trained on 11 task templates, can produce structured programs for all 11 task types (Table 7 confirms non-blank entries across Types 0–10). This is a genuine engineering achievement — prior systems handled subsets of these tasks with separate architectures. However, unification is demonstrated as an output-space property, not as an interactive or sequential capability. The paper does not show the model handling multi-turn interactions (e.g., "describe the parts" → "now remove the left armrest" → "what does the chair look like now?"), which would test whether the persistent BBox tokens enable stable part identity across turns. The claim that the grammar provides "stable part identity and grounding" rests on the architecture's design (BBox tokens as persistent references), but the paper does not empirically validate this — there is no experiment showing that the same BBox token sequence is consistently reused for the same part across different task prompts, or that editing a part by its BBox token actually affects the correct spatial region.

The model-agnostic property of the output grammar is also asserted rather than tested. Part-X-MLLM uses OmniPart for generation and Nano3D/VoxHammer for editing, but there is no demonstration that the same program output can be executed by multiple different backends to produce comparable results, or that swapping the backend does not degrade quality. The paper would be stronger if it showed, for example, that Part-X-MLLM's generated bounding boxes produce valid 3D assets through both OmniPart and an alternative synthesis pipeline.

Central Claim 2: The dual-encoder architecture avoids representational conflict and delivers consistent gains

Supported by the reported ablation (Table 2), but with important interpretive limits. The +7.06 BBox IoU improvement and the "uniform gains across all metrics" on Part QA and Multi-Part Grounding provide evidence that separating structure and appearance pathways helps. The pattern of improvement on both geometry-heavy tasks (box listing) and language-heavy tasks (Part QA) is consistent with the representational conflict hypothesis: if the single encoder merely needed more capacity, the dual encoder might help on geometry but would not necessarily help on language tasks (which could attend more to semantic features). The fact that both improve suggests the single encoder's fused representation is genuinely suboptimal.

However, the claim is tested with a single comparison (dual vs. single encoder) on a single test set (UniPart-Bench, 400 objects) with a single model scale, and the specific metric values are not provided in the main text for language tasks. The paper does not isolate why the dual encoder helps — is it the separate parameters (more total encoder capacity), the separate input modalities (structure sees XYZ+normals only; semantic sees XYZ+RGB), or the pretraining curriculum (structure encoder pretrained on geometry-only data)? A controlled ablation that equalized total encoder parameters between the dual and single conditions (e.g., single encoder with 2× width) would distinguish capacity effects from representational effects. Without this, the +7.06 IoU gain could reflect the dual encoder simply having more parameters devoted to encoding the point cloud, not a genuine resolution of representational conflict.

The paper does not test whether the representational conflict hypothesis generalizes beyond the part-decomposition setting. If the hypothesis is correct, dual-encoder designs should improve performance on any 3D-language task where color and shape provide complementary information — scene-level captioning, object retrieval, or 3D visual grounding. Testing on an established benchmark (e.g., ScanQA, ScanRefer) would strengthen the claim.

Central Claim 3: Semantic granularity control via part clustering

Demonstrated qualitatively, not evaluated quantitatively. Figure 6 shows the clustering algorithm reducing 22 parts to 2, which demonstrates feasibility. However, the paper provides no evidence that the clusters are semantically meaningful — there is no comparison against human-defined coarse groupings, no cluster purity metric, and no analysis of failure modes. The algorithm's sensitivity to hyperparameters (ε, minPts, α) is not explored. The paper claims this provides "flexible control over the level of detail without manual intervention," but the user must still choose the hyperparameters (or the target number of clusters, which the paper claims is not required — yet the Figure 6 trajectory implies some manual selection of stopping points). The semantic granularity control is presented as a major contribution (listed as contribution bullet 3 in the introduction), but the evaluation is the weakest of any claimed contribution in the paper.

Central Claim 4: Auto-located 3D editing provides a language-native frontend for geometry tools

Feasibility demonstrated, but practically unvalidated. Figure 5 shows qualitative editing results (addition, removal, modification), but there is no quantitative evaluation: no localization accuracy metric (how often does the predicted BBox actually enclose the correct part?), no edit success rate, and no comparison against alternative localization methods (e.g., using a separate part segmentation model to generate the mask, then applying the same Nano3D/VoxHammer backend). The paper claims the editing is "auto-located" — the model converts language into spatial regions automatically — but without measuring how often this localization fails, the practical utility is unknown. A user study or a systematic evaluation of localization accuracy on the edit program tasks (Types 8–10) would transform this from a qualitative demonstration to an evaluated capability.

Unaddressed Weaknesses

Single model family, single scale, single dataset. All experiments use Qwen 2.5 VL with the Hunyuan 2.1 3D Shape VAE encoder, trained on a single in-house dataset. There is no replication on other LLM backbones, other 3D encoders, or public benchmarks. UniPart-Bench, while spanning 11 task types, is drawn from the same distribution as the training data — the paper does not evaluate out-of-distribution generalization (e.g., on objects from unseen categories, or on 3D representations other than point clouds, such as meshes or voxel grids). The authors note that UniPart-Bench is a "held-out subset of our 85,771-object training dataset, ensuring identical data construction pipeline and distribution characteristics" (Appendix A.8). This means the benchmark tests in-distribution generalization, not transfer to genuinely novel data.

No statistical reporting. All results are reported as point estimates without error bars, confidence intervals, or significance tests. Given the test set size (400 objects for UniPart-Bench), per-task sample counts can be small (Table 5 shows raw counts per template, with balancing further reducing some counts). The reported improvements, while large in magnitude (+7.06 IoU, +18.7 SBERT), could be noisy estimates — without error bars, the reader cannot assess whether the dual-encoder advantage is robust across different random seeds or train/test splits.

Missing baselines. The most important missing comparison is a version of Part-X-MLLM where the grammar is replaced with free-form text containing bounding box descriptions (e.g., "the left front leg is located at coordinates (0.1, 0.2, 0.3) to (0.15, 0.4, 0.35)"). This would test whether the structured grammar (special tokens, quantization) contributes to performance beyond simply training the model to output spatial coordinates in text. If a text-output variant performs comparably, the grammar's complexity (coordinate quantization, special token vocabulary, edit operator syntax) may be unnecessary. The paper does not address this because the grammar is taken as an axiom — the entire system design assumes it is necessary for stable referencing and downstream execution — but it is an empirical question whether natural language coordinate descriptions would suffice.

No latency or computational cost analysis. The dual-encoder design processes two point clouds (40,960 and 10,240 points) through two separate encoders, then fuses the tokens in a large autoregressive decoder. The paper does not report inference latency, memory usage, or FLOPs, making it impossible to assess whether the reported gains are cost-effective compared to simpler approaches (e.g., single encoder with more parameters, or a two-stage pipeline where a part segmenter produces boxes and a separate LLM generates text). The absence of any compute analysis is particularly notable given that the paper claims the dual-encoder design "avoids representational conflict" — a claim about representation quality that should be weighed against the increased computational cost of running two encoders instead of one. The reported training costs (2 days + 5 days on 64 A100 GPUs) are substantial, and the paper does not discuss whether similar results could be achieved with less compute through alternative training strategies.

No evaluation of general language capability preservation. The limitation section acknowledges that "fine-tuning on 3D tasks may reduce the base LLM's general language capabilities," but this is not empirically evaluated. The paper freezes the original Qwen 2.5 VL token embeddings as a mitigation, but does not measure whether the transformer layer fine-tuning (which is trained) causes any degradation on standard language benchmarks. For a model that claims to serve as a "language-native frontend," preserving language capability is important — if the model becomes worse at understanding natural language instructions after 3D fine-tuning, the frontend is less useful.

Editing evaluation is entirely qualitative. The paper uses two editing backends (Nano3D and VoxHammer) but does not specify which backend was used for which Figure 5 results, does not compare the backends to each other, and does not report whether Part-X-MLLM's localization works equally well with both backends. The claim that Part-X-MLLM provides a "language-native frontend" for editing tools would be much stronger with a systematic evaluation: for a set of editing instructions, measure (1) whether the predicted BBox encloses the correct part (localization accuracy), (2) whether the edit is applied correctly within the mask (edit quality), and (3) whether unedited regions remain unchanged (edit preservation). None of these metrics are reported.

Confidence-aware segmentation is claimed but unvalidated. The face segmentation algorithm (Appendix A.4, Figure 7) is described as a downstream application enabled by the model's token probabilities. The paper labels this "high-quality" and "fine-grained" but provides no quantitative evidence. This is an example of a capability claim without empirical support — the algorithm exists and produces visual outputs, but whether those outputs are actually better than simpler baselines (e.g., assigning faces to the nearest bounding box by centroid distance without confidence weighting) is untested.

6. Limitations and Trade-offs

The Planning-Execution Decoupling Is Architecturally Validated but Not Empirically Demonstrated to Be Model-Agnostic

The assumption or constraint. The paper's central architectural claim is that Part-X-MLLM's structured token programs serve as a "model-agnostic control surface" that can drive "any compatible geometry engine" (Section 3.2, Section 3.3). The output grammar is designed to decouple symbolic planning (producing bounding boxes and edit commands) from geometric synthesis (executing those commands). This separation is presented as a key advantage: "a better geometry engine in the future can be swapped in without retraining the planning model."

The consequence. The model-agnostic property is asserted as a design principle but is never tested empirically. The paper uses exactly one generation backend (OmniPart, Section 4.1) and one editing backend (either Nano3D or VoxHammer, but the paper does not specify which produced which results in Figure 5). There is no experiment showing that the same Part-X-MLLM program output produces comparable-quality results when executed by multiple different backends. If Part-X-MLLM's bounding box predictions are implicitly tuned to the spatial requirements of its specific backends (e.g., OmniPart may expect certain box size distributions, padding, or coordinate conventions that other synthesis modules do not), then swapping the backend could degrade output quality—the grammar would not be truly model-agnostic in practice. Without testing this, a practitioner considering deploying Part-X-MLLM with a custom geometry engine has no evidence that the integration will work beyond the specific OmniPart/Nano3D configuration used in the paper.

What evidence exists in the paper. There is no evidence—no experiment, ablation, or even qualitative demonstration—comparing execution of the same program across multiple geometry backends. The paper states that "the rich information encoded in the generated token probabilities also enables advanced downstream tasks" (Section 4.1) and mentions using both Nano3D and VoxHammer as editors, but does not compare them, specify which was used for Figure 5, or test whether Part-X-MLLM's localization accuracy is backend-dependent. The editing results in Figure 5 are qualitative demonstrations with no quantitative metrics for any backend.

Mitigation status. Not addressed. The paper treats model-agnosticism as a design property of the grammar rather than an empirical claim requiring validation. Section 8 (Conclusion) makes no mention of this gap. A practitioner deploying Part-X-MLLM with a different backend would be operating without evidence.

Difficulty Estimation for Quality Is Absent — No Mechanism to Detect or Flag Unreliable Outputs

The assumption or constraint. The paper assumes Part-X-MLLM produces high-quality structured programs across all inputs, but provides no mechanism for the model to estimate its own confidence or flag unreliable outputs. This is distinct from the confidence-aware face segmentation application (Section 4.3, Appendix A.4), which applies post-hoc to already-generated bounding boxes. The model itself has no built-in difficulty awareness—it does not know when a part decomposition is likely to be wrong, when an edit localization is uncertain, or when an object is outside its training distribution.

The consequence. In a deployment setting, the system will silently produce incorrect bounding boxes, erroneous edit commands, or hallucinated part descriptions with no signal to the user or downstream system that the output is unreliable. This is particularly dangerous for editing (Types 8–10), where an incorrectly localized bounding box would cause the geometry engine to delete, modify, or add geometry in the wrong spatial region—potentially destroying parts of the asset the user did not intend to change. Unlike the reference example's Part-X-MLLM, which explicitly builds difficulty estimation into its core framework and uses it to route prompts to different strategies, Part-X-MLLM treats all inputs uniformly. The paper provides no analysis of when or why Part-X-MLLM fails—there is no breakdown of BBox IoU or edit localization accuracy by object category, part count, or prompt complexity.

What evidence exists in the paper. None. All results are aggregate metrics (Table 1, Tables 3–4, Table 7) with no difficulty-based or confidence-based stratification. The per-task results in Table 7 report average metrics but do not analyze failure cases, worst-performing categories, or conditions under which performance degrades. The confidence-aware face segmentation algorithm (Equation 5) computes per-box confidence from token probabilities, which demonstrates that confidence signals exist in the model's outputs, but the paper never evaluates whether these confidence scores correlate with actual output quality or could be used to filter unreliable predictions. The qualitative results (Figures 4, 5, 8, 9) are selected examples with no indication of failure rate.

Mitigation status. Not addressed. The paper introduces per-box confidence scores (Equation 5) as a post-hoc mechanism for face segmentation, but does not connect this to output quality estimation or failure detection. Section 5 (Conclusion) does not mention reliability, calibration, or uncertainty quantification. A practitioner deploying this system has no way to know when to trust the outputs and when to fall back to human intervention.

Evaluation Is Confined to a Single In-House Dataset with No Out-of-Distribution Testing

The assumption or constraint. All training and evaluation use the authors' in-house part-centric 3D dataset of 85,771 objects (Section 4.2, Appendix A.6). UniPart-Bench, the evaluation benchmark of 400 objects, is explicitly constructed as "a held-out subset of our 85,771-object training dataset, ensuring identical data construction pipeline and distribution characteristics" (Appendix A.8). The train/test split is a single deterministic partition at 0.5% with no cross-validation. The paper does not evaluate on any public benchmark, any dataset with different annotation conventions, any out-of-category objects, or any 3D representation other than point clouds.

The consequence. All reported results measure in-distribution generalization: how well the model performs on objects drawn from the same distribution, annotated by the same pipeline, and presented in the same format as its training data. This does not test whether the model has learned generalizable part-level reasoning or has simply memorized the annotation patterns of the specific labeling pipeline. Several specific generalization failures are untested but plausible: (1) the model may fail on object categories not represented in the top-19 categories (Table 6 covers everyday objects but likely misses specialized domains like medical devices, industrial machinery, or anatomical models); (2) the model may fail on 3D inputs in different representations (voxel grids, meshes, NeRFs) since it only sees point clouds during training; (3) the model may fail when part annotation conventions differ—the LMM labeling pipeline (Appendix A.6.1) imposes specific annotation patterns (≤15-word Q2 descriptions, specific Q1 naming conventions) that may not match how users describe parts in practice; (4) the model may fail on objects with significantly different part counts than the training average of ~23 parts per object.

The deterministic 0.5% test split (roughly 430 objects, with 400 used for UniPart-Bench) further means that all results are point estimates from a single data partition. Without multiple random splits or cross-validation, there is no way to assess whether the reported improvements (+7.06 BBox IoU, +18.7 SBERT) are stable or sensitive to the specific objects that happened to fall in the test set.

What evidence exists in the paper. The paper explicitly notes UniPart-Bench's in-distribution nature (Appendix A.8) and does not claim out-of-distribution generalization. However, the paper also does not discuss this as a limitation—it presents the results as evidence of general part-aware 3D intelligence without qualifying the scope. There is no per-category analysis (performance on furniture vs. vehicles vs. electronics), no analysis of how performance varies with part count, and no experiment where the model is evaluated on objects from entirely held-out categories. The category distribution in Table 6 shows top-19 categories, implying that some categories are represented by very few objects—the paper does not report whether these rare categories are present in the test set or whether performance degrades on them.

Mitigation status. The paper makes no attempt to address this. There is no discussion of generalization in Section 5 (Conclusion), no suggestion of evaluating on public benchmarks (ScanNet, PartNet, ShapeNet-Part), and no acknowledgment that the single-dataset, single-split evaluation limits the strength of the empirical claims. A practitioner considering deployment in a novel domain (e.g., industrial CAD, medical imaging) has no evidence that the approach transfers.

Edit Localization and Program Execution Are Demonstrated Qualitatively but Never Quantitatively Evaluated

The assumption or constraint. The paper claims that Part-X-MLLM enables "auto-located 3D editing" (Section 3.4, Figure 3 bottom row) by converting natural language instructions like "remove the handle" into structured programs (<dels> <boxs> tokens <boxe> <dele>) that geometry engines execute. The editing tasks (Types 8–10) are part of the 11 task families and are included in the instruction tuning data.

The consequence. The practical utility of language-driven editing depends entirely on localization accuracy: does the predicted bounding box actually enclose the correct part? If the model localizes "the handle" to a box that includes part of the door plus empty space but misses the actual handle, the downstream editor will delete, modify, or add geometry in the wrong region. Without measuring this accuracy, a practitioner cannot assess whether the system is reliable enough for production use. The paper's qualitative editing results (Figure 5) show successful edits, but these are selected examples—the failure rate, common failure modes, and dependence on prompt phrasing are completely unknown.

This gap is particularly acute because edit localization is a harder problem than part decomposition (Types 0–2). In part decomposition, the model lists all parts and their boxes—a reasonably well-decomposed output can have mediocre per-box IoU but still cover most of the object. In edit localization, the model must map a specific linguistic reference to exactly one correct box—an error means editing the wrong part, which is often worse than making no edit at all. The paper provides no evidence that the model can reliably distinguish between, say, "remove the left front leg" vs. "remove the right front leg" given their nearly identical structural signatures.

What evidence exists in the paper. Only qualitative visualizations (Figure 5). Table 7 reports per-task metrics for Types 8–10, but the paper only summarizes these results qualitatively: "Language-intensive tasks (T7 Part QA, T10 Edit—Add) obtain the highest SBERT/SimCSE and strong lexical metrics" and "T0/T2/T10 show the strongest geometric alignment" (Appendix A.8). These metrics measure text similarity (SBERT/SimCSE/BLEU/ROUGE/METEOR) and bounding box IoU against the ground-truth program, but do not measure whether the executed edit (the actual geometry output from Nano3D/VoxHammer) matches the user's intent. No metric captures: (1) whether the correct part was localized (did the predicted box enclose the intended target?), (2) whether the edit was applied correctly within the mask, (3) whether unedited regions remained unchanged, or (4) whether the final edited asset meets the user's specification. The text-similarity metrics evaluate whether the model's output program tokens match the ground-truth program tokens, which is a proxy for edit quality but does not guarantee that the downstream execution succeeded.

Mitigation status. Not addressed. The paper treats the production of a syntactically valid edit program as the evaluation endpoint, without measuring whether those programs produce correct edits when executed. Section 5 (Conclusion) does not mention this gap. A practitioner deploying Part-X-MLLM for editing would need to conduct their own evaluation of localization accuracy and edit success rate—the paper provides no guidance on expected failure rates or conditions under which localization degrades.

Computational Cost and Latency of the Dual-Encoder Design Are Neither Measured Nor Discussed

The assumption or constraint. The paper's architectural contribution is a dual-encoder design where two separate encoders (Structure and Semantic) process point clouds at different resolutions (40,960 and 10,240 points, respectively), and their outputs are fused in a large autoregressive decoder (Qwen 2.5 VL). This design is motivated by a representational hypothesis (avoiding conflict between structure and appearance features, Section 3.2) and justified empirically by the ablation study (Table 2) showing improvements over a single-encoder variant. However, the paper provides no measurement of inference cost: no latency benchmarks, no memory usage profiling, no FLOPs analysis, and no comparison of computational efficiency against the single-encoder baseline or against alternative two-stage pipelines.

The consequence. The reported accuracy improvements (+7.06 BBox IoU on box listing, "uniform gains" on Part QA and Multi-Part Grounding) come at an unknown computational cost. A practitioner deciding whether to adopt the dual-encoder design needs to weigh these gains against the additional inference time and hardware requirements. Several cost factors are unmeasured:

Encoding cost: The single-encoder baseline processes one point cloud; the dual-encoder processes two, including one at high resolution (40,960 points with 6 channels for XYZ+normals). This roughly doubles the encoding FLOPs, though the semantic encoder processes fewer points (10,240).
Decoder cost: The autoregressive decoder must attend to a larger token sequence (structural tokens + semantic tokens + text tokens vs. unified tokens + text tokens). The longer prefix increases the quadratic attention cost in the decoder.
Memory cost: Storing two encoder models (even if architecturally identical) and their output tokens roughly doubles the encoder memory footprint compared to a single encoder.
Latency: Two encoders can run in parallel on sufficient hardware, but the autoregressive decoder is inherently sequential—generating a full program with multiple bounding boxes and text descriptions requires many decoding steps, each attending to all encoder tokens.

The paper's ablation (Table 2) compares architectures at equal training conditions but does not control for total encoder parameters or FLOPs. If the dual encoder has roughly 2× the encoder parameters of the single encoder, the reported gains could partially reflect increased capacity rather than the representational benefit of modality separation. A capacity-controlled comparison (e.g., single encoder with 2× width, or dual encoder with 0.5× width per pathway) would isolate the architectural effect from the capacity effect. Without this, a practitioner cannot determine whether the dual-encoder design is more parameter-efficient than simply scaling up a single encoder.

What evidence exists in the paper. The paper reports only training time (Stage 1: 2 days on 64 A100 GPUs; Stage 2: 5 days on 64 A100 GPUs, Section 4.1). No inference metrics are reported. The ablation (Table 2) does not equalize parameter count between the dual and single encoder conditions. The paper does not discuss latency, throughput, or memory as design considerations, despite the dual-encoder design being the central architectural contribution. The t-SNE visualization of special token embeddings (Figure 10) and the confidence-aware face segmentation application (Figure 7) demonstrate the richness of the learned representations, but the paper does not connect this representational quality to computational cost.

Mitigation status. Not addressed. The paper does not acknowledge inference cost as a limitation or trade-off. Section 5 (Conclusion) mentions only that "longer sequences slow inference" (referring to the autoregressive decoding of programs with many parts), proposing "simple compaction and hierarchical grouping" as mitigations. The dual-encoder cost—the core architectural cost—is not discussed. A practitioner deploying Part-X-MLLM in a latency-sensitive application (interactive 3D editing, real-time part queries) has no data to determine whether the system is fast enough. A practitioner considering the single-encoder variant for efficiency cannot assess the accuracy-efficiency tradeoff because the efficiency side is missing entirely.

Baseline Comparisons Confound Architectural Merit with Training Data Mismatch

The assumption or constraint. The 3D understanding experiments (Tables 3 and 4) compare Part-X-MLLM against prior 3D MLLMs (PointLLM, 3D-LLM, GPT4Point, ShapeLLM, LL3DA, 3D-LLAVA) on UniPart-Bench, which is drawn from the same dataset and annotation pipeline that Part-X-MLLM was trained on. The baselines were trained on different datasets and were not designed for part-level tasks with structured output grammars. The paper does not fine-tune any baseline on the Part-X-MLLM training data or attempt a controlled comparison where all models receive equivalent training on the target task distribution.

The consequence. The reported improvements (+18.7 SBERT, +25.5 SimCSE, +21.3 BLEU-1 on part Q&A; +10.4 SBERT, +9.4 SimCSE, +18.8 BLEU-1 on object captioning, Tables 3–4) reflect a combination of (1) Part-X-MLLM's architectural advantages (dual encoder, structured grammar, instruction tuning) and (2) the fact that Part-X-MLLM was trained on the evaluation distribution while baselines were not. The paper cannot disentangle these factors. If the baseline models were fine-tuned on the same part-centric dataset with their existing architectures, some fraction of the performance gap would likely close—perhaps substantially. The comparison as presented tests "Part-X-MLLM (trained for parts on part data)" vs. "prior MLLMs (trained for holistic understanding on different data)," which is not a fair test of architectural merit.

This confound is particularly acute for the structured output format. UniPart-Bench expects answers with embedded bounding box tokens (e.g., "The seat cushion is located at <boxs> tokens <boxe>"). Baseline 3D MLLMs were not trained to produce such tokens—their output vocabularies may not even include the special box tokens. If a baseline cannot produce the expected output format, it receives a low score regardless of whether it "understands" the part structure—the model might produce a textually correct answer ("The seat cushion is in the center of the chair") that scores poorly against a ground-truth that contains BBox tokens. The paper does not address whether the baselines can structurally produce the required output format or whether the metrics are evaluating understanding vs. format compliance.

What evidence exists in the paper. Table 3 and Table 4 report the raw comparisons. The paper acknowledges that UniPart-Bench is a held-out subset with "identical data construction pipeline and distribution characteristics" (Appendix A.8) to the training data, but does not discuss this as a confound in the baseline comparisons. No baseline is fine-tuned on the Part-X-MLLM training data. No experiment controls for the amount of part-specific training data—all models are evaluated with their original (off-distribution) training.

Mitigation status. Not addressed. The paper does not acknowledge the training data confound, does not fine-tune baselines, and does not qualify the baseline comparisons as partially reflecting training data alignment rather than architectural superiority. A practitioner choosing between Part-X-MLLM and fine-tuning an existing 3D MLLM on their own part-annotated dataset has no controlled comparison to guide that decision—the paper's results cannot distinguish whether Part-X-MLLM's architecture or its training data (or the interaction of both) drives the reported gains.

7. Implications and Future Directions

How This Work Changes the Landscape

Part-X-MLLM makes an architectural argument, not just an empirical one. The paper's most consequential claim is that the planning-execution boundary in 3D interaction is a language modeling problem—that an LLM with appropriate 3D encoders and a structured output grammar can replace the heterogeneous, task-specific interfaces (part segmenters for decomposition, bespoke editors for manipulation, separate Q&A models for understanding) with a single instruction-following model whose outputs are machine-executable programs. This is a reframing, not a paradigm shift: it does not introduce fundamentally new capabilities in geometry synthesis or language understanding, but it reorganizes how these capabilities are composed. The shift is from monolithic 3D systems (where understanding, generation, and editing share the same architecture and representation) to modular systems where a language-native planner produces an intermediate representation consumed by swappable geometry backends.

The magnitude of this reframing depends on whether the modularity actually works in practice—and on this point the evidence is incomplete. The paper demonstrates that Part-X-MLLM can produce structured programs for 11 task types and that those programs can be executed by OmniPart and Nano3D/VoxHammer. But the model-agnostic property—the claim that any compatible geometry engine can consume the programs—is asserted, not tested. The paper never shows the same Part-X-MLLM output being executed by multiple different backends, or quantifies how backend choice affects output quality. If future work validates model-agnosticism (by demonstrating that Part-X-MLLM's programs produce comparable results across a range of synthesis and editing backends), this reframing has real force: it means language model training and geometry engine development become decoupled innovation tracks, with the grammar as a stable interface contract. If model-agnosticism fails (because Part-X-MLLM's bounding box predictions are implicitly tuned to specific backend conventions), the reframing collapses to a demonstration that an LLM can be fine-tuned to drive one specific geometry pipeline—useful, but not transformative.

What the paper does shift concretely is the research priority around 3D encoding architectures. The dual-encoder ablation (Table 2) provides causal evidence—not just correlation—that forcing a single encoder to handle both geometry and color produces worse representations than separating them, with the +7.06 BBox IoU improvement on pure box listing being the cleanest signal. This is a diagnostic finding: it tells the field that modality-specific encoding followed by cross-modal fusion in the decoder is worth investigating as a general design pattern for 3D-language tasks, not just for part-level work. The prior assumption—implicit in the single-encoder designs of PointLLM, 3D-LLM, GPT4Point, and ShapeLLM—was that a sufficiently powerful encoder could internally disentangle structure and appearance. The ablation challenges this assumption with evidence that the disentanglement must be architectural, not just learned.

The paper also reconciles a tension in the part-generation literature. Prior work split into two camps: 2D-lifting pipelines (Part123, SAMPart3D, PartField) that produce part segmentations but are "prone to view inconsistencies and weak 3D constraints," and native 3D generators (AutoPartGen, BANG, OmniPart, X-Part) that produce high-quality geometry but "lack a unified language interface." Part-X-MLLM shows that the language interface can be separated from the geometry generation—the planning model provides part decompositions and edit programs, and the geometry engines provide synthesis. This resolves the tension by showing that the two capabilities (language understanding and geometry generation) do not need to coexist in the same model. The implication is that future 3D part-generation research should focus on improving the geometry backends and the language planners independently, with the grammar serving as the integration point—a more efficient division of labor than building monolithic systems that attempt both.

One research direction that becomes less attractive after this work: building end-to-end models that directly output 3D geometry from language without an intermediate structured representation. Part-X-MLLM's success at producing bounding box programs that drive high-quality synthesis suggests that the intermediate representation (structured, auditable, machine-parseable) provides benefits—persistent part identity, semantic granularity control, edit localization—that end-to-end generation obscures. The field may shift toward Language → Structured Program → Geometry pipelines rather than Language → Geometry pipelines, with the structured program serving as the interface where interpretability and modularity live.

Follow-Up Research This Work Enables

Stress-testing model-agnosticism by executing the same Part-X-MLLM programs on at least three different geometry backends. The paper claims the grammar is a "model-agnostic control surface" but tests it with exactly one generation backend (OmniPart) and one editing backend (unclear whether Nano3D or VoxHammer for each Figure 5 result). A follow-up study would take Part-X-MLLM's bounding box predictions on UniPart-Bench (or a wider benchmark) and execute them through OmniPart, Hunyuan3D, and TRELLIS as generation backends, plus Nano3D, VoxHammer, and Instruct-NeRF2NeRF as editing backends. The key metrics would be (1) whether the same bounding box programs produce valid geometry in all backends (backend-agnostic success rate), and (2) whether output quality varies systematically with the backend (measured via CLIP similarity to the part descriptions, or user studies). If performance degrades substantially on backends other than OmniPart, the model-agnostic claim fails—Part-X-MLLM would be implicitly tuned to one backend's spatial conventions, and the grammar would be a protocol, not a universal interface.

Measuring edit localization accuracy against human-annotated edit targets to quantify the reliability gap. The paper demonstrates qualitative editing (Figure 5) but provides no quantitative evaluation of whether Part-X-MLLM's predicted bounding boxes actually enclose the correct target parts for edit instructions. A follow-up study would construct a benchmark of, say, 500 edit instructions across diverse object categories (e.g., "remove the left front leg," "change the material of the seat cushion," "add a knob to the top drawer"), with human annotators marking the ground-truth bounding box for each edit target. The evaluation would measure localization accuracy (BBox IoU between predicted and ground-truth edit target boxes), false-positive rate (editing an incorrect part), and false-negative rate (failing to localize a referenced part). This would quantify the practical reliability of the language-native editing frontend and identify failure modes—for instance, whether the model confuses structurally similar parts (left vs. right leg, front vs. back drawer) at rates that would make the system unusable in practice. The paper's per-task results (Table 7, Types 8–10) measure BBox IoU against the ground-truth program, which evaluates whether the model learned the training distribution, not whether it actually localizes edits correctly on novel instructions.

Evaluating out-of-distribution generalization on public part-annotation benchmarks to test whether the model learns generalizable part reasoning or memorizes annotation conventions. All training and evaluation occur on the authors' in-house dataset with its specific LMM-based annotation pipeline (Q1/Q2 naming conventions, ≤15-word Q2 descriptions). A follow-up study would evaluate Part-X-MLLM on established part-annotation benchmarks—PartNet (fine-grained part hierarchies for everyday objects), ShapeNet-Part (part segmentation with category-specific label sets), or the PartNet-Text benchmark (if available). Because these benchmarks use different annotation conventions, different part granularities, and different naming schemes than the training data, the evaluation would test whether Part-X-MLLM's part reasoning transfers or is tightly coupled to its training distribution. A strong result (comparable BBox IoU on PartNet despite annotation convention differences) would suggest the model has learned generalizable part decomposition from geometry. A weak result (substantial degradation) would suggest the model relies on distribution-specific cues and would need retraining or fine-tuning for each new annotation convention. The paper explicitly notes UniPart-Bench's in-distribution nature (Appendix A.8) but does not discuss this as a limitation—this follow-up would directly test the scope of the claimed capabilities.

Isolating the contribution of Stage 1 geometry-only pretraining by ablating it against alternative initialization strategies. The paper's training curriculum has two stages: Stage 1 pretrains the structure encoder on 3.6M geometry-only objects, and Stage 2 instruction-tunes the full model. The paper never reports what happens if Stage 1 is skipped—the structure encoder could be initialized from Hunyuan 2.1 3D Shape VAE and immediately used in Stage 2 without geometry-only pretraining. A controlled ablation would compare: (1) full pipeline (Stage 1 + Stage 2), (2) Stage 2 only (structure encoder initialized from pretrained VAE, not further pretrained), and (3) Stage 2 only with a randomly initialized structure encoder. The key measurement would be BBox IoU on Type 0 (pure box listing)—the task closest to Stage 1's objective. If skipping Stage 1 causes substantial degradation on box listing but comparable performance on language-heavy tasks (Part QA, captioning), the pretraining is essential for geometry decomposition but not for language alignment—and the language model can learn spatial reasoning from the instruction tuning data alone. If skipping Stage 1 causes minimal degradation across all tasks, the VAE initialization already provides sufficient geometry features, and the 3.6M-object pretraining is unnecessary. This would clarify whether the dual-encoder advantage comes from the pretraining curriculum or from the architectural separation itself—a distinction the current ablation (dual vs. single encoder) cannot make.

Capacity-controlled encoder comparison to distinguish representational conflict from parameter count effects. The dual-encoder ablation (Table 2) compares a dual-encoder model against a single-encoder model, but does not control for total encoder parameters—the dual encoder likely has roughly 2× the encoder capacity. The reported +7.06 BBox IoU gain could reflect increased capacity rather than the benefit of modality separation. A follow-up study would compare the dual encoder against (1) a single encoder with 2× width (equal parameters, still fused XYZ+RGB input) and (2) a single encoder with the same width but also processing two separate input streams internally (late fusion). If the 2×-width single encoder closes most of the gap, capacity—not representational conflict—is the primary driver of the dual-encoder advantage. If the 2×-width single encoder still substantially underperforms the dual encoder, the representational conflict hypothesis is strengthened: simply adding parameters cannot overcome the entanglement of structure and appearance in a shared feature space. This experiment would determine whether the dual-encoder design is a compute-efficient way to add capacity or a qualitatively necessary architectural separation.

Training difficulty-aware output quality estimators using the token probabilities already computed by the model. The paper demonstrates that per-box confidence scores can be computed from token probabilities (Equation 5, Appendix A.4) and used for face segmentation. The model produces these confidence scores for free during autoregressive decoding, but the paper never evaluates whether they correlate with output quality. A follow-up study would measure the correlation between per-box confidence (Equation 5) and per-box IoU against ground truth across a large number of predictions. If the correlation is strong (e.g., boxes with confidence > 0.8 have IoU > 0.7 in 90% of cases, while boxes with confidence < 0.3 have IoU < 0.3), the confidence scores serve as an effective failure detector—the system could flag low-confidence bounding boxes for human review or fall back to alternative strategies. If the correlation is weak, the confidence scores capture something other than correctness (e.g., model calibration during training) and are not useful for deployment. This is a low-cost experiment (no model retraining needed, only evaluation on the existing test set) that would substantially improve the practical deployability of the system by providing a mechanism to detect unreliable outputs—one of the key gaps identified in Section 6.

Practical Applications and Downstream Use Cases

Interactive 3D asset editing for non-technical creators. Part-X-MLLM's core value proposition—converting natural language instructions into spatially-grounded edit programs—directly addresses a pain point in 3D content creation: artists and designers must currently learn complex modeling software (Blender, Maya, 3ds Max) to perform even simple part-level edits like "remove the handle" or "make the legs thinner." These operations require manual vertex selection, masking, or working through modal editing interfaces that demand spatial reasoning in 3D viewports. Part-X-MLLM, if its localization accuracy proves reliable (the key gap identified above), could enable a workflow where the user issues language instructions and the system automatically localizes the target, generates the cuboid mask, and delegates to a geometry engine. The paper's 11 task types already cover the core edit operations (add, delete, modify) and part-level Q&A for inspecting results. The specific benefit is reducing the expertise barrier for 3D editing—a user who can describe what they want in language can manipulate 3D assets—though this utility is contingent on the localization accuracy that the paper does not yet measure.

Automated training data generation for part-aware 3D models. The paper's dataset construction pipeline—LMM-assisted labeling of 85,771 objects with per-part bounding boxes and two levels of text annotation—is itself a contribution that can bootstrap further research. Part-X-MLLM, once trained, can be used to annotate additional 3D data at scale: feed point clouds through the model, collect the structured programs (bounding boxes + Q1/Q2 descriptions), filter by confidence score (Equation 5), and use the high-confidence outputs as training data for downstream models (part segmentation networks, geometry generators conditioned on part layouts, robotic grasp prediction models that require part-level spatial understanding). This addresses a key bottleneck: manual part annotation of 3D assets is expensive and inconsistent, and LMM-based annotation is limited by what the LMM can see in 2D renders. A 3D-native language model that understands part structure could produce more geometrically consistent annotations than 2D-lifting approaches because it operates on the full point cloud rather than rendered views. The paper's results on bounding box generation (Table 1, outperforming PartField and OmniPart, both of which are 2D-based or SAM-based) suggest that Part-X-MLLM's outputs are already competitive with existing annotation methods, and the inference cost (unknown, from the paper) is the main barrier to deployment as an annotation tool.

Part-level search and retrieval in 3D asset libraries. Many industries maintain large libraries of 3D assets—furniture catalogs for interior design, vehicle components for automotive engineering, character models for game development. Current search is typically metadata-based (searching by filename, tags, or holistic descriptions) because part-level search requires knowing which assets contain, say, "a cylindrical wooden leg with a tapered bottom" regardless of the asset's overall category. Part-X-MLLM's output representation—each part described by a bounding box and a natural language description—enables a straightforward indexing strategy: for each asset in the library, run Part-X-MLLM to produce the part-level decomposition, then index the Q2 descriptions with a text embedding model (CLIP or similar). A user query like "find all assets with curved armrests" or "find chairs with spindle backs" becomes a nearest-neighbor search in the part description embedding space, with the bounding box providing the exact spatial location of the matching part within each asset. The paper's semantic granularity control mechanism (Figure 6) could also support hierarchical browsing: coarse clusters for high-level filtering ("show all chairs with armrests"), fine-grained individual parts for precise matching. The bottleneck is inference throughput—the paper reports no latency numbers, and processing 85,771 training objects required 5 days on 64 A100 GPUs for a single training epoch—suggesting that batch indexing of large asset libraries would require substantial compute investment.